Readings
- Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
- If your were a CIO, what metrics would you want? How many is reasonable to have?
- Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
- How can measurements become obstacles to change?
- What measures are being used in your organization? Do they make sense?
The Star Ambulance Case: Take Two
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and bring it to class (Jan’s Section) or post it on the class blog (Rich’s section).
1. Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
I think the interest in measuring is because quantifying something through tangible measurements is for the most part easy to do, and it quickly conveys success or failure to those who care about the measurements. Measuring gives assurance a view to whether a decision that was made was the right decision based upon where you started, the end goal of the decision, and whether the decision is succeeding or not.
It is overkill to try and measure everything. Too much measurement could lead to “analysis paralysis” which is essentially too much information to the point one can’t discern whether something is working or not in the context of measurements.
I would want my organization to decide what to measure based upon what the basis of the decision made was. That is, what was the end goal of the decision that was put in place. If the end goal was, for example, to increase the speed of delivery of a product measurements should be based upon efficiency in the processes to deliver that product, availability of the processes to deliver that product, amount of rework done to deliver that product, and the failure/error rate of delivering that product. By choosing measurements that relate to the goal of a decision the organization would get a clear picture whether or not the goal of the decision was being met, or not. Measuring other factors may be beneficial, but they could also be a waste of time and resources. Even worse, measuring things that don’t give a representation of the success of meeting the goal of a decision could give a false impression that the goal is being met when in fact it is failing miserably.
Sean,
Good points. I’d add that measurements are regarded as valuable because they’re seen as objective and consistent over time.
Professor,
Thank you for the comment. When I was writing I kept thinking tangible/intangible and ease of ability to measure, when it was really the opposition of objective and subjective that I wanted to articulate but just couldn’t remember the right word at the time. Are measurements, since they are consistent, also helpful in identifying problems with projects and processes? Or are they used simply to determine if the decision to go forward with a project was the right call or not?
Sean,
Measurements can definitely help spot problems. Looked at over time, they also reveal trends and negative trends can be addressed before an urgent problem develops.
I agree measurements log data. Our compliance team believes the more data the better, Than their job is to make connections or sense of the data. And yes it can spot problems and finds trends and tendecies of how the organization works.
Sean,
Along with spotting problems and identifying negative trends, I believe measurement is the best way to seek continuous improvement. Demming’s PDSA, puts emphasis on evaluating and measuring performance to make improvements. In the Study step of PDSA, outcomes are monitored to test the validity of the plan for signs of progress and success, or problems and areas for improvements.
Sean, I agree that too much measuring is a bad thing. As you said, measurements are vital because they communicate value in an objective, tangible manner that business professionals can understand. In the world of executive management, value has to be proven through hard evidence. In order to appeal to head management, you need to chose a small number of measurements that strongly articulate the value of the IT department and demonstrate how IT helps the business to achieve its goals.
So then the important question is what to measure? How would you choose? What criteria would you use?
The first metric I would use would be up-time. In line with the readings, the first thing an IT department needs to establish is trust throughout the organization. High levels of up-time demonstrate the reliability of the IT function. The next set of metrics are organization specific. The measurements you use need to reinforce that IT is aligned with its own strategy as well as the business goals of the organization. Depending on the maturity of the IT department, the organization might see IT as a means of increasing efficiency. If this is the case, you would want to use change in operational costs as a KPI. Another KPI might be the success rate of IT projects, or more valuable measures might be the impact those projects have on cost-savings or increasing revenues.
It would be difficult to produce a comprehensive list, but if you can clearly identify the role of IT in an organization and understand where the company sees the value of IT, you can choose KPIs that reinforce that role whether its cost-savings, “keeping the lights on,” or driving innovative solutions.
Anthony,
How would you measure uptime. At the core server, if there is one? At each node of the network and take an average? What good is up time if its so slow for users that it is unusable. This gets very tricky, very fast, particularly if you are running a global network.
I totally agree with you Sean. Measuring is very important for companies because numbers are the easiest and quickest way to know the results. But something can’t be measured, so there are tangible and intangible benefits. Intangible benefits can’t be expressed by numbers. Trying to measure everything will be definitely overkill.
Jan’s Section
f your were a CIO, what metrics would you want? How many is reasonable to have?
If I was a CIO I would most likely pay close attention to the following metrics: financial metrics, communication effectiveness ( to measure communication is between employees, suppliers, shareholders and others) , customer relationship(focusing on customer attractiveness and customer relationship on 1 to 10 scales), project management ( including the following sub-measures: budget/cost performance, milestones met, quality/performance, innovation) and employee satisfaction.
As far as how metrics are reasonable to have, I’d say that it depends on the type and goal of the business,. The idea is really to find metrics that matter to stakeholders. However, It can become a sickness when there are too many metrics flying around and some are going up, others going down, and actions are paralyzed because there’s too much discussion about whether some test has illustrated an improvement or not.
Alexandra,
Be careful not to stray to areas outside IT’s explicit strategic plan. Certain project management and financial metrics are likely appropriate, as you suggest. However, unless I’m misunderstanding your idea of customer attractiveness, I don’t see that as a likely element of an IT strategic plan. Customer satisfaction could certainly qualify.
I like the idea of selecting measurements that appeal to interested stakeholders. Measuring the correct items for stakeholders can really help generate excitement, interest, and continued buy-in for projects if those stakeholders continually see important measurements improving. Those measurements may not necessarily be the “best” measurements for a particular project, but then it is usually stakeholders who get the project and its funding approved, and continued, and not the “best” measurements. Now, the tricky part may be finding the right balance between the measurements a project team knows are important and what the stakeholders want to see to convey the right message to the business to show the real value added to the company and its strategy.
Jan’s Section:
If you were a CIO, what metrics would you want? How many is reasonable to have?
Metrics can play a vital role in an organization. It ensures that organizations pay attention to their performance by prompting management to make adjustments when goals are not being achieved. If I were a CIO the metrics I would want would be the following:
– operational metrics: application performance and availability
– delivery metrics: project delivery, cost and satisfaction
– organizational metrics: performance reviews and employee expectations
– financial metrics: budget cost, modifications and resource cost
By using these listed above metrics I believe as CIO would be very beneficial to my organization as well as its IT function. These metrics would help proactively by allowing my organization as well as I to address issues before they can arise. As for the reasonable number of metrics, I believe the metrics I listed are the standard. However, based on the organization itself and its functions there might be more.
Magaly,
What you chose to measure and track should be directly related to IT’s strategic plan and the objectives it sets forth. In this regard, customer satisfaction deserves KPI status. Customers should be thought of broadly as both internal and external.
Magaly,
I really like your list, short and to the point. I do have a question if you had to rank those metrics going from most important to least how would you rank them? Also, how do you deal with subjective metrics? Would you agree that performance reviews are subjective? If not why aren’t they?
4. How can measurements become obstacles to change?
One example of measurements becoming obstacles to change is when an organization makes decisions based on vanity metrics. Vanity metrics are measurements that look good, like increasing users, page views, etc, but aren’t actually providing insight into the success of the business. If an organization is consistently presenting these vanity metrics, it can convince itself its more successful than it is and lure the company into a state of inertia. In reality, if the company were looking at actionable metrics, it might be able to pin point specific areas of improvement to keep moving the company forward. This issue stresses the importance of choosing specific, meaningful measurements that provide real insights and understanding about the company.
I have never heard of the concept of “vanity metrics,” but that makes total sense. Without knowledgeable stakeholders who know what KPI’s should be used to properly measure a project implementation then the IT/PMP can pass off those vanity metrics and paint the type of picture they want to paint for senior personnel. Another obstacle I was thinking was, for example, a minimum ROI set for all projects. Well, not all projects can be simply measured by ROI in terms of the value added to a business or its key processes. Not meeting a minimum ROI set by a business could become problematic since key projects could be passed up because a lot of the value is intangible in nature, or just too difficult to measure with a KPI as simple as ROI. I imagine a lot of businesses use financial measurements when making decisions, and leaving those measurements to become project hurdles could have negative consequences in the long run by passing up on projects that keep a business competitive in a fast changing business environment.
I came across the concept of vanity metrics in a book called The Lean Startup, its an interesting read and provides a lot of good insights! I never thought much about the importance of selecting metrics until I learned of that concept. However, even after being introduced to the importance of selecting the right metrics, this week’s comments demonstrate how difficult it is to choose meaningful metrics. In order to pick information that’s going to prove the worth of IT, you need an intricate knowledge of the company, IT’s role within the company, and the meaning of the metrics themselves. I found myself hard-pressed to identify appropriate metrics.
As for your suggestion of a minimum ROI for projects, I could see that holding a lot of weight with executives. I think this would provide a solid baseline for determining if a project is even worth considering or not. Although, as you said, sometimes its difficult to quantify the value that an IT project is adding to the company. I think in the early stages of an IT department, most, if not all, of the projects need to be backed by hard numbers because there isn’t trust between IT and the rest of the company. Once IT has a strong track record, is recognized as a value center, and is trusted by the organization, I could see looser requirements for hard, financial metrics.
Anthony,
Thanks for sharing the term “vanity metrics”. I too had also never heard it.
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
The interest in measuring is important because it allows you to track your progress, as well as allows for effective communication to senior members of your company. Being able to track your progress using a performance dashboard demonstrates the results that the IT department is contributing to the overall business goals. With the rise in information security in the last ten years, IT is now shifting itself from being just another expense to a driving tool that can help with business goals and profits. With that being the case measuring their success is more paramount. By measuring the progress your able to take a snapshot of the positives and negatives, thus adjusting your plan accordantly to achieve the proper alignment with the business goals.
In my opinion, I would say that it is overkill to try to measure everything. I would think that if you tried to measure everything you wouldn’t have time to analyze what needs to be a focus, thus never fixing the issues. Besides not fixing the issues you also waste resources in trying to measure everything, thus bringing little value to the business.
I would want my organization to decide by looking at both the aligned IT strategic goals and business goals. For instance, if you’re a company that sells products online, then customer satisfaction and application functionality should be at the top of the list to be measured.
I like you focus on business and IT strategic goals. You might also want to remember the COSO list of attributes that an organization wants from its IT organization. Would you have metrics around any of those?
Yes, I would definitely use the COSO list of attributes. These components work to create the foundation for sound internal control within the organization through directed leadership, shared values and a culture that emphasizes accountability for control. The information from these metrics will allow for risks to be identified and business objectives to be achieved.
Joseph, I agree with you. If every measure everything, it takes away from the focus of the important IT drivers that you want for the business. Yes it will be nice to measure everything possible, but when dealing with IT and how to drive it along with business strategies – it is best to focus on what is better. I would say in my opinion when working on projects in the context of what the class is about – IT Governance should be measured since it focuses on stakeholders, future of the company, organization’s contribution to IT, operations.
Jan’s Section:
If you were a CIO, what metrics would you want? How many is reasonable to have?
Using the appropriate metrics is critical for an organization’s success and help achieve organizational goals, especially for the success of IT within an organization. Metrics are vital elements for setting program priorities, allocating resources, and measuring performance. If I were a CIO, the metrics I would use would be the followings:
Compliance Metrics – This metric can be used to evaluate progress against the accessibility goals and objective through measurement of periodic performance report. Cumulative compliance metric can evaluate process against agency goals over time and give insight into the current level of risk exposure.
Process Metrics – This metric is used to evaluate IT process and process improvement. It gives the CIO visibility into the process of the IT functions to measure the effectiveness.
I believe in an organization, where there might be different metrics implemented according to different business needs. However, I think most organization would use compliance metric and process metric to monitor progress against business objectives, especially for alignment with IT functions and evaluate the effectiveness and efficiency of business processes.
Source:
https://cio.gov/performance-metrics-and-measures/
I would like to add some specific metrics on process metrics. I think it also can see as delivery metrics. Here are some specific metric under this category. 1. project cost 2. defect containment 3. supplemental delivery metrics 4. project satisfaction
Professor Jan section
Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
In todays environment the evaluation of a IT process does not just depend on two factors whether the IT project was on time and on budget but also depends on an important aspect whether the particular IT process has delivered value to the business thus the measuring comes in play.
Measuring business value is best done through defining Key process factors . Key Process factors are quantifiable measurements that are agreed to by stakeholders and help in evaluating whether process is aligned with its initial vision, strategy, and objectives.
It is true that it is overkill to measure everything because when we measure everything then it is like proving someone that we did in a line everything what they told us to do.It dampens the idea of bringing out new changes and adaption to the process.
The organization before deciding what to measure should decide what is the primary objective of the particular IT operation.If a IT process deals with an IT call center then the top priority is the customer satisfaction.It can be measured in terms of no of calls received and no of calls resolved satisfactorily.
This will help in achieving target of better customer satisfaction
Hi Vaibhav, I like how you stated that “evaluation of IT process is not just dependent on whether IT project was on time and on budget..” Many times management might think of IT as just a “cost center”, measuring IT on the basis of how much money IT is costing the firm. However, if IT organizations use metrics/measurements the right way, they can present business another perspective. They can show how IT is providing value to the firm, how much time it has saved, how business is having its objectives met. If these organizations are successful at showing these metrics to business, maybe the old-school mindset of IT being a “cost center” will be changed and management will seriously consider investing more into IT in the future.
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
The metrics I would focus on are
– Velocity of idea-to-offer cycle this measures the overall velocity as a concept goes from business idea to a production system.
– Business Value: Organizations need to measure Business Value in terms of three key performance areas: Revenue Growth, Cost Reduction, or Cost Avoidance. Creating a common taxonomy of tailored metrics around these three areas will guide and prioritize investments, and improve controls for timely delivery and decision making.
– Security: Metrics on patching, password strength and compliance are important but don’t always translate well to business executives. An alternative is to measure metrics that can be understood easily such as total number of network intrusion attempts thwarted per year, number of security breaches (if any) by extent, etc.
– IT Financial Maturity: Today’s CIOs are asked to run a value driven organization that aligns IT spending to corporate objectives and efficiently manages Capital vs Operational spend. Maintaining year over year productivity measures such as FTE, sourcing mix, and technology spend versus revenue/number of employees is a good starting point to assess IT Financial Maturity.
In the site below they have 10 for new CIO’s, but depending on your experience level and business it may be less or more.
http://www.ciodashboard.com/metrics-and-measurement/top-10-metrics-for-a-new-cio/
Hi Joe, I like your list on the type of metrics you want to measure. Its interesting you brought up Security as a metric. It is obviously a very important topic that companies might not have measurement based on. One metric my company has for Security, is around what applications have open “breaks” on them. Basically, if a application does not meet a control, a “break” is opened and the app dev manager needs to resolve it ASAP. Every week a report to the CTO goes out that shows the apps and the # of breaks opened against them in their organization. Management takes this report seriously. This is one way we use measurement for Security in my company.
Joe,
Who is IT’s balanced scorecard for? Do you think they would be interested in all of the metrics you proposed?
I believe this balanced scorecard would useful for startup companies and for companies looking to evaluate new projects. I say that a startup company would be interested all of these metrics as they go hand and hand. For example, if your startup company has frontend web application, you can use these metrics to get a basic idea of what is working and not working. These metrics also allow you to translate how IT is performing to the other C-level people and stakeholders in a language they understand.
I think its more general than that applying to most companies. I was trying to get to the idea that the IT BSC is for the company’s executives and thus its important to know that the metrics used will be of interest to them. For instance % uptime, or number of patches means little at an executive level (although necessary within IT).
1. Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
There is so much interest in measuring because it is a flexible tool that can measure the success of IT in different ways. Measuring can provide a visual to management about the progress of a project, the value created through IT, and how IT is helping business achieve its goals (effective communication, gartner). In a nut shell, the purpose of measuring is to show management the results of IT in the organization.
It is overkill to try and measure everything. If a company is measuring everything it really isn’t doing a good job prioritizing what is important. It takes time and effort to measure things, if everything is being measured it will just use up more resources than is needed.
I would want to measure what is the most important to the firm. What are the business objectives? I would build a measure based around business objectives to show management what IT is bringing to the table. Another good measurement would be around cost savings, how is IT reducing costs? and how much? Measurement can also be used to show progress for a project, keeping management in a loop on how well the project is doing and what can be improved.
Ahmed,
You almost said it but didn’t. Think continuous improvement. Can IT figure out what its stakeholders think defines a quality IT organization and then measure those attributes?
Stars Ambulance Case: Take 2
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and bring it to class (Jan’s Section) or post it on the class blog (Rich’s section).
As the CIO of Star Ambulance, I would want the following BSC metrics:
Financial Stewardship:
– Percentage Decrease in Consultation Costs
– Savings % of a centralized IT budget
Internal Business Processes:
– System/Network up time %
– Trouble Call round trip completion time
– # of TC’s completed in first attempt
– % of security issues stopped at prevention compared to detection and remediation
Organizational Capacity:
– # of projects completed
– # of Project proposals aligned to business strategy
Customer/Stakeholder:
– Customer Satisfaction Rating for delivery of service(s)
Sean,
This is a very good list
I would like to add the following to the following categories:
Financial stewardship:
– SLAs met – percentage of key jobs that finished on time
– IT Cost Variances (operational & project)
– Business Value of IT (revenue growth, cost reduction, cost avoidance)
Internal Business Process
– Additional Security Metrics ( intrusion attempts and thwarted)
– Outages affecting critical process (i.e., outages of communication system at the ELC)
– PMO – timeline, milestones, budget, scope, and quaility
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
I would start by keeping all the metrics that is based around business objective. These would be vital as it shows how IT is meeting business objective and providing value to the business. Irrelevant metrics would be removed. I would also interview management and get an understand on what their needs and wants when it comes to BSC, and what they are expecting. This would further help identify which metric to keep and which to remove from BSC. Using these strategies will help prioritize between the metrics from most important to least important. The least important metrics can be removed if management agreed and maybe delivered in some other method.
I think you touched on the two types of metrics to contain on a BSC. First, the BSC should have metrics contained that shed light on the value added by IT, and that the projects are aligned with business objectives. Second, the metrics should also be what management considers important so they are more receptive to the projects and likely to get behind the projects in a stronger manner. The key is probably finding the balance to meet both needs while not overlooking the other.
Ahmed,
I think that you hit the types of metrics pretty well. I believe consideration to the audience is also important when creating a BSC. Who is the BSC intended for; Executives, middle-management, IT Staff, etc. The lower-level units would have more use for specific metrics like what has been accomplished and what needs to be achieved, while upper-management might seek correlation between business objective and IT value-added. If you had too much information on the BSC, an option is to tailored it to the intended audience(s).
Additionally, often times I find that lower-level employees are the person that are deriving the metrics and really don’t know if it’s really being used or if it’s adding any value for decision makers. All they do is provide reports, which costs money and time. Like Sean has eluded to, it is important to get feedback for the BSC. Is the metric understood, is it useful, is it relevant, and ,sometimes, if it’s even used.
Great comments.
In addition, i think having more metrics than can fit on one balanced scorecard might not be such a great idea. Agreed that in the process of developing the BSC, an organization might go overboard, but these metrics can be prioritized.
Using the STAR ambulance case as an example, Khans phrase after identifying 24 projects currently underway with many of these projects cited as being critical, his comment was “They cant all be Critical!”
So I would think prioritizing,(and of course taking into consideration alignment of business values with IT) would be a great starting point. Put the metrics on a scale of importance and start with the most important and relevant to the business.
Also citing a reference from one of this week’s reading – – “Effective communications:Performance Dashboards”, it is advised that when selecting a BSC method that works for your organization, remember that less is best. Select a few key measures that are meaningful to the enterprise and that help demonstrate what IT is doing and where its resources are focused on.
Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
The main aim of organization is to meet its objectives. It is important to measure the effectiveness of organization towards meeting its objectives to ensure productivity and continuous growth with minimum waste. To understand current profile it is a must to measure everything,like performance, growth, finances.
1. Assess current profile – Measuring will give a sense of where organization is currently and how far is it from the target profile.
2. How far are you from objectives? Measuring of performance, progress will determine the next plan and steps to be executed.
3. Total investment – Measuring gives the facts about time effort and resources executed for completion of tasks. This becomes a recorded fact sheet taht can be used in other and future projects.
4. Future investment estimation. Measurement will help estimate how much time, money , effort and resources must be invested to reach to determined objectives.
5. Understanding critical components based on
6. It helps in identifying factors that are driving under performance and eliminate them. Also, it will help identify factors that are a boost. This will help to focus on only important factors.
7. Measurement will help discover areas of improvement
Would it be an overkill?
It would be an overkill sometimes if not restricted to what is important. And that is why there must be a measuring factors and documenting them prior to beginning the process. But going through the measurement process is extremely important to ensure business growth. It is important to clarify what will be measured if not everything, depending upon objectives, future plans, audience and stakeholders.
What would I want in my organization?
I would to make strategic directions, establish goals, execute decisions and monitor their state and behavior as the company would move towards its goal. I would want to first decide what is to be measured. There are various criteria to decide what needs to be measured. Even if it is a extra effort, I would want my company to know where we stand rather than not knowing what stage and state we are in.
I would address measuring in following way –
– Define what performance is, define what we have and what moves will take company forward.
– Define factors what will lead to measured value outcomes tomorrow.
– Differentiate between types of measures ex. Objective measures can be independently measured but subjective cant be.
As defined by Vince Kellen, measures are also typically classified as,
Objective / subjective
Financial / non-financial
Lagging / leading
Complete / incomplete
Responsive / non-responsive
Inputs / process / output
Critical / non-critical
Tangible / intangible
I would establish related measuring methods to save time and energy as one measuring method rule may not apply to all.
Nice listing, Priya!
I would like to add something on the question of “would it be an overkill?” If we measure everything, it will be wasting time for measuring something like employee motivation. So in order to save time and max productivity, it is very important to choose the right metrics in an organization.
Pray,
Your list is real good and give many detail about the measuring. I think that measuring can tell us where we are now, where we should go, and what risk we may meet. It is a good method to evaluate company and improve company.
For the overkill part, I think documenting them prior to beginning the process is real good. However, I think improving through the process may more important than beginning. At the beginning, we may just give a brief structure of the measuring without more specially details, which just like constitution, because it is easy for us to find the problem and add some thing in it to make more suitable for our company. Time changes constantly and employees also change from generation to generation. Different times and generation need different method to monitor. Thus, I think making a good plan or documenting measuring is real helpful and important for the company. Meanwhile, we also need pay more attention on the process of using them, evaluate them, and improve them.
If you were a CIO, what metrics would you want? How many is reasonable to have?
If I were a CIO, the following metrics deserve high attention:
Self-improvement metrics – in the current climate of rapid technological change, it’s becoming necessary for knowledge workers, especially IT specialists, to be in a continuous learning mode to fit in new trends and tackle emerging threats.
Performance metrics – assist management in knowing how well business is running and whether its products and services conform to the mission.
Customer satisfaction metrics – poor customer experience would drive them to seek for other suppliers that will meet their needs.
Financial metrics – gain a picture about company’s finance situation
Too many metrics would create information overload, therefore, the total amount of metrics could be limited to somewhere between 10 – 15, or three to four metrics for each of the four perspectives.
Ming,
No doubt customer satisfaction is important but it is more complex that just the end users. Who is IT’s customer? Is it the end users, the lines of business, or the corporation. I believe its all three. Think of email, suppose IT outsources email to Goggle’s Gmail. The end users may want something fancier, the lines-of-business may want something more integrated with their special needs but the corporation sees value in a single global solution. Who’s right? If you measure end users and satisfaction is low should the company do something else?
Your questions highlight the need to agree on what drives the business’s core functions. If the end-user dissatisfaction grows to the level where it’s affecting the the core, then you could argue that perhaps a change is necessary. If not, then it makes more sense to stay the course with Gmail.
Ming,
I think customer satisfaction can be viewed as tangible or intangible. By doing customer survey, I am sure we can see some results. But you can’t one hundred percent sure what is happening. I think an organization can use other metrics to know how the business doing now. Customer survey can be an add-on measurement.
2. If your were a CIO, what metrics would you want? How many is reasonable to have
According to COBIT 5: Enabling Processes, there are 2 metrics categories: The Enterprise Goal Metrics and the IT-related Goal Metrics.
If I was a CEO, the metrics I would want (IT-related Metrics using Cobit 5 as a guide) would be
– Financial
Realized benefits from IT-enabled investments and services portfolio
– Customer
Delivery of IT services in line with business requirements
– Internal
IT compliance with internal policies
Delivery of programs delivering benefits, on time, on budget, and meeting requirements and quality standards
Availability of reliable and useful information for decision making
– Learning and Growth
Knowledge, expertise and initiatives for business innovation
I think the number of metrics reasonable to have might depend and vary. But the number should be reasonable and achievable and shouldn’t be too high to be able to keep focus
I like how you added Learning and Growth as a metric to a list you would have. A lot of companies probably don’t consider this as part of their BSC. At my company, every quarter a survey is released to all the employees. In order to build expertise in specific areas, we grade our knowledge in those areas. If we are lacking in a certain area, we try to improve ourselves. All this is aggregated and reported to management to see if there is any improvement to our knowledge over time, and management will provide training to targeted areas. This is a great way to build SMEness and use the Learning and Growth metric. In the end, its not about just tracking a metric, its about how can we use this metric to improve our company. Measurements need to lead to action, otherwise you are just looking at numbers.
Ahmed,
Great comments, actionsare needed not just words. But actions cost money (usually) so an organization needs to know that it is spending where it will reap rewards. If not, it can lead to actions that generate costs with not payback which is bad for business.
If you were a CIO, what metrics would you want? How many is reasonable to have?
CIO is the most senior executive in the enterprise responsible for managing information security and computer system that support enterprise goals. For this the main focus of a CIO will be in the metrics which supports IT functions.
Following are the metrics which a CIO would want:
Velocity of idea-to-offer cycle: This is to measure the number of releases per year per application.
Quality of idea-to-offer cycle: This is to track the quality by team/project managers to find ways to learn from their best performing assets.
Business Value: Helps analyzing organizations need to measure Business Value in terms of three key performance areas: Revenue Growth, Cost Reduction, or Cost Avoidance.
End user Satisfaction: There are a number of ways to measure satisfaction with IT: surveys, focus groups, etc. Depending on the need of the organization, CIO can pick one.
IT Financial Maturity: This helps CIOs to run a value driven organization that aligns IT spending to corporate objectives and efficiently manages Capital Vs Operational spend.
Security: Security metrics helps in understanding total number of network intrusion attempts thwarted per year, number of security breaches (if any) by extent, etc. which in a way helps in assessing the enhancement needs.
Business Alignment: This allows CIOs encourage to track, the time spent by their direct reports “in the field” talking to their respective business owners versus “office time” focused on internal activities.
Critical outages impacting business: This provides details on what constitutes a critical outage, based on either % of revenue impacted or % of employees affected.
There is no specific answer to how many is reasonable to have as it depends on the need of the organization and their business processes. To make a decision on what and how many metrics are needed, it is important to understand the business of the organization and the customer’s needs as well as demand.
Deepali,
I like the security metrics you present, they are more business oriented than most. I’ve some questions about Velocity and Quality of Idea-to-Offer. This seems to be a new way to look at IT based projects which are usually viewed as “on time, on budget, on scope on quality”. My question is, should an IT organizational always have important business projects or is there a point when a business’ IT systems are good enough? If so, what would this look like?
Professor,
I am not able to understand the question. Can you kindly elaborate on this?
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
I think measuring is an important tool and rightfully so should be a major interest to organizations. I look at measuring from a quality perspective and can understand why there is so much interest in it. Measurements and metrics yield continuous improvement to products or services rendered. Organizations that measure different aspects/functions of their business objectives put themselves in good positions to deliver quality products/services. Looking at measuring from an IT perspective, I would say the interest in measuring comes from IT organizations wanting to receive information in the form of metrics, feedback etc. in order to enable them to detect service defects, correct them, and avoid them in the future through continuous improvements. In summary, interests in measuring is to enable organizations to provide better quality.
I necessarily wouldn’t say it is overkill to try to measure everything. I don’t believe there can be such a thing as too much measuring when it comes to getting the information needed to know exactly what your customers/consumers want and transferring that into your product/service (i.e quality). This can mostly be achieved by measuring. I think most businesses would rather prefer to over-measure, learn from past metrics/measurements, and then iterate to get the adequate/exact amount of measuring for future measurements than to under-measure, which could in some cases prove costly to their business.
It’s simple! – just with any other reporting tool used by organizations to gather metrics, they would want to measure aspects they care about or that provides the needed information. A retrospective should also be used to decide what should be measured. Organizations should be able to look back at past measurements to tell whether or not they are necessary and helpful to be included in future ones. (i.e did past ones provide answers to questions that were being asked?).
Alexander,
I suspect that you would agree that there are some things that are too silly to measure in a business. FOr instance there is the absurd,( ie counting how many left-handed people we have in each department). To the extent that each department has key performance indicators (KPI’s) that are aligned with that department’s mission to add value to the company, then the measurements make sense. When TQL first came to companies many over measured. They tried to look at and improve everything all at once. Hence they didn’t maintain any focus and ran up costs without seeing benefit. I like you last paragraph a lot. Departments learned to simplify and focus their measurements on what was important (ie aligned and impactful) and continuous improved really got rolling..
Prof. Flanagan,
So I would never have expected things like that to be measured, especially not in business. But if we are speaking in plain/general terms, then yes, it absolutely is absurd. But I understand how using this as an example explains how businesses measured every and mostly things that didn’t matter. Just like you have stated, I believe it’s key for businesses to understand that everything cannot be measured especially as they continue to improve because some metric will be deemed irrelevant to certain aspects of the business that is being measured.
Also, a key point you raised, which at the time I didn’t think of is how measuring almost everything will cause businesses to lose focus, thus possibly neglecting to measure what really matters most or failing to give them their deserved significance. I understand this can be dangerous to a business.
Thanks!
2. If I were a CIO, I would want these metrics to manage the IT within our organization:
Alignment of IT investments to Business strategy: if the IT strategy and the business strategy are not aligned and linked, we can not deliver sustained business value. We have to achieve business value goals by growing revenues, improving margins through greater efficiency, improving quality of service, expanding strategic partnership, and expanding its customer-facing applications.
IT Spend ration: I would also focus on the total IT spend ratio, many IT organizations find themselves each year of increasing amounts of the budget and leaving less and less to spend on new initiatives. Measuring and reporting IT spend ratio can be a key indicator of both the efficiency and IT as well as IT value creation.
Critical business service availability: this focus on the customers of IT and their satisfaction with the services IT provides.
Operational health: this focuses on operational health and stability, and measures the variability of performance and availability.
I would choose these four metrics that are relevant to the business and have the most impact on business outcomes. The four that i choose meet the criteria for relevance and impact are investment alignment to business strategy, business value of IT investment, service level excellence, and operational excellence.
Yulun,
I like your addition of aligning IT investments with business strategy. Without this metric, how do we know we are meeting our stakeholders needs? How do we know we are continuously improving our business value?
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
As a CIO, I will want metrics that proves and demonstrates the importance and value of IT to the business. These will be metrics that will show functions of IT aligning with and meeting the business objectives of the organization. I would want metrics, for example, that track the efficiency of IT-related activities and the effectiveness of IT’s contributions to the organizations goals. As a CIO, a couple of these metrics I will have in mind are:
— Average time for IT (i.e help desk technician) to respond to (properly route) user requests/tickets
— Average time for IT specialists to properly resolve L3 user issues
— Dollar value of reductions in IT maintenance costs (p.s: this translates into how much money the IT organization is saving the company)
— Time taken to implement and deploy company-wide business applications
— Company-wide usage and adoption of IT applications and processes (p.s: this shows IT isn’t wasting company money on the wrong applications that are being bought, and whether or not they are aiding employees in their job functions)
On the question of how many is reasonable to have, I would say as many as possible, as long as these metrics provide the needed information and answers to specific questions that the CIO can use to determine whether IT is aligned with and meeting the business objectives (i.e providing value to the business of the company).
Alexander,
Your suggested metrics are heavy on IT operations and are certainly important within the IT organization. What do you think the board of directors would want to see?
The metrics Alexander listed all could be translated to something that an executive would be more interested in. For example:
– Average time for IT (i.e help desk technician) to respond to (properly route) user requests/tickets could be translated into getting business line leads to quantify their satisfaction
– Average time for IT specialists to properly resolve L3 user issues becomes a decreasing number of critical issues interrupting business functions
– Time taken to implement and deploy company-wide business applications translates into how these projects are adding to the business functions
Prof. Yeomans,
If I clearly understand where you may be headed with this, I now see that the board of directors of a company may not really be interested in those IT operation-related metrics, which may be more for VPs and IT directors. Perhaps the board of directors will want to see metrics related to things like business value (i.e cost reduction, revenue) and alignment of IT processes with the business objectives as well as customer satisfaction.
Are there specific metrics you are looking for that I am missing out on?
The Star Ambulance Case: Take Two
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and bring it to class (Jan’s Section) or post it on the class blog (Rich’s section).
Below are some of the metrics I might use as a CIO, depending on the situation.:
Start Ambulance Score Card
Financial
-IT Project Budget Estimate vs Actual
-Cost savings by decommissioning redundant apps and unifying IS
-Measure showing costs saved by relying less on Consultants
Employee
-Contractors to Employee ratio
-Training metric for Employees
-Employee satisfaction to measure morale
Customer
-Customer satisfaction rate
-Ontime helicopter rate
-Measure displaying requests vs capacity
Information Systems
-Metric showing projects delivered
-# of applications available
-Measure that shows time spent on maintenance of apps
Ahmed,
Great examples. This made the BSC make a lot more sense to me.
Jan Yeoman’s Section
How can measurements become obstacles to change?
I think measurements can be obstacles to changes in two ways. The first way being that an organization is reluctant to take on changes in order to maintain their current measurements or KPI’s. This it to say that an organization might not want to set up an expensive ERP system or overhaul their shipping process, because it means that the KPI’s their shareholders utilize will become more negative. While I am unaware how often that is the case, I do think that an organization’s leaders might be reluctant to take on certain changes if it means that future financing might be affected.
The second way I believe measurements are obstacles to change is in the way projects are supported within organizations. In order to get initial support for a change, this change must meet certain expectations. Areas such as cost and ROI are two measurements that a project must fall within a certain threshold to even get the initial support needed for change. This obstacle doesn’t just apply to getting the initial support for change, but a change must meet measurements within the first couple of periods or face the threat of having the change discontinued. Therefore, it becomes crucial for a project to meet those pre-determined measurements which ultimately places extra pressure on those implementing the change. While it makes sense that measurements are used to make sure that organizational changes are well thought out and are controlled correctly, I do think that this makes it overall a more difficult process.
1. Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
Measurements provides for a baseline to analyze success/failure and under-performing/outperforming in relation to its desired outcome. In simplest terms, how can you tell if your company is making a profit from a product if you don’t have measurements like cost of goods sold (COGS), Price, and Revenue. In IT terms, how can you tell if a project is a success if you don’t have measurements that reflects the scope, budget, time, and quality of a project? Measurements gives you a baseline to measure progress against. Measurements can instill confidence in stakeholders and provide analysis for business leaders to make decisions.
It is definitely overkill to measure everything. Decisions on what to measure depends on the company and the maturity of the IT organization; operational metrics for less mature organizations, and IT to business alignment metrics for more effective IT organizations. Metrics should be based on the IT and business strategy. Each business or IT goal should have a well-defined and agreed on metric. For example, when eliciting and defining requirements from stakeholders, a performance metric must also be defined and agreed upon by the stakeholder and IT. This enables IT to know if they are meeting the stakeholders requirement and allows the stakeholder to know what IT is supposed to deliver.
Loi Van,
I really like your questions, which help me think a lot about measurements and get the same idea with you that measurements are really important in some areas. They help us know our company well. Before we know where are we and who we are, we can not find out what should we do and how to improve. In the other words, measurements help us find our position in the company or in the market. Showing our advantage and disadvantage. This is the reason why more and more company like using measurements to lead people know how good we are.
It is true that abusing measurements cause many serious problems, which can kill employees’ enthusiasm and creativities. I think your method it real good. We should always connect IT with business rather than separating business. They should work together. Sometimes I think our company should pay some attention on future. Although, stakeholder’s requirement is very important and necessary for IT, sometimes stakeholder does not actually know what they what. In this case, we may pay more attention on the future.
1. How can measurements become obstacles to change?
Measurements can become obstacles to change if actual measurement are effective and satisfactory. In fact, if a specific measurement measures effectively a given outcome and provides good analytical tools to decision makers, it will be hard to implement changes as there are no incentives. And as we say, “never change a winning team”.
Plus, I think that the real obstacles to change are people. In fact, they would not change anything if what they have implemented is working, or if they think new measurement will take time.
Said,
What about a situation in which metrics consistently show the process being measured in in control, and people become complacent? Is what is being measured is still the right thing? Is the value IT can add to the organization still being optimized, or have things changed?
Pr. Yeomans,
This is where people satisfaction come into play. If they are satisfied with the metrics as its given them good analytics tools, they won’t think of implementing new metrics. Also, I think it is a matter of point of view. If what being measured is seen as satisfactory it will be seen as the right thing. Change will only come if it is not working anymore.
And it will be IT role to show and prove that things need to change.
Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
The interest in measuring the efficiency of IT, and the contributions it provides to the organization goals is at such a high level because businesses are trying to think of creative ways to determine the ROI. There is no debate over technology increasing production capabilities, but the question is, “Has our investment in technology reached the maximum value, and now producing diminishing returns?
The process to measure IT takes time, money, and human resources to complete, and because it is such a new form of practice, the costs of the procedure may outweigh the benefits of the measurement. Measure your entire IT infrastructure would be a waste of time, and can’t see how a such a project could be justified to top management.
The balance score card is a nice tool for measuring IT in an organization. It is still in the early stages, but is more flexible compared to Applied Information Economics & Earned Value Management. The BSC measures IT value, and the governance policies.
In my opinion, IT will soon be viewed through the eyes of a LEAN / Six Sigma point of view.
If your were a CIO, what metrics would you want? How many is reasonable to have?
The two metrics I would want on my balance score card are the Reliability and Functionality. My most important question to answer, Is everyone / department using the IT in the right way? Are they utilizing all of the features in a way that makes the company better with the technology? When we answer this question, I would want to know if it reliable. Are the features and capabilities fully operational 24×7?
These are leading questions to, Alright, “How much is it costing the organization”. The idea behind measurement is to understand the value in the investment. How does it work, and how can we use it still be one of the first things I would want measured?
I believe three metrics are reasonable, that are based around the above questions. They would include Employee, Customer, & Financial. I would throw in a 4th for projects.
Fred,
When I read your comments I think you are talking about personal productivity, is that right? If so, I think you are missing the vast majority of significant value that IT can help create in a company. Its not all about making people faster and more productive by using better tools, much more value comes when organizations change their processes and structures to eliminate work and people. Thus it is possible for a new system to make someone’s job much more time consuming and tedious and yet, that change eliminates the need for another person making it very valuable indeed. Therefore, value measurements need to be at the business level.
My favorite example is Campbell Soup, who for awhile measure all IT projects and operations on the basis of average cost per case at the customer’s location. If changes reduced that cost, they were on the right track. If not, they needed to stop.
Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
In the event I had too many metrics, I would see if in some way I could consolidate the measurements by using information from other metric results (2 for 1). The important thing is to now get too carried away with measurements. The process can produce great results and identify issues that may cause issues in the future, but there should be a balance between cost and benefit.
How can measurements become obstacles to change?
One of the most difficult things to do is convince a successful business professional that there is a better way of doing the thing they have become successful at. This is especially true when the measurements are not producing obvious results for change. The problem is that sometimes by waiting to implement modern technologies, you put yourself behind the competition, and with the learning curve, it could put you in a “catch up” position for several years. This is what we will be doing when pitching an idea to top management. The best thing to do is to show value. Focus on the current metrics and “guesstimate” how your change will make the metrics better.
Fred,
I like your answer; it is a very logical explanation as to how measurements become obstacles to change. A successful business staying the course may become complacent based on their return results. Their current measurement results may not show a dramatic need to change or reposition of its business processes. It is vital to keep continuous improvement in mind based on what your stakeholders deem to take precedent.
I agree with Fred. If I am doing a good job in coding and productive for an extended period of time that i will have the mentality if its not broken than why fix it or why change a good thing. I think management has to show me as an individual why and how there is a better way. If its just being preached and not indiviualized than there is no personal element. As a good employee, I would think of they are not talking about me just other less productive employees.
Also I have seen where if management puts in metrics to monitor employee breaks, lunch time, smoke breaks, internet\phone usage than the employee feels micromanaged or under the microscope. Its a no win situation but management has to be behind this change and preach on why it is and will be used.
Some of the measures being taken in my organization are following HIPPA regulations, C-II Drug logging, secured logins and insurance regulations. In the pharmacy, we have to follow HIPPA because of sensitive information that we don’t want unauthorized people to misuse such as name, insurance info, address, etc. Although we may know the patient themselves, we may not know their family members so we can’t divulge information unless we get authorization from the patient themselves. C-II Drug logging is another measure because the medication is expensive and can be used for illegal purposes in wrong hands. We don’t just fill it unless we know the doctor and patient. We log date dispensed, qty given, doctor’s name, patient’s name, drug name – this was a fairly recent regulation in pharmacy industry. Some automate it but as a small organization, we do it by hand. Secured login helps with when typing prescriptions and refilling because the system automatically puts our initials which allows for tracking who was responsible if something goes wrong. Of course if someone else is using the system, they can put their initials instead. We have to follow insurance billing rules because we need to make sure what the doctors prescribes and what we bill to the insurance matches otherwise during auditing, they will think fraud was committed and/or take back the money which we billed to them. These are some of the measures my organization uses everyday.
Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
For one thing, there is no one way to establish what a business is worth. That’s because business value means different things to different people.A business owner may believe that the business connection to the community it serves is worth a lot. An investor may think that the business value is entirely defined by its historic income. In addition, economic conditions affect what people believe a business is worth. For instance, when jobs are scarce, more business buyers enter the market and increased competition results in higher business selling prices.
There are three fundamental ways to measure what a business is worth:
Asset approach: The asset approach views the business as a set of assets and liabilities that are used as building blocks to construct the picture of business value. The asset approach is based on the so-called economic principle of substitution which addresses this question: What will it cost to create another business like this one that will produce the same economic benefits for its owners?
Market approach, as the name implies, relies on signs from the real market place to determine what a business is worth. Here, the so-called economic principle of competition applies: What are other businesses worth that are similar to my business? No business operates in a vacuum. If what you do is really great then chances are there are others doing the same or similar things. If you are looking to buy a business, you decide what type of business you are interested in and then look around to see what the “going rate” is for businesses of this type.
The income approach: this takes a look at the core reason for running a business – making money. Here the so-called economic principle of expectation applies: If I invest time, money and effort into business ownership, what economic benefits and when will it provide me? Notice the future expectation of economic benefit in the above sentence. Since the money is not in the bank yet, there is some measure of risk – of not receiving all or part of it when you expect it. So, in addition to figuring out what kind of money the business is likely to bring, the income valuation approach also factors in the risk.
http://www.valuadder.com/valuationguide/business-valuation-three-approaches.html
1.Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Measuring allows an organization to tangibly discover the value being added to the organization. Measurements act as a benchmark to the industry, past performances, and drives future predictions. They give the stakeholders an accurate representation of the organization at that moment, which will affect the value or the organization. In IT, this helps the organization understand the value in a project, or see efficiency gains from a successful endeavor.
It may seen like overkill to measure every aspect of the business, but without measuring, parts of the organization may slip beneath the cracks. This can lead to inefficiencies, fraud, or creation of inaccurate goals or benchmarks.
In an IT sense, I would want to organization to base measurements based on what the stakeholders would perceive as the most valuable metrics. This allows the people in my organization to continual focus on these metrics and achieve a higher stakeholder value.
Kevin,
Keep continuous improvement in mind. These measurements, if well chosen, are the scorecard by which any organization can show that it is improving. Without them, its just all noise.
How can measurements become obstacles to change?
Like others have said, I think that business are making life and death decisions based on the metrics they deem most important. A poor decision to focus on the wrong metric can doom a company to failure.
Prof. Jan’s class
If your were a CIO, what metrics would you want? How many is reasonable to have?
CIOs can have metrics like, with all the investments we made how much cost savings we were able to get in our operations. Then how much did we work with the business community to produce more opportunities. They can also have metrics (KPIs) like, what is my uptime (e.g.: server was down a day). Such KPIs may not be relevant to the core of the business, like, it may not attract a new customer but what if when the infrastructure went down, it also irritated a few customers; so it is indeed important.
The first set of metrics can be directly aligned with the operation metrics. For example, SLA, number of tickets generated and solved, downtime, available time, etc. These are the metrics that are directly relevant to the availability component of the CIA triad. Second set of metrics can include customer satisfaction, contractual obligations, etc. I think the most important is key process improvements facilitated by the IT. For instance, if last year it took is X amount of money to support our business activities and this year, through automation, process improvement and digitization I paid Y amount, so it’s also a productivity measure; these can be hard to measure.
As there are a very clear targets and operating expense for any CIO, a CIO should ensure that the value proposition is clear. For instance, if there is an economic meltdown, CIO needs to make sure that he/she is communicating as to how IT can help restructure; how IT can help the organization get through it.
2 If you were a CIO, what metrics would you want? How many is reasonable to have?
2 As a CIO, the metrics I would want, to better understand the state of my IT systems, enable me & my team to make better decisions and fulfill my role to the best would be :
• Customer satisfaction
• Average time to attend to and resolve IT issues
• Compliance metrics (such as patching activities)
• # Security incidents and their resolution
• Service Availability or System uptime & downtime and business impact
• New Project completion rates and their KPIs
• Incident, Problem and Change Management metrics
• Metrics that would directly tie in with the company’s Business goals like Productivity and Resource Utilization
I believe that it would be reasonable to have 8 or at most 10 different metrics. Having too many metrics could distract employees and take their focus away from critical metrics. This way, unknowingly the importance of critical metrics would get diluted. This could lead to adversely impacting decisions and eventually, Business results.
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
Ans. Performance optimization is the process of improving service performance along with improving productivity of information system to the highest achievable level without additional costs. For effective performance management measures are used for assigning accountabilities, comply with the requirements and create action plans to improve performance.
For effective performance the performance goals must be defined and effective metrics should be in place to monitor the achievement of these goals. Balanced scorecard is the process management evaluation technique in accessing the IT functions and processes. Balance score card uses four perspectives i.e. Learning and growth, Business process, financial and customer and develops metrics for each of these by collecting data and analyzing the data relative to them. KPI (Key performance indicator) is a measure to determine how well a process is performing in enabling to reach the goal proposed.
Defining and prioritizing these metrics is very important and should be done to achieve the business objective. A decision can be made by:
1. Listing all the important performance indicators aligning directly with business objective(Business process)
2. Access the short term and long term effects of these indicators to the business.
3. Determine which is more important: short term or long term goals by accessing the capital available(financial)
4. Evaluate marketing strategies using these indicators, employee efforts and need for training. Take feedback from the management as well as employees.(Customer)
5. Reevaluate the KPI every six month to be sure if it is still aligning to the IT strategy. (Learning and growth)
Q2. If your were a CIO, what metrics would you want? How many is reasonable to have?
I think what metrics should depend on which industry my company on. We can make assumption I am the CIO of a retail chain, I would chose the metrics as following:
1. Customer Traffic: number of customers is the most straightforward metric for you retail business. Even a child gets that the place that’s crowding with customers must be doing good. You normally don’t go to an empty restaurant, don’t you?
2.Retail Conversion Rate: we already had to distinguish retail visitors and retail customers. Some visitor don’t buy anything.
3.Average purchase value: how much cash flow we got.
4.Gross margin: rule of thumb is to set the gross margin high enough so you have plenty of room to cut back. Even a successful retail business will have some goods that are harder to sell.
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
Metrics are important for IT shops that hope to achieve organizational goals. However, they can be dangerous, and using the appropriate metrics is critical.
IT success is achieved by excelling in five areas: running the production systems; delivering new functionality; having an engaged workforce; keeping costs in line; and exploiting the capabilities of new technology. These five areas can be summarized into four categories. So if I were a CIO, I will choose metrics for this four categories:
• Operational metrics: 1. Online application performance 2. Online application availability 3. Batch SLAs met 4. Production incidents 5. Supplemental operational metrics
• Delivery metrics: 1. Project satisfaction 2. Project delivery 3. Project cost 4. Defect containment 5. Supplemental delivery metrics
• Organizational metrics: 1. Attrition 2. Performance reviews 3. Supplemental organizational metrics
• Financial metrics: 1. Budget variance 2. Resource cost 3. Supplemental financial metrics
Link:http://www.cio.com/article/2955777/best-practices/12-critical-metrics-for-it-success.html
Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
I would start by re-evaluating all the metrics and identifying what performance measure each metric measures.If possible, eliminating the metrics that measure the same performance and picking the ones that are aligned with the organizational IT strategy. This also involves meeting up and interviewing upper-level management to determine which performance measure should be prioritized and if the metric used are properly aligned with the organization goal.
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Wikipedia describes performance measurement as “the process of collecting, analyzing and/or reporting information regarding the performance of an individual, group, organization, system or component.” The evolution of the Balanced Score Card (BSC) from a tool to measure financial and non-financial metrics to a method for consistently measuring IT performance was a natural progression due to the lack of an evaluation technique that could demonstrate the value of IT to an organization.
There is no need to measure everything. In fact, guided by the “less is best” principle, it is recommended that organizations select a few key measures to report on rather than diminishing value by presenting too much information or that which is not considered meaningful to the overall enterprise.
In order to decide measures to include in a BSC, I would recommend that management examine its strategic objectives and determine which are most critical to the organizations success (e.g. improve customer satisfaction), determine which IT systems and processes are critical to achievement of goals underlying this objective (such as customers’ ability to connect to the internet, watch TV, and utilize their wireless phones), and then design measures around the availability (or lack thereof) of these services during a specific time frame, e.g. monthly.
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
The metrics I would want to have in a Balanced Score Card (BSC) if I were a CIO largely depends on the nature of my business, its stakeholders and alliances; the type of products or services provided; the customers (internal and external); the organizations reliance on IT systems, functions and services; etc. As I was recently employed by a clothing retailer, I will answer this question from the perspective of that organization.
• Compliance with PCI standards was paramount, and the company either conducted or outsourced the performance of routine penetration and vulnerability testing to demonstrate that customers credit card information was adequately protected.
• A third party performed annual scans of the network and shared drive folders to identify customer, employee, vendor, etc. proprietary information (including credit card and social security numbers and birth dates).
• Independent self-assessments of IT general controls were performed internally on a quarterly basis.
• A Privacy Committee was formed when a gap assessment and process audit detected opportunities for improvement. The Committee met weekly to discuss the implementation of corrective actions.
• IT audits were performed by third party specialists, results were communicated in writing to the Executive Committee downward based on their nature and severity, and their status was monitored jointly between Internal Audit and IT.
• Since the company outsourced manufacturing, finished goods were received at the warehouse daily, and they were sorted, packed and shipped – all automatically – to the appropriate stores based on forecasted demand within 24 hours of receipt.
• Store Operations were dependent on the functionality of their Point of Sale (POS) systems in order to sell merchandise. If the systems were down, they had to resort to cash purchases only, which negatively impacted sales, customer satisfaction, and potentially customer retention.
• Sales were polled nightly from the POS system to a staging system where automated integrity checks were performed (including gaps in transaction numbers and stores with $0 sales), after which sales transactions were interfaced to the general ledger.
• The Help Desk was on-site, services were provided by third parties, and the process was overseen by IT operations personnel.
• There was a Capital Expenditure Committee that determined which IT initiatives would receive funding. In some instances, decisions required more information, e.g. ROI, and the project would be pending.
• Pilot programs (e.g. Omni Channel Selling, Customer Loyalty Programs) were often conducted at a select group of stores to determine their viability before investing in equipment and resources to roll them out to the entire chain.
• IT budget vs. actual financial reports were generated monthly, and more scrutiny was placed on IT financial performance quarterly in light of external reporting requirements.
Each of these bulleted items represent one or more of the measures I would include in a BSC. Using this example, I believe it is reasonable to include between 8 and 12 measures.
Candace,
Great breakdown and use of real world experience! I am interested in the measure around IT audit. Can you shed a bit more light on that as a measure.? Do you know the details of the measurement and how they would’ve have formulated a baseline?
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
Metrics are very important to making sure an organization proactively addresses symptoms before they become real problems rather than reacting when a problem is real and there is a crisis.
If I am a CIO, I would like to have metrics that focus on the following four categories:
1. Operational metrics
a. Online application performance- the average time it takes to render a screen or page. It is also important to measure the variability of performance.
b. Online application availability- the percentage of time the application is functioning properly.
c. Batch SLAs met- the percentage of key batch jobs that finish on time.
d. Production incidents- the number of production problems by severity.
2. Delivery metrics
a. Project satisfaction- the average score from post project surveys completed by business partners.
b. Project delivery- the percentage of projects delivered on time.
c. Project cost- the percentage of projects delivered within the cost estimate.
d. Defect containment- the percentage of defects contained to the test environments.
3. Organizational metrics
a. Attrition- the percentage of employees who move to other jobs.
b. Performance reviews- The percentage of employees with current written reviews. Providing employees with constructive feedback is one of the most important steps an organization can take to improve productivity.
4. Financial metrics
a. Budget variance- actual costs compared to budgeted costs. This should be done for both direct expenses (salaries) and inter-company expenses (allocations from other areas).
b. Resource cost- the average cost of a technology resource. The average cost of a technology resource.
Source:
http://www.cio.com/article/2955777/best-practices/12-critical-metrics-for-it-success.html
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
I would need to understand why there were too many metrics to begin with. If the Balanced Score Card (BSC) was developed appropriately initially, there should not have been to many metrics.
Let’s assume; however, that there were organizational changes (e.g. a new product line, or newly outsourced services). In this instance, there would need to be a reprioritization about which existing measures to keep or replace with new ones. If the BSC is being utilized as a management tool, decisions of this nature should not be made by the CIO, or even IT alone. I would identify stakeholders throughout the organization and seek their input as to which metrics are useful to them and why. If this approach produced consistent results, the metrics to exclude could be readily determinable.
Another approach would be to develop different BSC’s for different functions and stakeholders, e.g. have separate BSC’s for systems development, the Help Desk, and computer operations.
Thats right Candace, i think it makes sense to develop different BSC’s for different functions and stakeholders
Also, while doing my research for this week’s topic, I read that the Working Council for Chief Information Officers did an extensive review of IT scorecards and found that the most advanced Scorecards share some basic structural attributes one of which is Simplicity of presentation. The very best scorecards are limited to a single page of from 10 to 20 metrics written in nontechnical language. (not really sure about that though, is there any standards/ guidelines on how many metrics should ideally be on a Score card?) Another attribute is that the scorecard should be tightly coupled to the IT strategic planning process and assist in tracking progress against IT’s key goals and objectives.
So, having more metrics than can fit on one balanced scorecard might mean an organization is working on areas that don’t need improvement and in the process ignoring/postponing areas that do.
Again, In this instance, you comment “there would need to be a reprioritization about which existing measures to keep or replace with new ones” would be applicable
4. How can measurements become obstacles to change?
It is conceivable that consistent achievement of favorable IT Balanced Score Card (BSC) results could cause complacency. However, if there is a process in place to routinely re-evaluate scorecard metrics (e.g. every one to three years, or when there are changes to the strategic plan), this is unlikely to occur. Incidentally, another method to routinely measure and respond to the effectiveness of IT BSC reporting is if it is included as a performance management objective of the CIO.
In light of the rapid changes surrounding IT systems and processes, the heavy reliance that is placed on the fitness of IT products and services, and the ever-increasing computing demands, I do not envision significant risks associated with IT performance measurements becoming an obstacle to change in this day and age.
Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Monitoring and measuring are important aspects to IT governance. It is critical for the IT department to provide value to the overall business which is usually measured in efficiency of IT activities as well as the effectiveness of contributions towards the overall organizations goals. In many companies it is critical that IT and the core business initiatives are aligned and working towards a common goal. Monitoring and measuring are essential elements of measuring the success of an organization as a whole, but also down to the individual project levels. You need to be sure that the limited resources that you have are working on the most mission critical projects as well as the efficiency of such projects. The most common way that this is accomplished (based on the readings) is the BSC. The challenges related to measurement of activities are numerous but some include communications between IT and other departments so they are aware of the value that the IT organization as a whole is creating. This is critical in some organizations just to get the IT group viewed as a strategic partner to the overall business rather than just an operations division the keeps the core business running. Also some of the challenges related to this is trying to measure success which is naturally qualitative in nature but can be hard to articulate the level of value. The BSC allows you to quantify this value which makes it much more easier to identify what you’re group is bringing to the rest of the organization. While some people believe that there’s never too much of a good thing, you need to be very selective in what you’re measuring to ensure you don’t get bogged down into the weeds when trying to analyze the effectiveness. This obviously would vary from company to company depending on how the organization is viewed and what level of strategic partnership they are viewed as from the rest of the executive management. As I’ve learned in my experience in the business world and what executive management wants as far as communication, It is critical to be succinct and address the most critical areas necessary to give the most holistic view of whatever part of the business you happen to be measuring, i.e. telling the entire story. As mentioned previously, this will vary depending on a number of factors mentioned above.
If you were a CIO, what metrics would you want? How many is reasonable to have?
If I were a CIO, I would focus on the following metrics: Financial metrics, operational metrics, and organizational metrics. First of all, the financial metrics includes checking the budget variance, which compares the actual costs with budgeted costs. This allows management directly evaluate whether the expenses are controllable. Moreover, the financial metrics also includes the resource cost. This metric provides a good view of how well managers are controlling costs by using cheaper outsourcing labor, being thoughtful in the use of higher priced temporary labor and managing an organization that is not top heavy with expensive employees.
From the operational metrics’ perspective, the online application performance is also very important. The average time it takes to render a screen or page. It is also important to measure the variability of performance. According to Bob Ronan’s article, management should also notice the importance of online application availability. The percentage of time the application is functioning properly. This can be difficult to define. If the application is available for some users but not all, is it “available?” From a CIO’s perspective, all these metrics are very important for the IT success.
Source: http://www.cio.com/article/2955777/best-practices/12-critical-metrics-for-it-success.html
1. Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
Measuring is a very important aspect of an organizations growth and improvement even personally. I had a long term objective 5 years ago and a short term objective 1 year ago, I take measures occasionally to see if I am on track and will be able to meet my personal objectives, I check where I am doing ok and where I might need improvements to meet/achieve my set objectives.
The same should be for an organization. Measures are like tangible management tools that sometimes lets an organization know whether to close shop or make improvements in its deliverables
Isn’t it overkill to try to measure everything? How would you want your organization to decide?
In reality, not “everything” can/should be measured. In addition, too much of anything is not good. Water is good for the body, but too much water will drown you.
I would want my organization decide on measures by analyzing what is important to the business/organization
Again, one of this week’s readings – “Effective Communications:Performance Dashboards” advises that when selecting a method that works for your organization, remember that less is best. Select a few key measures that are meaningful to the enterprise and that help demonstrate what IT is doing and where its resources are focused on.
4. How can measurements become obstacles to change?
Measurements can become obstacles to change when they are used to justify incorrect time periods. What I mean by this is that measurements can make the difference between budgeting and forecasting. Budgeting is adhering to the measurements to determine future decisions while forecasting is combining some measurements over history and outside information to try to see the future of the company. Often times management gets rewarded on how quickly they improve the business, and will use short term success to prove growth. They will look to measurements to prove this. This leaves out the forecast where the decision they just made may lead to negative down the road.
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
If I was a CIO, I would look at metrics for Risk Management, Project Management, Data Center, and Service. Within each area, I would look at compliance metrics, process metrics, and workload measures.
I would use only the necessary metrics and measures required for decision making and for continuous improvement in order to reach the organizations objectives and to align IT with the organizations vision. One reason I would minimize metrics as much as possible, is that metrics drive behavior. Too many metrics or bad metrics could cause bad behavior and unnecessary stress on employees. The number of metrics depends on each area and what you’re trying to accomplish.
Reference:
https://cio.gov/performance-metrics-and-measures/
Definitive Guide to IT Service Metrics, McWhirter, Gaughan
https://books.google.com/books?hl=en&lr=&id=jSRhAgAAQBAJ&oi=fnd&pg=PA5&dq=%22Definitive+Guide+to+IT+Service+Metrics%22&ots=6HJxXXNUPa&sig=R3s6-VffoVpvg_Fmy-wXrk-faj8#v=onepage&q=%22Definitive%20Guide%20to%20IT%20Service%20Metrics%22&f=false
4. How can measurements become obstacles to change?
Measurements drive behavior in an organization. An old example is a call center, whereas performance reviews are based on the number of calls. The person answering the phone, would help them for a couple minutes then drop the call without resolving the customer problem. The persons call statistics would go up, but the customer satisfaction would go down.
Thoughtful measurements should be taken that reflect and align IT with the organizations vision, goals, and objectives.
I definitely agree Jason.
Your example about the call center may lead to an over emphasis on internal performance and a lack of awareness of external factors that also could influence the company’s operations.
The scorecard should be aligned with business strategies, tightly coupled to the IT strategic planning process, and assist in tracking progress against IT’s key goals and objectives. So measurements in this call center example should consider customers and also factor in other key performance indicators, such as competitors or changes to the business environment.
That’s a great example of measurements holding up change.
It’s tone deaf and doesn’t take into account other factors that may be important, like customer satisfaction, as you mentioned. I see the call duration as a good measurement for the manager of the employee, but from a larger business perspective (providing the business is aligned around customer satisfaction) it’s useless for measuring that aspect.
Jason,
It is a real simple but valuable example for us to consider measurements. Measurements are very good because it can help us find out the problem and evaluate employees’ works. I believe that in this example, the purpose for that policies, which evaluating by number of call, is encouraging employees take more phone and help more people to solve the problem. In this case, users do not spend a lot of time for waiting. However, users also cannot get satisfied reply. On the other hand, some people think that we should evaluate employees base on the number of phone call. We just need focus on whether users get help. It also cause some problems. First of all, employees works slowly. Solving three or four questions for one day is more relax than solving eight or nine questions. In this case, users have to spend much time in waiting. During the waiting, users cannot do anythings because they cannot know whether they will be the next person or they will forget about this and lose the connection.
Therefore, I think changing the method of measurements according to the problem which happened in the process can real help company become more stronger.
5.What measures are being used in your organization? Do they make sense?
Measures used in my department are publishing the number risk reviews and meeting timelines.
No, these do not make sense and are counterproductive. One review make take 15 minutes to do and another review may take over a month to do. Analysts are assigned reviews based on their availability and the section of the United States they live in. This has caused unnecessary stress on the analysts and high turnover. Compounding the issue, new software, training time, and travel time have not been taken into account for the measurement.
Jason,
This sounds a lot like when I had to run reports for the desktop group at Comcast, their metric was how many tickets were closed by techs each month. I found this metric to be flawed and worthless. This metric never show the true worth of a tech or the issues that were never fixed. So what suggestion would you make to either get rid or change the metric?
Interesting example Joseph,
My question is, does it mean something to somebody? Just as everybody has different metrics for measuring success or progress, this could be interpreted as simply productivity of the tech support, or simply ticket resolution numbers. I understand your view saying that it might me worthless, but imagine at a executive just trying to capture a big picture, do the really want granular details or just big picture? The truth worth of the tech should be readily identifiable by middle-management or supervisor, that may decide to show the type of tickets, how well the technician handle the problem, or time to resolution metrics. Another thing that you pointed out was should the metric be disposed or changed. I will say it depends, on who still needs it, how/what decisions it is used for, and its really required for the mass.
Loi,
I agree with you, from a high-level view, this may be worth it. I also agree that the truth worth of the tech should be readily identifiable by middle-management or supervisor, but unfortunately half the time that isn’t the case. In some of my experiences, it’s not what you know its who you know. With your point the question should be rephrased as not how you would change the metric, but why is the metric used and who uses the metric?
I would think that that the metric in Jason’s department would be used to gather information for senior management. Without knowing any more information it sounds like it was used for more than just gathering information since Jason expressed the unnecessary stress on the analysts and high turnover rate. I’m just guessing that maybe it’s a departmental workload issue, which if that is the case then the metric would need to be adjusted, would you agree Loi?
Hi Joseph,
My understanding is that the metric is being used to justify the number of analyst in each department. The metrics are used as a major part of the analysts performance reviews.
1. I believe there is so much interest in measuring because, ultimately, the processes, projects, and objectives of an organization have to make sense from a financial standpoint. Ultimately everything a company does has to reflect positively on the bottom line and the best way to tell if that occurs is through hard measurements. An easy measurement might be something like Return on Investment and from this, losses or gains will clearly be shown. Customer satisfaction might not immediately show a profit or loss but negative customer satisfaction will surely lead to losses in the future, hence its necessity to be measured. I believe another reason so many things are measured in companies is that, over time, it is realized that by measuring new aspects of a business, that business can better determine how well it will do financially in the future. An example of this might be when a new CIO enters a company. He or she might feel measuring aspect A of the IT organization is crucial to the organizations’ effectiveness. Some time later, that CIO might leave and a new CIO will bring in new views of measuring performance. As this cycle continues, varying measurements will most likely become embedded within the organization.
2. If I were the CIO of an organization, I would want several metrics in my IT BSC. I would want to ensure, though, that the metrics focused on the following aspects in general: the finances of the company, the internal processes of the company, our customers, and or employees. Regarding finances, I would use the most common metrics such as ROI, IRR, or ROE. For internal processes or audits and/or risks factors, I would use industry standards for passing or failing our audits. Most likely, I would mandate that an audit score, per the industry standard, below ‘x’ would be a failure for us as well. In terms of customer satisfaction, I believe the most important metric to determine this would be repeat purchasing if our customers were external to the company. If customers were internal, I would like to track how often tickets were created to make changes. In my experience, IT departments are doing best if no one is talking about them. When IT gets noticed, something is usually wrong.
3. If there was an opportunity to add metrics to our balanced scorecard at my company, I would simply put a change management-type of process around it. What I mean by that is, I would make sure that the change to the scorecard went through the proper channels and received proper authorization before being implemented. Without questions the first group to sign off on the new metric would be senior management or executive board members. I would certainly want to avoid a situation where a new metric was being tracked and a new area of performance needed to be met without this groups’ knowledge. Next, I would want someone from the finance department to say whether or not tracking a new metric makes sense financially or that the company can actually benefit from this new endeavor.
4. As metrics are implemented into a companies’ BSC and as time goes on, the more these metrics will be relied upon by management to show success or failure. A metric may become relied upon so heavily in some cases that the metric might never be able to be removed or its removal may be incredibly tedious. When a change is suggested and when that change would cause the disruption of that metric or even the removal of that metric, the change may not be able to be implemented at all. The greatest obstacle to change is complacency and overall comfortability with the current state of something. If a company is chugging along and doing well with a certain metric, removing or changing that metric will absolutely be met with resistance.
5. I do not have much visibility into what measures my organization uses to track performance but with a high degree of certainty I can say they focus on some of the areas I described above. The finances, the customers, the employees, and the internal processes are more than likely to be in consideration when crafting metrics. I can say that our customers are internal to the organization so something that determines the likelihood they are to return as customers would not be applicable here. I can say that, as a unit, we get ticket requests for IT services and an expected delivery date is attached to the ticket. If the actual date of delivery is after the expected date, the ticket is escalated to our manager. This, in some ways, determines the satisfaction of customers in that they should expect to receive their requests in a reasonable amount of time.
4. How can measurements become obstacles to change?
As professor Richard has said, measurements, if well chosen, are the scorecard by which any organization can show that it is improving. Without them, it’s all just noise.
Measurements can become obstacles to change if
The right elements have not been selected for review and if the information used to evaluate progress is incomplete, inaccurate and irrelevant to the area being addressed. For instance, in evaluating the effectiveness of training efforts, the number of people being trained is not as relevant as the training they received.
If organizations don’t set appropriate data measures and don’t input the right information consistently, they run the risk of getting inaccurate results. This could prompt them to work on areas that don’t need improvement and to ignore areas that do
If your were a CIO, what metrics would you want? How many is reasonable to have?
Compliance Metrics: This evaluate progress against the program’s goal. In this area, we need a metrics for short-term goal and long-term goal. For short-term metrics, it can help company get current process as soon as possible to avoid risk. For long-term metrics, it can give a blueprint which can lead us go straight.
Process Metrics: This evaluate business processes. This metrics is conclude cost and benefit. Financial also is a serious problem and red line for all program. Thus, how to use less money to get more benefit is real important.
Workload Metrics: This evaluate employees’ workload. Employees are the soul of companies. If our soul fall down, our body or company must get sick. Therefore, evaluating employees’ workload is real important, which can help CIO know can they get more works or they should abandon some programs.
I think all of these that we should use for IT department. We need enough metrics to monitor our department and evaluate it through different parts. Programs always lose its way and coast extra money, so we need enough metrics. However, how many is enough. I think It is hard to say about this. If this department also does well, which can finish program on time with finishing goal and in the budget, we do not need more metrics. We just need some basic things. However, if this department cannot, we need more specially metrics to push programs go well.
Source:
https://cio.gov/performance-metrics-and-measures/
http://www.cio.com/article/2955777/best-practices/12-critical-metrics-for-it-success.html
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
Metrics play an important role in an organization achieving their business goals and objectives. Metrics allow an organization to monitor their performance and to make adjustments when goals are not being achieved. If I were CIO, the following metrics would be important to me.
• Operational metrics: applications/services performance and availability
• Delivery metrics: project delivery (The percentage of projects delivered on time). Cost (the percentage of projects delivered within the cost estimate). Project satisfaction (The average score from post project surveys completed by business partners).
• Organizational metrics: percentage of employee’s voluntary moving to different jobs (internal vs external). Number of employees with written performance reviews.
• Financial metrics: budget variance (Actual costs compared to budgeted costs), Resource cost (The average cost of a technology resource).
• End User Satisfaction metrics: Use of surveys and focus groups to measure satisfaction with IT.
• Security metrics: Number of prevented network intrusion attempts per year, number of security breaches.
Through my research, the standard metrics used can vary. Depending on the maturity of your business, and your business offerings, this number could change.
1.Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
The interest in measurement is justifiable. Measuring is an important strategic business tool utilized by organizations. Measuring helps IT align results with your organizations vision and needs. Measuring brings IT and the business together to enhance and continuously improve the value that IT delivers. Continuously measuring helps IT stay proactive when business objectives change.
Deciding what to measure is the difficult task. I understand that measuring every part and process of the organization may lead to over analyzing, but without measuring certain core competencies of the organization, you will have processes go without check. Not measuring may lead to poor quality in your processes, inefficient services and not being aligned with your overall business goals.
From an IT perspective, I would want my organization to decide what to measure based on priorities from the board and stakeholders to be aligned with business needs.
5.What measures are being used in your organization? Do they make sense?
I know the CIO of Temple needed metrics on the help desk department to justify hiring additional employees.
He wanted to know the following info:
-How many HD tickets are put in per day
-How many HD tickets a HD rep receive, process, close per day
-How long does it take a HD rep to complete a HD ticket.
-Which avenue do customers prefer to contact HD via telephone, in person, ticketing system.
-Metrics on how many HD tickets are received, processed, closed after normal business hours.
-Metrics on turnover rate
-Metrics on customer satisfaction
I think their metrics made total sense to when trying to justify why additional employees are need for the department.
sorry Andrew I didn’t mean for this comment to go under your reply.
Andrew,
I like the way you put it, “Measuring brings IT and the business together to enhance and continuously improve the value that IT delivers.” It is critical that deciding to track performance is not enough; that leveraging the information gleaned from KPI’s facilitate the continuous improvement.
How can measurements become obstacles to change?
Measurements are good because it can give us a direct cognition about what happened and what should we improve. However, everything has it negative side. When we overuse measurements, it will become a bad news for the company.
Employees will just focus on measurements. They just do what monitor policies ask they do. They no longer care about what they should do. When they finish that, they think they are good enough and they do not need to do anything. However, if employees do same thing for a long time, they will lose enthusiasm.
Changing sometimes means break current rules. In the other words, we cannot get hight score in the measurements. Then we cannot get more salary in the end of the year but if I follow it, I can get more. Therefore, a lot of employees will give up changing and sleep in the measurements to get what they can get. If this trend continues, employees will lose creativities.
Both of them are very dangerous for the companies because in the market, which company does not improve itself, it will be abandoned.
The Star Ambulance Case: Take Two
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and bring it to class (Jan’s Section) or post it on the class blog (Rich’s section).
Financial Perspective
Centralized IT budget vs departmental IT budget
Total cost of outsourcing on non-specialized projects
Customer Perspective
Average speed of answer calls in relation to a medical emergency
Internal Process Perspective
Percentage of projects completed by the PMO group vs completed by individual departments
Percentage of critical projects vs noncritical projects completed
Organizational Perspective
Turnover rate
The percentage of solutions added, deleted, or updated monthly via the knowledge base
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
If I were a CIO, I would want the following metrics
-Financial metrics
-Metrics for customer satisfaction with IT departments.
-Project completion metrics
metrics of how often the project went over budget or end-time
-Security breaches metrics
-Software/Hardware Metrics
As the CIO of STARS Air Ambulance, I would want my Balanced Scorecard to focus on the following:
Finance:
-ROI on newly purchased infrastructure
-savings increase in dollars on elimination of consultants
-budget adjustments on computer-assisted dispatch system project (within 10% of original cost)
Internal Processes:
-100% pass on all audit reviews
-percentage increase in delivery of IT products/services due to implementation of single, streamlined IS function
Customers:
-no less than 90% satisfaction rate of customers
Employees:
-increase in retention rate of IT staff
-% increase in IT staff ability to do job that was once done by consultant
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and bring it to class (Jan’s Section) or post it on the class blog (Rich’s section).
The Balanced Scorecard Links Performance Measures and Demonstrates IT Value. The STARS mission speaks of “providing a safe, rapid, highly specialized emergency medical transport system for the critically ill and injured”
As the CIO, with STARS mission in mind, the Metrics I would want on the BSC (using Cobit 5 as a guide) include:
1. Financial
Goals
-Realized benefits from IT-enabled investments and services portfolio
-Commitment of executive management for making IT-related decisions
-Realized benefits from IT-enabled investments and services portfolio
-Transparency of IT costs, benefits and risk
Measures
-Develop means to communicate the importance of information systems to the other STARS managers and gain their buy in for improving IT functions in STARS compared to ignorance of the other managers.
-Having a target mix before starting to approve projects to successfully deliver business objective compared to having multiple (24) concurrent projects all identified as critical
2.Customer
Goals
-Delivery of IT services in line with business requirements
-Providing expert pre-hospital care and transporting Critically Ill and injured patients to definitive hospital care within one hour
Measures
-core commitment to deal with serious trauma within the golden hour
3.Internal
Goals
-Security of information, processing infrastructure and applications
-Optimization of IT assets, resources and capabilities
-Enablement and support of business processes by integrating applications and technology into business processes
Measures
-Bring together a comprehensive plan to stabilize information services and create conditions for change to provide high quality service
-Role definition, responsibility and accountability for staff and consultants compared to staff anc consultants who cant define their role
-Reduce the number of hired consultants to reduce the huge consulting costs (looks like the consultants have been exploiting the company)
-Create a formal Project Management Office and have a Project management portfolio vs not having timeline for projects/having sketchy project plans/project lacking a line of responsibility from the project manager
-All projects to be proposed in a uniform way that leads to conformity and standardization vs having consultants employ their own methodologies.
-Centralized IS functions and IS personnel
-Need to ramp up and stabilize aspects of the physical infrastructure
-Prioritize and develop a methodology for Security of Network and Systems vs handling security matters on a firefighting basis
4.Learning and Growth
Goals
-Competent and motivated business and IT personnel
Measures
-Trainings etc to raise awareness across the organization of the critical nature of IT over time vs ignorance of staff
1. Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
I do not think it is overkill to measure everything. The key this is that the measurements are being used for decision making. I believe as many executives and compliance team members do in that the more data is only good. I want to measure turnaround times, client satisfaction, service times for building servers or pcs or even writing code. These measurements show trends and lets management understand expectations and performance. Hence this helps management make decisions to improve service. The worst thing or overkill is when all this data is given or measurements and NOTHING changes. That make employees and staff have the mindset that things will never change. Its similar to sports team looking at stats as an objective and third party data that show what is occuring.
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
As a CIO, there is no amount of metrics that are unreasonable. The key for me would be that the metrics are being used by myself and appointed executives in decision making. No use in not using metircs after putting in effort to get metrics which is a waste of effort. The metrics i would like would be:
Project management metrics: are projects being completed on time as a whole? are milestones withing projects being reached? are the projects successful and what percent? Are projects going over budget?
Operations: where is the budget going? what are we spending most money on? How can we cut money on office supplies and PC hardware? How often are we re-imaging computers?
service: How many tickets or tasks are being closed? what is the client and customer satisfaction? what leads to higher results? which employees are doing a better job and which metric to base their performance off of?
Are there security issues? what causes network issues? how much system downtime in a designated period of time? what sites are people going to on the internet at work? what can be used to improve productivity?
5. What measures are being used in your organization? Do they make sense?
we use most of our measure with Help Desk and End User Support and Project Management Office.
– turn around time for help desk tickets
– surveys from end users on how satisfied they were with service given
-how long before an individual comes to their PC or remotes to their PC
-how long does the PC last? is it OS or harware realted
– are new projects addressing user’s concerns
-are projects being completed timely and within budget
Stars Ambulance Case: Take 2
If I were the CIO of Star Ambulance, I would choose the following metrics:
Operation metrics: 1. Online service availability/performance 2. Information security level
Delivery metrics: 1. Project satisfaction (every Star Ambulance project is related to life) 2. Per project cost
Organizational metrics: 1. Attrition 2. Performance reviews
Financial metrics: 1. Budget variance 2. Resource cost 3. Cost saving
The Star Ambulance Case: Take Two
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and bring it to class (Jan’s Section) or post it on the class blog (Rich’s section).
As a CIO of this company, I will consider followed metrics:
1. Financial:
a. The rates of IT costs in total costs.
b. The proportion of the maintenance, and update costs.
c. Money for developing.
2. Internel: (employees means other department employees)
a. The efficiency of IT server.
b. The frequency of using IT server.
c. System safe running time.
d. System maintenance and recover time.
3: Users:
a. Users’ satisfaction.
b. The frequency of using IT server.
4.Education:
a. The time of learning.
b. Computer capabilities.
c. Time of maintenance.
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Today we live in a world where data is everything. Data is collected for various measures to be able to analyze and understand what is going on and projections for the future. Analyzing this data decides or atleast lets organizations know if the decision making process they have is working or failing. It can be overkill to collect data on everything as it can be costly to store if there isn’t real use for it. Therefore, an organization should collect data on items that are vital for the organization and understand how they will use all of the related data they are collecting to understand their decisions. If the data isn’t relevant to their goals or understanding, then wasting time measuring all data points that may be wasting resources.
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
As a CIO I would measure metrics that are critical to the success of my department and company. Per an article in CIO.com I would measure Operational Metrics such as online application performance and production incidents; Delivery metrics such as project satisfaction, delivery, cost as well as defect containment; Organizational metrics such as performance reviews and Financial metrics such as budget variance and recourse costs. Having a few key metrics in each function of the business is ideal so you can have an overall picture for primary components of vital business purposes.
5) What measures are being used in your organization? Do they make sense?
In a past company that I worked out, the metrics used for IT were a lot like those that have been previously mentioned. Metrics included help desk tickets closed, length of time to complete help desk tickets, surveys of customer satisfaction, and the time taken to complete projects. Of course, when viewing these metrics it is reasonable to say these don’t give a holistic view of an employee. Although when trying to view the effectiveness of an employee I think the previously mentioned metrics should be considered, but should not be the only deciding factors.
4.How can measurements become obstacles to change?
Measurements can become obstacles to change when the data points being recorded aren’t actually the correct data points needed to make up the measures being evaluated. For example if I am analyzing how many projects I am completing but don’t identify if it is for projects completed on time or under budget the measurements I am using will not align with the actual requirements or reporting needs of the company. The stakeholders will not correctly make decisions for this scenario. The definitions of measurements and reports need to be identified.
5. What measures are being used in your organization? Do they make sense?
Due to the size of the company I work for, I am not privy to the kind of measures/metrics that are used by other organizations within the company, however I can speak a little to the measures that are being used by the IT department at the company I work for. Measurements/metric are primarily based off SLA hours and Age of customer tickets in different states (i.e Avg & Max active age, Avg & Max response, time spent in pending state, Avg/Mac SLA hours etc.), which are then categorized by priority, application, personnel, IT organization/group.
I think they make sense because those metrics helps the IT personnel responding to customer tickets know their individual rate of efficiency and how effective we are in responding to tickets. IT managers and senior level management use information from these measurements to judge how well each organization is doing and how they affect company in providing value.
The Star Ambulance Case: Take Two
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and bring it to class (Jan’s Section) or post it on the class blog (Rich’s section).
As the CIO I would like my Balanced Score Card (BSC) to include the following:
— Financial Metrics:
-Cost reduction
-Return on Investment (ROI)
-Revenue
— Internal Processes:
-IT services delivery rate (i.e efficiency)
-IT processes streamlined
— Customers:
-Turnover rate of customer requests
-Satisfaction
— Employee Maturity & Growth:
-Average time for IT staff to learn what is required in their role.
Star Ambulance Case:
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO.
Here are some metrics I would want if CIO:
Start Ambulance Score Card
Financial
• Project Budget to Actual
• Consultant reduction cost savings
• Consolidation of supported systems cost savings
Employee
• Training metrics for IS employees
• Percentage increase of tasks completed by IS staff vs consultants
Customer
• Customer satisfaction rate for services delivered
Internal Processes
• Security Metrics (Number of prevented network intrusion attempts per year, number of security breaches)
• Core process outages (network disruptions, CM failure/rollback)