Readings
- Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
- If your were a CIO, what metrics would you want? How many is reasonable to have?
- Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
- How can measurements become obstacles to change?
- What measures are being used in your organization? Do they make sense?
The Star Ambulance Case: Take Two
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and post it on the class blog .
Rich
Pascal Allison says
Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
IT delivers services, and there are needs for controls when these services are provided to meet customer satisfaction and business goals.
How does management, customers (internal & external), or regulatory institutions gain knowledge of services provided for correction, invest in better equipment, or improve activities?
The most be monitoring and balanced scorecard (measuring) which say the effectiveness of activities (performance) compare to plans and plans compare to requirement.
In short, much interest is in measuring because it helps in determining the need for improvement, direct organizational focus (outcomes), and aligning business activities with strategies for improvement.
Yes, it is overkill to measure everything. Measuring countable result is doable. How do you measure noncountable activities? How do you measure the performance of a person who supervises others? How do quantify performance that has its input from outsiders? How do you count outcome that originates from influence? It is just not easy to measure things that do not have direct input in activities. If the typical measurement process is used for all, it could undermine the organization vivacity and missed opportunities.
The organization should establish a simple and flexible approach to measuring. A methodology for decision making that is not limited to performance measurement(numbers), but opportunity and recognition should be instituted.
Richard Flanagan says
Pascal – great focusing on controls but don’t forget the basic SLA’s for a service. This is not just quality, compliance or security, its all of them rolled up together.
Pascal Allison says
Absolutely Instructor Flanagan, pointed control as a focus (example).
Every aspect contributes to the success of a project, service, goal, or meeting expectation . Thus, each will have a measure or metric for improvement.
Since they all contribute differently to success, a fair metrics or balanced scorecard (measure) is required.
Donald Hoxhaj says
Pascal, you brought an important point about the impact on measurement on direct organisational focus. I too consider this an important point because many companies fail or many projects fail to adhere to cost, quality, and time because of failure to understand the organizational goals and project objectives. Another interesting aspect you brought is the need to have opportunity and recognition measures. You nailed it because these measures are non-quantifiable and employee recognition directly impacts the project and its success too.
Michelangelo C. Collura says
Your point about measuring non-countable activities is notable, but such measurement can occur, though perhaps not quantitatively. I would consider office satisfaction to be difficult to measure but very important to achieving objectives and avoiding other problems such as increased turnover.
Patrick DeStefano (tuc50677) says
Michelangelo,
I see where you are coming from here and I agree that non-countable activities, such as office satisfaction, can be greatly useful if done properly. I disagree that it can be overall difficult. Yes, maybe not as clear cut and simple as looking at sales figures or hours spent. A company I am familiar with uses a survey similar to the Meyers-Briggs exam to gauge employee satisfaction in many key areas. They ask the same question in several different ways and ask employees to rate from 1-5 on an agree/disagree scale. After the employees take the survey, they have their KPIs which tells the IT Governance team which areas employees are satisfied with, which areas need improvement, as well as helping to gauge improvement over time if the survey is given several times over the years.
Michelangelo C. Collura says
Insightful reply, Patrick – thanks. Your example shows that qualitative can be quantified, and that’s a very good thing for firms!
Pascal Allison says
If your were a CIO, what metrics would you want? How many is reasonable to have?
Metrics are important for success in achieving organizational goals. Metrics help in achieving goals in that they direct management focus for adjustment (improvement). When the right metrics are not use, metrics becomes a risk. It is necessary to use metrics as required. Because, not all operations and project require all the metrics usage. In most cases it is reasonable to have four (4) metrics (operational, financial, delivery (feedback), and organizational).
Donald Hoxhaj says
Pascal, I concur with you that an organization requires the 4 essential metrics i.e. operational, financial, delivery (feedback), and organizational. In fact, in most organizations, the consolidated version of the metrics is linked directly to the organizational metrics that basically sums up the most important business data for decision making purposes. Something interesting that you mentioned is that metrics that are not used can become a risk. Though I able to get your point to some extent, it would be great if you can give an example because I have a feeling that unwanted metrics can lead to complexity and confusion across the leadership. Thanks for sharing your thoughts.
Richard Flanagan says
Donald – what do you mean by organizational metrics, can you give us some examples please?
Donald Hoxhaj says
Professor Flanagan – I refer to organizational metrics as the business-level metrics that takes up information from individual functions within an organization. Example, Revenue Metrics for C-suite board. This would collect information from Sales Departments, SCM departments, Operational costs, etc. to arrive at the Revenues for a particular period.
Michelangelo C. Collura says
I would opine that measurements shouldn’t exist if no aspect of the firm needs to know them, but some may be necessary for given functions/departments/etc. For example, I don’t necessarily need to tell the CFO about help desk task completion rates, but I’d want to know them in the Help Desk office. In a sense, information silos are useful for measurements – but not for much else!
Pascal Allison says
Hey Donald,
I meant “the right metrics.” If the right metrics are not used, decision making will be hampered and will not product valuables.
And I totally agree that unwanted data cause complexity and confusion. Another issue about unwanted data is “cost.” Why use resources on something that is not required, needed or of any value? Data collection, consolidation, storage, and analysis cost a lot of resources (time, money, space, etc).
If I am understanding Michelangelo right, if the data or measurement are not needed by a department control it. If the organization does not need the data, get rid of them. The organization will use resources and there will be confusion among decision makers which is risky.
Vince Kelly says
I suppose its all a matter of perspective right? If I’m an employee I wouldn’t be that interested in KPI’s like payout ratios, debt to equity or liquidity ratios. Even the CFO and finance organizations probably wouldn’t care that much about them either other than the fact that they are probably the first thing that potential external stakeholders like bond holders and investors look at.
For internal employees they are a yawn, for the CFO’s team they’re essentially window dressing that really only serve to pique external interest but for external investors and creditors they are critical in getting a ‘feel’ for a business they are thinking about.
As you point out Pascal , the risk is in having your business rely on the wrong ones
Pascal Allison says
Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
If there are more metrics than can fit on one balance scorecard, I would regroup the metrics into groups and sub-group for clearer understandability, information processing, and decision making.
At the organizationally level I will use less components or categories. According to Heather Colella, “the less is best.” The balanced scorecard could be Robert Kaplan or David Norton’s or business goals or capabilities.
Pascal Allison says
How can measurements become obstacles to change?
A balanced scorecard is not one sized fit all. Different product strategies, environment, competition, and services require a different balanced scorecard. Balanced scorecard must be customized to fit goals, strategies, technologies, and culture.
Using a balanced scorecard that does not match the operations, products, improvement activities, or service is a recipe for failure. If there is a mismatch, miscalculation is apparent which will result in a wrong decision.
The balanced scorecard should be grounded in the organization strategic, objective, and competitive demands. The BCS should be used as a continuance process to determine the present status and prepare for the future. Some critical things that make a BCS an obstacle to change are:
• Improper definition of metrics;
• Missing data and reporting;
• Inadequate review;
• Lack of corrective measures; and
• Excluding external variables.
If these issues are not addressed, the balanced scorecard will misdirect decision making which becomes a problem to change.
Patrick DeStefano (tuc50677) says
Pascal,
I agree with the direction you’re going with this. Organizations need to think very carefully what types of metrics and measurements they are looking at and to be sure that they fully understand what the measurements mean. They also need to be sure that the measurements are accurate enough. Simple things like rounding a value up or down in a formula may be fine if you are only reporting the data on a 20 person team, however if that same formula gets adopted for an entire Line of Business or organization with 10,000 employees, that rounding may become a much bigger of an issue. I had an internship back when I was in undergrad where I was working on analyzing reporting done in our mortgage banking management team. I was looking through the reports and providing analysis on how to improve them. I came across a formula used in one of the rows which didn’t look quite right. It was an hours spent measurement for IT Projects for the quarter for over 1,000 resources. The formula was taking the average of an average of an average. I recalculated the value using the actual data and the two numbers were way off. It’s simple errors like these that can really paint an incorrect picture to governance teams. In order to make proper decisions, management needs to have confidence in the metrics
Pascal Allison says
What measures are being used in your organization? Do they make sense?
The measures used in my organization make much sense. Each measure represents an essential part of the organization’s goals. The components are flexible but do not change frequently and concomitantly, because the availability of data is not the same across the components. The frequency of data availability does not divert or distract the focus or goal.
Based on my observation, the components of the balanced scorecard (performance measurement system) are:
Customer satisfaction
Employee satisfaction
Business result (quality – efficiency & quantity – output)
The measures help establish priorities, promote performance, and align performance mark and organizational goals. The measures help with feedback to determine areas for improvement and meet customers and stakeholders expectations.
In short, the measures examine the system efficacy and relay outcomes.
Richard Flanagan says
Pascal – what do you mean by “output” on the Business Results line?
Pascal Allison says
In this case output means, the outcome of effort and resources utilized over a period of time to achieve expected goal. When a case is closed or work/project completed over a period of time, the expected result or product.
Donald Hoxhaj says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Measuring is not a non-profit activity, but it indirectly contributes to the growth of the company. It allows for greater control of cost, performance, resources, so that everything functions as a whole, rather than separate entities towards the ultimate goals of the company. Even in IT Services, different departments and functions have different KPIs for measurement of internal performance, client satisfaction, project delivery, quality of checks, errors, etc. They allow for greater efficiency and help the organization to become better consistently.
It is definitely overkilling to try and measure everything because too much of control is never good and it kills the scope for innovation and change in process too. An organization should decide measurements based on the KPIs of a company and the most important actions that lead to customer success.
Richard Flanagan says
Donald – can you drill down from company KPI’s to show how they could be used to build relatable metrics for say the email service or warehouse systems?
Donald Hoxhaj says
Professor Flanagan – Yes, definitely an organization can do. What many keep forgetting is that Drill-Down and Drill-up functions are not a technical perspective, but a way of organizing your thoughts. It ultimately boils down to the fact that you are simply penning your thoughts on a paper to achieve more efficiency. Warehouse systems can easily be linked to Supplier Information, Stock availability, Planning, and Forecasting of next stock, etc. All these metrics are in fact linked with each other and if an organization views this as different metrics, then it is probably wrong.
Donald Hoxhaj says
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
Metrics are key indicators for the Financial, Operational, and Overall health of the business unit. As a CIO, I would like to have metrics across 3 broad areas i.e. Operational, Financial, and Delivery. For Operational Metrics I would use Application or Service Performance Metrics, SLA Trends, etc. For Financial Metrics, I would like to gauge the health using Cost of Resources, Financial Trends, and Revenues Impacted Metrics. Finally, for Project Delivery I would like to gauge the health using Project Cost, Defects, Resolution Time, etc. The number of metrics is dependent on the project per say and not an idealistic number that can be predicted beforehand. But in normal circumstances, 5-10 metrics that focus on these 3 broad areas should be good enough.
Donald Hoxhaj says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
The best approach to reorganize the metrics is the group them based on functions, departments, and key indicators. Another method to efficiently reorganize metrics that cannot fit in one balanced scorecard, it is convenient to have drill-down and drill-ups as the best option as they not only save space in the scorecard, but are more intuitive and visually appealing too.
Donald Hoxhaj says
4. How can measurements become obstacles to change?
Introducing new measurements is not an easy task because it requires not only approval from the higher management, but also approvals from every employee within the organization’s hierarchy. Most organizations have legacy systems and processes that do not follow Industry standard practices of measuring health. Also the employees within the organization are used to working in a particular style. Resistance to change or adopt a new metric or measures is the biggest obstacle faced by companies as people do not readily accept change. Example, look at Kodak. Despite having all the necessary resources for digital films way ahead in time, the company faced resistance from both employees and from board to change because their tape films were doing profitably well. The company went bankrupt because it failed to change. Measurements that are obstacles to one’s work or processes become problems in the long-run and the organization becomes an Elephant in the room.
Richard Flanagan says
Donald – Kodak is more of a strategic direction failure, although this can show up in their measurements.. Even though they invented the digital camera they were, at heart, a chemical company. They invested heavily into the new digital world but within the existing business structure. Not surprisingly a bunch of chemists who knew how to make money making film didn’t know what to do with digital and so their efforts were not successful. Compare with Fuji, another film producer. They created a new division, with people with digital backgrounds, to pursue the new business and were successful.
You do have a point about the wrong KPI’s. Their focus on gross profit and customer satisfaction may have hurt Kodak. They were making great money on film and saw that customers were much happier with the quality of pictures taken with film cameras until it was too late. Thats the problem with disruption, it changes the business model which often makes the old KPI’s different from the new ones.
Michelangelo C. Collura says
Not to diminish the value of measurement, but the Kodak case feels like a difficult one to predict. I would certainly fault them for not diversifying, but I recall as a photographer how people used to think digital could never top the quality of film photography (too much pixelation). Of course, this eventually changed, and Kodak was done.
As for KPI’s, a firm in such a disruptive environment might benefit from measuring adoption of competitive products/services, regardless of quality, assuming that eventually – as demand increases – quality will find a way to improve. This seems a bit reckless, but it might work in disruptively changing technologies or fields.
Vince Kelly says
Donald,
Was your point that blind, obsessive addiction to metrics while ignoring other important opportunities like taking a step back to survey the (potentially changing) strategic environment is where measurements become obstacles?
I once saw an album cover, (I forget the name of the group) that had a picture of a guy wearing sun glasses, lying on a cot, sipping on an umbrella drink and getting a sun tan in the middle of a toxic waste dump;) The album was called “Crisis?, What Crisis?” :):)
Even in that case, the guy had all of the right metrics,(the proper distribution of SPF 50 sun tan lotion, a full drink with the correct proportional metrics necessary to produce the perfect rum and coke) – he just oblivious to the bigger, strategic picture :););)
Richard Flanagan says
Guys – exactly, the metrics must be appropriate for the situation and, unfortunately for many companies., industries going through disruption may not recognize it until its too late to change.
Donald Hoxhaj says
Vince, Thank you for bringing up your points here. Yes, we need to realize the fact that metrics management in itself is a painstaking task and requires resources and cost to get the data churned out. Too much obsession with the wrong metrics is definitely a big No. But identifying the right data for your company and working out the metrics for them will obviously lead to greater efficiency, higher visibility into unexplored territories, and lesser cost too.
Paul Needle says
I like this topic even though it’s brought up often. Kodak’s profit margins on film were much higher than on digital. They certainly had the technology to move into a digital platform. Their problem was that they thought digital film WAS the disruptive technology. The real disruptive technology was in form of sharing online content. This is why it’s so important to include innovation in the overall IT strategy and then measure the vision with appropriate metrics.
Donald Hoxhaj says
Professor Flanagan – Yes you are right that this is more of a strategic failure, but if we closely observe the case of Kodak, there was a time when the management did try to diversify into digital films but there was huge resistance. I feel the the point here is not about directions, but measurements that people fail to realize are important or priority over others. But yes, you have a great point that you added.
Michael Gibbons says
I think another piece that is overlooked here is the accuracy of the data. While in theory, the balanced score card and measuring these items can be valuable, without sufficient controls in place, how do you trust the results? I guess when the company fails or something goes bad, this type of data may be looked at more closely but the process as a whole should be built following any other SDLC project and needs to ensure segregation of duties, a clear documented source of where the data is coming from so it can be reconciled for accuracy.
Donald Hoxhaj says
5. What measures are being used in your organization? Do they make sense?
In my company, we have metrics and measurement across different departments and functions and an overall metrics that consolidates from these individual departments for management purview. Individual departments used Balanced Scorecard, while project delivery teams in Advisory and Tax use metrics related to Project cost, Resource Utilization, Efficiency Rate, Percentage of Defects, and Customer Impact and Success metrics. Yes, they do make sense. One interesting metrics that usually organizations do not think is the Customer Success. While all companies measure success of work internally, it is important to measure success from Client’s perspective. Example, if in a project, the client has spent 80 hours trying to explain the business problem to the vendor teams, then the success is low. If the client spent 10 hours in explaining the problem and communicating other details, the success from the Client’s end is high.
Michelangelo C. Collura says
Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
A firm cannot properly assess any process if they aren’t measuring the impacts periodically. Measurement is necessary to improve efficiency, cost-savings, and many other aspects of a firm. There are limits to this, however, and so some metrics are measured more frequently or commonly than others. Of course, cost overruns on projects are a useful metric, even in non-IT areas, but perhaps employee satisfaction is also a concern for an IT department with high turnover. It may be overkill for a firm to concern itself with such a metric, but this is a singular needs-based assessment.
To decide on what my firm needs to measure and how often, we’d need to take stock of the firm’s core competencies and functions. Of these, which parts are profit centers? How about areas that support those centers, such as HR? If we did not know how well a given area is performing, would the firm be harmed financially? If so, then this would be an area to measure. The more impacting an area is, the more intensely or frequently we should scrutinize its performance.
Vince Kelly says
Dead on Michelangelo, trying to get a handle on which measures are the most appropriate is like trying to grab Jello. It depends on a multitude of variables, what type of business is it? (commodity dependent?, capital intensive? operating expense sensitive?), what kind of industry is it? (Monopoly?, competitive intensity?), where is it physically located? (near easily accessible raw materials?), in a developed, financially stable country? Even the philosophy and approach of senior management to the business is a factor – are they more tolerant of higher leverage ratios than a more fiscally conservative competitor in the exact same industry?, etc., etc., etc., etc.
The firm needs to continually assess processes and performance but needs to keep in mind that they are only windows or snapshots of a particular point in time. They are really only effective in the context of other tools (e.g., trending, regression analysis, etc.) and the proper strategy.
Michelangelo C. Collura says
Good point about the snapshot. Without knowing the moment, we may have a problem. Without knowing a series of moments, and thus a trend, we may have a catastrophe.
Michelangelo C. Collura says
If your were a CIO, what metrics would you want? How many is reasonable to have?
It greatly depends on the nature of the firm. In a standard consultancy, I would want to know utilization rates in terms of billable hours. I’d want to closely follow client feedback to re-assess pricing our work/hour based on higher or lower acceptable rates. I don’t know if we’d need to know a closing rate on new contracts/clients, but this would also be worth following closely, both within the organization overall and specifically within individual teams sent out in the field. I’d also want to know about Help Desk response times and rates of success, if that was in-house. I would say there is no reasonable number of KPI’s to have in an objective sense. It entirely depends on what the firm management wants to know and is willing to measure and study.
Michelangelo C. Collura says
Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
An aggregate score, based on adding together weighted metrics, based on assessed impact to the firm. This is a very simple scorecard as seen in many industries such as the consumer credit score, and we could use it for easy and comprehensible dissemination to our executive leadership, management, or even all personnel. IT gives an immediate sense of things at a high level and may help to effect change more quickly since we can connect with more staff on such a level.
Michelangelo C. Collura says
How can measurements become obstacles to change?
I think firms can sometimes miss the forest for the trees, with focus on measurement leading to losing a sense of what those measurements are meant to enable. In one personal experience, I was trying to measure social media access for a project, knowing who uses which platforms. The goal was to then tailor social media outreach to get the best results, but it ended up being a way of convincing everyone to use specific platforms at the expense of all others, even though the data didn’t imply this was a good approach. In some profit-focused ventures such as high-frequency trading, I imagine measurements could be used to reinforce the profit centers, even if other measurements may indicate risk in this approach. An example of this would be the increased mortgage defaults starting in 2006 – a useful warning given the likelihood of an economic downturn around that time. This measurement, particularly in lender banks, would have helped to avoid hardship in 2008, but it was ignored in favor of default swap returns on investment.
Richard Flanagan says
Michelangelo – having the right data to warn you of a risk is one thing, taking the right actions to mitigate it are the other big step. I’m not well versed in the financial marketplace but clearly some traders saw the data and took advantage of it.
Michelangelo C. Collura says
Very good point, Professor. And yes, some traders did take advantage of it. This advantage took the form of either betting on things going south (a good idea given what happened) or betting on the good times to keep going (an unfortunate choice made by Lehman Brothers).
Vince Kelly says
Reconsider the Stars Ambulance case that we discussed the first week of class. If you were the incoming CIO what would be your top 3-5 objectives for your first year or two? Think about what quality means in relation to these objectives. Then think about what metrics you would use to measure that quality and what data you would need to collect to support those metrics. Post your thoughts online.
The following are major problems that seemed to be the most salient to me:
1. STARS had a feudal system of governance and a distributed, fragmented IT budget. Other departments had their own systems staff and projects
2. Departments also used their own outside consultants to do maintenance – each had their own systems development methodologies.
3. The role of staff had not been fully defined. Consultants and staff had little accountability and basically did whatever they wanted or deemed to be important. One side effect of this was that consulting costs were “huge”
4. Even though teamwork is considered to be a critical success factor, staff felt that they had no visibility or importance to the rest of the organization
5. Twenty-four unrelated projects underway with no prioritization or linkage to the strategic requirements of STARS
6. No business continuity or backup plans
7. According to the article, “the CIO needed to improve the quality of service provided by IS and to provide real value to the organization as the bottom line.”
In order to address these deficiencies, the new CIO should break these down into the following three focus areas or steps:
1. Identify, document, and communicate (internally to IT and externally to the rest of the business) the following:
– The requirements and needs of the business as well as organizational process and structure to how the requirements are met by IT
– IS capabilities from the perspective of its people, process and technologies.
– Aligning the goals and objectives of IT with the tactical goals and objectives of the various STARS organizations as well as the strategic goals and objectives of the company
2. The need to create a ‘roadmap’ that shows where we are now and where we are heading
3. The need to establish a level of excellence in IT governance, change and operational management.
STEP 1:
Restructure around a centralized IT governance model.
Identify, document and communicate the requirements of the business, its current IS capabilities,(in terms of its people, processes and technologies) as well as aligning it’s tactical and strategic goals technical goals with business goals:
This would include understanding where IT was in terms of its current organizational maturity level and how to ultimately transition IT from a basic unreliable ‘service provider’ to a trusted business transformation enabler.
This might entail understanding, among other things, to what degree is IT,(i.e., the people, processes and the technologies) capable and focused on the business? That is, are they limited primarily to siloed, domain-specific activities? e.g., are individuals focused on just being great ‘coding junkies’ or ‘router jockeys’ (the lowest IT organizational maturity level) or can they think in terms of the next level of IT organizational maturity – leveraging overall cross-architecture approaches to technology? Can they think beyond the technology in terms of how to leverage IT to deliver specific solutions or how those solutions ultimately impact the business? Or do they have the ability to function at the highest level of IT organizational maturity by being able to grasp the technology and to understand how that technology can be used to create specific business solutions in a way that enables their business partners to innovate and transform the business in new ways?
STEP 2:
Create a common ‘roadmap’ of where IT is are now and where it is are going. This includes:
– Implementation and adoption of an enterprise architecture. This would include the definition, implementation, integration and adoption of their business architecture, information systems architecture and technology architecture.
STEP 3
Establish a level of excellence in implementation IT governance, and operational management.
– Adopt COBIT5 and ITSM as the standard process models for the entire IT organization.
– Establish feedback and customer satisfaction surveys
– Change employee compensation plans to incentivize customer responsiveness, teamwork and entrepreneurial spirit.
– Create/ensure well-defined job descriptions
– Provide clear, consistent and timely employee feedback and performance reviews.
– Develop a culture of continuous learning and personal growth
IMPLEMENTATION and BENEFITS:
STEP 1 Benefits:
– Restructuring around a centralized governance model would consolidate budgets, streamline the budget management process and eliminate waste and duplication of effort.
– Reducing the number of outstanding projects and focusing only on critical projects that directly contribute to meeting business objectives in a measurable way.
Identifying, documenting and communicating the requirements of the business, its current IS capabilities,(in terms of its people, processes and technologies) as well as aligning it’s tactical and strategic goals with those of the business:
– Understanding and being able to articulate these elements would create the ability for IT to demonstrate its value to the organization
– It would improve the quality of IT service because everyone (both IT and business) would be ‘on the same page’
-Would allow IT to prioritize the outstanding twenty-four ‘critical’ projects down into a manageable set of project activities that directly relate to the businesses needs and that had a specific timelines.
– The business requirements, goals and objectives (and how IT had addresses them) would be made part of an “IT Performance Dashboard’. The dashboard would be updated every quarter, would be available for everyone in the company to see and could be accessed via a link from the company home page
STEP 2 Benefits:
Creating a common ‘roadmap’ of where IT is are now and where it is are going would have many benefits including:
– Enabling the creation of performance baselines for the business and IT.
– Enforcement of standards across the company. Adopting an EA would eliminate the problem of different departments using different outside consultant methodologies by defining one standard architecture development methodology.
– The creation of consistent target states – ‘where we need to get to’ – that align with the businesses vision and strategy.
– The creation of a gap analysis for each architecture (business, IS and technology). This gap analysis would also leverage what was learned in step 1 (understanding of the businesses requirements, its capabilities and its goals and objectives)
– An EA would provide a ‘living’ repository that would be consistently updated. This would substantially accelerate new employee learning curves and acclimation as well as creating transparency and a fully auditable environment.
– An integrated EA would provide the continuity, process and links between architecture change management, implementation governance, the technology architecture, the IS architecture, the business architecture and the overall company vision. In this way, proposed changes to one could be analyzed for its ‘ripple-effects’ on the other areas before being implemented.
– An EA would create IT agility because of the architectural principle of reusability – i.e., creating architectural ‘building blocks’ for one project that can be stored in the repository and reused again for other projects.
STEP 3 Benefits:
Establishing a level of excellence in IT governance, change and operational management.
– DR and BCP plans would be put in place and be available immediately. This would substantially reduce the exposure and risk for unexpected events.
– Feedback on how IT is adding value to the business directly from the customer in the form of semi-quarterly surveys and customer satisfaction scores. IT employee incentives and compensation would be influenced by the results.
– Recognize and reward innovation, collaboration and cross- organizational interactions through the use of ‘teamwork and innovation monetary awards and incentives’ would encourage teamwork and innovation.
– Clearly defining roles and responsibilities within IT would create an atmosphere of accountability.
– Providing timely employee feedback, continuous learning and personal growth opportunities would improve the quality of IT service overall
HOW the 3 INITIATIVES WOULD be MEASURED:
Step 1 Measures of Success:
– The number of projects would be determined by creating an IT steering committee that consists of business and IT leadership.
– Transparency and goal attainment would be measured by the creation of an IT performance dashboard which would be directly linked from the main company home page.
Step 2 Measures of Success:
– Creation of baselines for the business, IS and technology architectures – “Where we are now”
– Creation of gap analysis for business, IS and technology architectures
– The creation and availability of the EA repository
Step 3 Measures of Success:
– Semi-annual employee surveys and customer satisfaction scores
– Use/consumption of the ‘teamwork and innovation incentive’ awards
– Training participation and employee certifications
– Documented job descriptions for every IT employee
– Employee job performance feedback as part of every employee performance review,( performance ratings from their perspective).
– Anonymous employee management surveys
– Employee compensation, retention and turnover ratios relative to the rest of the industry
Richard Flanagan says
Great job of thinking through the organizations situation and drilling down to what’s important and how to measure it.
Vince Kelly says
1.. Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
Measuring is really the only way to objectively understand if the organization is meeting the needs of the business and its stakeholders. It also provides a way to transparently demonstrate the value of IT to the business, how effective governance is and if scarce resources are being used in the most efficient way.
As the Balanced Scorecard article points out, “IS can be evaluated in terms of the efficiency of the activities associated with IS development and operations and the effectiveness of those that use IS to improve personal productivity and strive to help attain corporate goals.”
It would be overkill to measure every aspect of everything but this is also unnecessary. Measurements to be constrained and limited to only key performance indicators (KPI’s).
I would think that the organization should decide to measure based around its own goals and objectives in order to make sure that they were being achieved. This would include KPI’s like financial measurements (is the business successful relative to its competitors?, is it leveraging its resources in the most effective way?, Is it achieving and maintaining a high level of customer satisfaction and repeat business?, etc.)
Vince Kelly says
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
I would focus on three key areas:
1. Company financials. Performance indicators that paint a picture of how the company is doing relative to its peers and within its industry in terms of stability and liquidity. Indicators that spell out how efficiently the company was using its resources,(its capital and labor for example) and how productive it was relative to it competitors.
2. Internal Indicators: Performance indicators that provided insights about employee morale, retention, attrition, experience, skill levels and satisfaction.
3. External indicators: Indicators that provided insights as to how well the company is doing with customer satisfaction, repeat business, is it favorably regarded by customers, suppliers, partners, company stakeholders and the in the industry in general, etc.?
Vince Kelly says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
Typically there are more metrics available to gauge the success of a business than there is paper available to print them on;) Gartner points out that,,”…components of a performance dashboard vary significantly by organization…” “….When selecting a method that works for your organization, remember that less is best.”
This is why focusing on a few Key Performance Indicators that identify whether or not your achieving company goals and objectives is happening or if there needs to be a change in strategy is the best approach. You want to identify a couple of strategic ‘gauges’ or ‘levers’ that relate how critical aspects of the business are functioning.
Vince Kelly says
4. How can measurements become obstacles to change?
Slavish devotion to measurement can be as destructive as not bothering to measure at all because among other things, it can create a level of institutional inertia that is hard to overcome.
Relying on measurements above all else can also have a demoralizing effect on employees and customers. In other words, obsessing on the quantitative aspects of the business to the exclusion of everything else could potentially create ‘qualitative blind spots’ in the business by taking some of the more unpredictable qualitative elements out of the equation. – elements like illogical, emotional human behaviors 😉
Vince Kelly says
5. What measures are being used in your organization? Do they make sense?
Our business relies on many types of measurement and KPI’s – if for no other reason than there just happens to be many aspects to our business (which is probably pretty common for all businesses). Its situationally dependent but we use KPI’s that identify/expose financial, internal, customer and resource utilization aspects of the business. Yes, I think they do make sense.
Pascal Allison says
If I was the CIO, my BSC will include:
Financial
Organization (learning & growth)
Operational (internal processes)
Customer perception (feedback) metrics.
where all metrics will support the business vision and strategy.
Richard Flanagan says
Pascal – your list sounds very internally focused on IT. Would you include any indicators other than end-user satisfaction to measure the impact of IT on the business? If so, what might they look like?
Pascal Allison says
Other indicators other than satisfaction are:
Customer Engagement – this measure determines patients and community response to the services provided which measures the value of service
Timeliness – measure duration of cases (start to finish)
Data interchangeability – effectiveness of information exchange
Customer attitude – customers response to service provided
Market shares – invites and response to number of cases
Outcome of response
Richard Flanagan says
Pascal – OK what I wanted to highlight is that there is more to customer/business metrics that just end user satisfaction. The corporation is a customer and they may want a limited budget, the lines of business are customers and they may each want a new system, the external customers and internal end-users are also customers and may want different things. It best to agree to multiple customer metrics that represent these varied customer views.
Jonathan Duani says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Measuring is the only quantifiable way to collect data across an organization. Once an organization starts to measure things they can see how they change over time and then see how they compare to others. I think this is important because they can see what to change or tweak in order to grow as a company. Since this is the case it is important to measure everything that could be deemed useful to an organization. I think it is overkill to measure every little thing that an organization does because at that point you are just regurgitating the live data over again. The measurements will take specific aspects and keep track of them for a later time when they are analyzed. I would want my organization to discuss what they are measuring openly. It should be a discussion, for example if a new project is starting the talk should be what you would want to measure. Also I think they should base it on their goals and mission. If this year’s yearly goal is to complete 10% more tickets with a higher approval rating, they should make sure they are capturing all the required data to keep track of before and after the changes are made so they can actually track the goal properly.
Michael Gibbons says
I think the point of an organization being able to adapt on the fly is extremely important with measurements. If the measurements are focused on critical business processes and the organization has done a good job implementing continuous monitoring, they will be able to adjust in near real time to any situation that is causing them to miss a key target.
Jonathan Duani says
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
I think a lot of other students had the right idea with this one. I would keep track of the company financials, internal and external indicators. For the financials I would keep track of where and what we are spending on certain things so that we can reconvene at the yearly budget reviews and make sure that everything is up to par and maybe some cuts can be made that aren’t detrimental to the organization. I would look at internal factors like employee performance for example, how many tickets a tech can close in a month and the rating that they are receiving from the end user. I would also keep track of some external things like looking at return customers and the customer satisfaction.
Richard Flanagan says
Knowing how the company is doing financially is certainly critical but you might also think of a business process metric that IT has a significant impact on and tie that to IT. My favorite example is Campbell Soup which measures IT cost per case of product delivered at a customer. The logic is that the cost per case metric is used as the most prevalent business process metric so this aligns IT directly to the business. IT’s biggest value add to the company is probably in supply chain, which is also directly tied to this measure. Thus, if IT costs per case halve then IT is either cutting costs significantly or the supply chain is getting twice as many cases to the customer at half the cost. By connecting that to IT projects introduced to the supply chain and you have a reasonable case to support the value of IT to the company.
Jonathan Duani says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
I think there are always too many metrics then there is enough room to fit them all. However I think it is important keep track of the most important metrics that directly impact the organizations goals and objectives. I think it is important to find a couple key point to monitor and keep track of them so that you can look at your goals and see if they are being met.
Jonathan Duani says
4. How can measurements become obstacles to change?
I actually see this a lot in my company on a daily basis. They are so fixated on the number they don’t really care about the quality of service at time. We work off a ticket system for a lot of our request. When a ticket is assigned to a tech it start a timer on an SLA. There are standards set for the type of request and the priority. I see a lot of manager focusing more on the numbers that saw for example John Doe closed X amount of tickets this week and is doing better than Jane Smith because he closed more when in reality Jane only close 2 tickets but they were 2 massive projects take took 20 hours each. I think because of this people can be more focused on the numbers and not focused on the end goal.
Michael Gibbons says
I have also seen this with support desk metrics. I know of an organization where the head of the support desk has an annual incentive tied to a metric like the one described above. He enacted a rule for his support desk that no support desk ticket could be reopened, if a user called in with the same issue because the first one was not resolved, they received a new ticket so that the SLA and 1st call resolution rates (all data pulled by this same individual) remained above his incentive target. I think this area gets overlooked with measurements – especially when they are tied to performance targets, it is always a good idea to have a second set of eyes to reconcile the numbers.
Jonathan Duani says
5. What measures are being used in your organization? Do they make sense?
We measure everything, or at least it feels that way. One of the big things that we measure is the amount of work requests we receive, how quickly they get done, the volume they get completed in and the customer satisfaction at the end. This is important because we can see if our department is operating at the highest capacity for the current work load and if everyone is pulling their weight. I think to an extent they make sense but I think they also should be taken with a grain of salt. For example, if I close 10 tickets out that took me 10 min each and another tech only closes out 2 but it took the same time to complete they should not be penalized because they did not close as many tickets. Unfourtently we do see this a lot with in the organization where everyone is so close minded they 0only focus on what is in front of them in black and white and do not look at the overall picture.
Richard Flanagan says
Jonathan – you point out one of the problems with tickets systems. Another one is are all these requests small help desk type requests or do some of them get into more significant requests. If the later, are they governed to see if they are really valuable?
Jonathan Duani says
Rich, are you speaking in terms of the request themselves or the actually measurements and stuff that management are tracking. I think in management eyes’ it is very valuable for them to have these measurements in place, however if you look at the view point of the tech it is an unnecessary amount of stress that is put on them inadvertently. The reason i say this is because they are always under pressure to close more tickets and get more done no matter the quality of their work because they are worried about getting talked to about their performance.
Patrick DeStefano (tuc50677) says
Rich, I completely agree. Speaking from an end-user perspective of a ticket system, I had ownership over maintaining the hardware and systems in a computer lab used for testing. This lab maintenance very often required administrator assistance, which was requested via a button ticket system. I got to know the administrators on a friendly level over time. They told me that they were always being measured on their ticket closure time and number closed per day. This frustrated me because we had 25 PCs and their assistance often required a much more significant amount of time than a simple end-user with only their PC. This created an inefficiency where there was almost a bit of gaming the system so they could still get the work done that need to be taken care of and not get penalized. Often times, they would ask me to open several tickets for the same thing so they don’t get penalized. Not all metrics are telling the whole story.
Heiang Cheung says
I think this was a great example of how some companies only focus on numbers and don’t really see the details in things because the person could be working on a different issue and different situations take
Duy Nguyen says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
• Measurement is a tool for senior IT management to show the value of IT to Business Managers. It’s also a tool used to track efficiency and activities. Additionally, properly measuring is also a great way to know if the organization investments are worth it.
• Measuring everything isn’t effective in my opinion. Measurements and KPI’s should be decided based on how an organization uses IT.
Richard Flanagan says
Duy – I don’t think you meant this as an IT only answer but it kind of comes across that way. Measurement is an effective tool for the whole organization.
Duy Nguyen says
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
• Depending on the maturity and the usage of IT of the organization, Gartner states that for a new or ineffective IT organization, roughly 70% of metrics should be basic uptimes. Once the IT organization has had success and reputable, basic metrics of uptimes should be less than 30%. Once an organization has some success in IT, the organization can then move on to KPIs that would assist in tracking internal and external facets.
Duy Nguyen says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
• Based on Gartner, the key components of IT scorecard should include Vision, Operational success, Risk, and Tell a story. Organizationally I would assess these metrics and choose the ones that closest fit to these 4 components.
Duy Nguyen says
4. How can measurements become obstacles to change?
• Measurements create a tunnel focus on a certain task or goal. Once KPIs are created, an organization can be so consumed and just focus on improvement of those numbers and loses focus on other facets of the business. An organizational must be dynamic and so should KPIs.
Patrick DeStefano (tuc50677) says
I agree Duy,
Metrics and KPIs don’t always tell the whole story and there will always be some outliers. They can be greatly useful over long term reporting where you can see trends, however sometimes the number should be taken with a grain of salt as they don’t always tell the whole story. For example, management may see metrics showing that a project is way over budget and possibly think that the resources are unskilled or look down upon the project team, however what they may not see is that the project team uncovered a large Existing Production Issue (EPI) while attempting to build the project. In order to get the project working, they needed to fix the EPI as well and this may have cost an additional 20% in hours to do. As a member of management or a governance team, you need to be able to understand that KPIs are not just in black and white, but they tell a story.
Duy Nguyen says
5. What measures are being used in your organization? Do they make sense?
• My organization has a range of KPIs per department. One of our most important reports tracks rent collection percentage per site. Other departmental KPIs tracks key department goals/identifiers. For example, tracking of various procurement, measurements are key for Supply Chain and Invoice payment times for Financial AP group. These reports are also made available to all employees making these KPIs transparent across the whole organization.
Paul Needle says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Measuring lets an organization know if they are on track to achieve the goals they set out to obtain. It demonstrates progress towards the IT goals or overall business goals. Measuring helps remove emotion from the process. However, it can be over kill to measure everything. The unique one-off situations should be handled on a case by case evaluation. The idea is to evaluate a range of important organizational areas with a single approach. I would want my organization to measure operational success on the alignment of IT Strategy and how that supports the business strategy.
Paul Needle says
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
It would seem reasonable to have 4 or 5 metrics to choose from. Any more and the CIO or employees probably won’t be able to focus on any one metric. Any less and you would be missing information that can provide important feedback. The first would be a financial metric such as measuring profit margins, rate of return, etc. The next would be Internal Business Processes. This would include manufacturing productivity such as inventory turnover. Another key metric would innovation. This can be measured in a number of ways but one that comes to mind would be employee turnover. High turnover may indicate a stagnant workforce without innovation. Finally customer categories such as complaints or returns can be measured.
Paul Needle says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
There are many things that can be measured in an organization. It is important that the score card select only a few key measures that are meaningful to the organization. They should demonstrate what IT is doing and why they are doing it. To determine how to pick the appropriate metrics there are few guidelines that every scorecard should incorporate. Vision, operational success, risk and a story that relates the IT strategy back to the vision. Following this strategy will help when determined what metrics to track.
Paul Needle says
4. How can measurements become obstacles to change?
The IT strategy should set the overall vision and tone of the IT organization. The metrics are directly aligned with the vision to prove that the strategy is achieved. The organizational efforts to achieve success will be stagnant if the IT Strategy does not support innovation. It is important that the strategy has innovation built in to the overall vision of the IT organization so that the measurements support innovation.
Paul Needle says
5. What measures are being used in your organization? Do they make sense?
One of the most looked at measurements in an insurance company is the amount of new business written by an underwriter in one year. An underwriter’s bonus and salary are directly weighted against the performance to plan regarding new business. This has always seemed like a conflict of interest as it easy to put new business on the books by undercutting price particularly on high risk account. What isn’t commonly tied to an underwriter is their loss ratio which shows claims to premium written. It is common for an underwriter to change roles, territories, or titles so the loss ratio is easily forgotten. It would make more sense if an individual underwriters overall profitability was measured over time rather than just a premium figure.
Richard Flanagan says
Paul – does IT connect any of its measures to either, or both, of these measures.
Paul Needle says
Professor – IT does play connect new business numbers and thank you for reminding me to tie it back to class. We use software as a service in many forms one of which is salesforce.com. One way that salesforce aligns with IT and our business strategy is by providing a pipeline of new business opportunities. The intent is to categorize and prioritize new business opportunities. Salesforce also provides an indication of success and allows management to provide accurate forecasts. The clarity of the pipeline also helps in facilitating cross sells. Overall I would say that if we are addressing IT Metrics than I think IT in the form of Salesforce for our company aligns with strategy and does make sense as an IT Tool.
My issue is that it does not take into account the overall risk of new business. Higher risk would mean higher loss ratio’s making an insurance carrier unprofitable. My issue seems to be a cultural issue with all insurance carriers rather than an IT issue.
Mohammed Syed says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Since measuring is a process to accurately gauge if the business needs of an organization are met, it shows how resources are used, by tracking profitability, customer perspective, manufacturing productivity, and employee turnover. It is an overkill to measure every aspect, however it is very cost-effective when it comes to tracking IT activities. I would want my organization to use measuring around its own objectives and goals to ensure that everything is working efficiently, and that that business needs are being met.
Mohammed Syed says
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
It think it would depend on the organization, as to how many metrics that can be handled by the employees. The main ones, I would focus on are financial perspective, external factors such as customer perspective, internal factors in regards to the growth and learning of employees, and manufacturing productivity.
Mohammed Syed says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
I would focus specifically on the metrics that are essential for any organization. Especially the financial factors, as well as the internal and external factors. By focusing on few key metrics it would ensure that there is no sudden change within the organization in regards to their goals and objectives.
Michael Gibbons says
I think it would be interesting to go into an organization with a decentralized IT structure (each department does their own thing) and gather some data related to applications/system costs and tie it back to a measurement that shows multiple departments are using a similar application/system but all have chosen different applications/systems. Total dollars spent = X which may be a large amount over what would have been spent if the organization had a formal IT Enterprise Architecture. The main purpose of the measurement would be to eliminate or just bring light to the duplication of efforts/systems.
Mohammed Syed says
4. How can measurements become obstacles to change?
It would prevent an organization from focusing on qualitative aspect of a goal as their would focus mainly be on the quantitative aspect of the goal. For example my organization needs to meet a certain number of quota every month. As the deadline to meet the quota nears there is definitely a decrease in the quality aspect as all the employees only focus on meeting the goal.
Mohammed Syed says
5.What measures are being used in your organization? Do they make sense?
My organization does have measurements in place that track performance of the goals that need to be met each month. Reports sent out at the end of each month detail which site met the goal for the month .The report details employee performance to ensure employees are aware of their priorities. The report helps management by showcasing the areas of excellence vs. the areas that need improvements.
Anthony Quitugua says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Measuring is the only true way to gauge an organizations performance against a set value of metrics. The measurement gives you a data point in which you are able to quantify the performance of an organization for easier analysis on performance and efficiency.
Trying to measure everything is overkill. You should determine what your organizations priorities are and focus your measurements off of those. This will not only focus your efforts on those key metrics, but it will better focus the attention of whomever is monitoring your metrics.
My organization would decide depending on whatever priorities that we have established.
Anthony Quitugua says
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
That is completely dependent on what message I am trying to communicate to the enterprise/business as a whole. I would tailor my metrics and reporting to facilitate that communication.
The number of metrics is also dependent on the same. You want enough to drive justify your point, but not too much that you will lose the message with too much detail.
Anthony Quitugua says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
Include the critical metrics on the BSC. These would be determined by which metrics best align with the intent of the BSC. The remaining metrics could be added as an addendum to cover any follow on questions or add additional detail if needed.
Anthony Quitugua says
How can measurements become obstacles to change?
You easily become focused on improving the measurements instead of improving the process as a whole via change. By focusing on the measurements your main driver becomes the improvements of the metrics those measurements are derived from. You can quickly become boresighted and focus on the minute changes in metrics instead of looking at the bigger picture of what might have caused that change.
You become less open to making large scale changes because of the fear of how it might negatively effect your metrics.
Anthony Quitugua says
5. What measures are being used in your organization? Do they make sense?
I work in Risk Oversight so all I is look at measurements everyday. We have metrics that cover the entire spectrum of card transactions and fraud types. They all analyzed looking for opportunities to reduce loss, increase profits and improve the customer experience.
They actually all make sense. We are constantly evaluating what metrics are viable and what are not. We keep those that are useful, and retire those that are not.
BIlaal Williams says
1 Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Measuring performance is paramount because it helps demonstrate the effectiveness of a process. Providing the right metrics for measurement can help an organization monitor IT’s progress and ensure it is continually aligned with enterprise goals. Metrics provide the medium for all members of an organization to see how IT-related processes add value to the business. If a service or process is lacking, metrics provide a way for the organization to be alerted and plan on how to remove the deficiency. Metrics from a successful IT-related process can be used to communicate the value of IT to the organizations stakeholders and educate the enterprise on how technology helps achieve business goals.
The organization should decide on a few key measures that are meaningful to the enterprise and can help demonstrate what IT is doing and where its resources are focused. The measures should provide indication of progress in the following areas:
• Delivery of IT services in line with business requirements
• Operational success
• Optimization of IT assets, resources and capabilities
• IT compliance with internal policies
All goals and metrics should be approved by stakeholders and measured against agreed-on goals and metrics. Also, these metrics should be continually evaluated to make sure the proper performance measures have been chosen.
BIlaal Williams says
If your were a CIO, what metrics would you want? How many is reasonable to have?
Metrics that communicate the value of IT to the stakeholders and show how technology is enabling the business to reach its business objectives. If possible, there should be enough metrics chosen to show progress in the key areas outlined in the COBIT framework, namely:
• Delivery of IT services in line with business requirements
• Operational success
• Optimization of IT assets, resources and capabilities
• IT compliance with internal policies
These metrics will be used to prove the effectiveness of IT delivery as laid out in the IT strategy, reinforcing Its context in the enterprise.
BIlaal Williams says
Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
If metrics were unable to fit on the scorecard I would first want to make sure that key performance indicators are chosen and if any metrics are not needed. The organization should re-evaluate with stakeholders to see if we chose the correct metrics, and if these metrics accurately measure business results. As performance dashboards can shift over time, certain basic operational metrics can be discarded once IT has achieved the desired level of respect and credibility in the organization in favor of reports that highlight IT’s contribution to overall business value to help promote the idea of IT as a business enabler that adds value to the organization.
BIlaal Williams says
How can measurements become obstacles to change?
Measurements can become obstacles to change when the focus becomes mainly on improving numbers with no regard to the actual process or quality. This can happen when proper measurements are not in place. For instance, a factory puts a quota on the amount of widgets that are produced in a month, but does not measure quality. The organization meets the quota, but quality is lacking. With no metrics on quality, such as customer satisfaction etc.., the company will not know that its process needs updating, and will continue to produce low quality widgets.
BIlaal Williams says
What measures are being used in your organization? Do they make sense?
My company uses a BSC which has categories for financial, internal business process, innovation and customer categories. It attempts to connect vision, core values, strategic focus areas and operational objectives. The KPIs are looked at quarterly for each department, and an organizational scorecard is generated yearly and reviewed company wide at the annual town hall. Since we have undergone a recent merger, strategic initiatives that align with the “new” vision of the organization are highlighted.
Lezlie Jiles says
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
I believe this question would best be addressed by first identifying what key business goals and targets my organization is trying to achieve. On the lower level of implementations, I would use a process that was recently implemented in my department. We implemented a call center to handle all incoming departmental calls. Our organizational leaders believed that the call center would assist in capturing all incoming call and provide an overall customer satisfaction rating. Therefore, some of the KIP’s that I would utilize in measuring this service outcome would be quantitative, process, output, and directional indicators. I believe these KIP’s would best assist in measuring the value of implementing the call center. As for how many are reasonable, I would have to say whatever was needed to best identify the value, so if that meant all of them then so be it.
Lezlie Jiles says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Implementation is performed to improve business process, therefore the measurement of a particular implementation identifies the efficiency and effectiveness. The interest in measuring is important because it assists in identifying whether the implementation is meeting the expected goals as it relates to financial interests, customer satisfaction, business processes, and the organization’s growth. No, I don’t believe it is an overkill. In my opinion, it’s the best tool to best quantify IT’s value as it relates to project implementation, completion, and ongoing performance.
Heiang Cheung says
1. Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
There is so much interest in measuring because it help evaluate the business. You need measurements to see how effective a certain project is doing. You could measure whether or not a project is helping with efficiency, productivity etc. it is an overkill to measure everything that is why you want to measure key performance indicators. Measuring would also take up too much time.I would want my organization to decide to measure things that align with the business goals and objectives
Heiang Cheung says
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
If I was a CIO I would just have a couple metrics. It depends on what type of company I’m in but I think the most important is the financial metrics making sure the cost of certain project don’t outweigh the benefits. Having a customer metric internal and external making sure everyone is happy. Also process metrics making sure things are efficient and effective.
Heiang Cheung says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
If I have more metrics that can fit on one balance scorecard than I would have to evaluate the metrics I have a little more and see which one fit better for the business and is more important because having too many metrics would make you lose focus on what is really important.
Heiang Cheung says
5. What measures are being used in your organization? Do they make sense? I see a bunch of measurements especially as an accountant because we need stuff in a timely manner to be able to do our accruals and monthly close. We could see how fast account payable is processing invoices. We could also see how many requisition and purchase orders are still outstanding. These measurements in a way helps because if someone is not doing what they are supposed to be doing we could just ask them what’s going on this particular invoice been sitting for too long. If need be we could escalate and have proof. Measurements also put some accountability on the worker.
Heiang Cheung says
4. How can measurements become obstacles to change? Measurements could be an obstacle in change because it could make the organization only focus on the measurement and not the whole picture. There things that can’t really be measure accurately that could affect decision if decisions were only based off numbers.
Lezlie Jiles says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
Assuming that I had more metrics than I could fit on a balanced scorecard I would select the methods that worked best for my organization. As stated in our readings Heather Colella points out that “less is best”, and the key components of a scorecard are the vision, operational success, risk, and telling a story. These key components would assist in identifying how IT is being utilized, it’s continued improvements and any risks/successes associated with the outlined vision.
Lezlie Jiles says
4. How can measurements become obstacles to change?
Measurements can become an obstacle to change when the organization becomes more focused on the measurement then the quality of a preformed function. Again, I am going to use my call center theory because this is the most relevant issue within my organization today. One of our department utilizes an internal call center operation, which in my opinion is the pits. On a given day they may answer five hundred calls out of six hundred. The department measures their success on the volume of calls answered. Believing that five hundred customers were assisted, therefore their reaching a larger base of their incoming calls. However, the service action was never completed, therefore those five hundred people are calling back again and again. This goes on for about three months, or until the last possible minute where it’s either too late for anything to be done or the department is now so overwhelmed that those customers are now coming in person. The department believes their call center is working at an optimal level because the calls are getting answered (e.g., measurement). However, this is not a good measurement of the process, which in turn has hindered the department from recognizing a needed change. This cycle goes on every year, and the department still believes they are reaching their expected goal and refuse to change. Unfortunately, the only people who actually suffer are the customers.
Lezlie Jiles says
5. What measures are being used in your organization? Do they make sense?
We utilize measurements for the quantity of work we receive, and our efficiency in processing the work. We are then able to identify if our employees are processing transactions at an optimum level and if our customers are satisfied. Yes, fortunately, and unfortunately these measurements are working. By measuring our processes we are able to adjust or correct any inefficiencies quickly to best serve our consumer base. It has also allowed us to create new processes to maintain customer satisfaction. I say unfortunately because over the years these measurements have allowed us to be so efficient that our face to face customer interaction has reduced so much so that we are now seeing more online transactions and inquiries. Nevertheless, this was indeed the focus of my department’s objective a few years ago so I would say that it is indeed working.
Tamekia P. says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
There is interest in measuring because there needs to be a benchmark for output/production. Similar to the way that we need to receive a grade for this course, the grade quantifies our production in this course on some scale (whether or not we agree the scale). At a minimum, it allows comparison especially between things that are not considered to be similar.
It is overkill to measure everything but you’ve also can not chose to measure nothing. There needs to be a happy medium. Measuring is hopefully a tool used to determine where to focus efforts, gain efficiency, etc. I️ would want my organization to chose the most meaningful metrics and explain the metrics to the individuals within the organization. It is helpful to give people an explanation of what they are being measured against and why. It’s one thing to be given a target, another to be given an explanation on why the target helps the organization meet its objectives. I️ would want organization to chose targets that meet objectives and are also possible.
Tamekia P. says
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
As a CIO, in this current environment, I️ would want to focus on cost efficiency metrics. Where are we utilizing resources efficiently? Where are we creating inefficiency or redundancy?
The number of metrics should be weighed against the cost of measuring. When monitoring of metrics > then output then the metrics should be scaled back. I️ would want to start with a list of metrics and then determine the cost and time necessary to monitor each metric against its value. Where value > than cost and time then those are the metrics we should utilize as an organization.
Tamekia P. says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
You should prioritize the metrics. Only select the items that add the most value to organization. Depending on the department, it may be more important to select the metrics that drive growth/add value than monitoring metrics that maintain the status quo.
Tamekia P. says
4. How can measurements become obstacles to change?
Measurements can become can obstacle to change when everyone becomes focused on being measured. The worst case scenario is that people start trying to game the system. They become focused on the outcome and not the process.
Tamekia P. says
5. What measures are being used in your organization? Do they make sense?
For my department, one of things we measure the open and closure of control deficiencies and how long the issue has been open. It’s an interesting metric for us because we have no control over when management remediates the control deficiency. We can provide our recommendations but it is up to them to facilitate the correction.
MEA02 is most closely related to the functions of my department. We are part of how management performs independent assessment of the internal controls in addition to audits performed by external auditors and internal audit.
Michael Gibbons says
Tamekia – does your department have a reporting structure where you can elevate items where you feel management is taking on an excessive level of risk by not performing the remediation?
Tamekia P. says
The Star Ambulance Case: Take Two
Project completion per consultant – Monitor number of projects completed by consultant and the amount of time spent on either monthly or quarterly basis
Consolidated budget – Monitor total IT budget spent across the organization. Organize the budget for each group – Aviation, MedOps, etc and allocate budget accordingly.
Vendor service monitoring – Monitor vendor service visits based on need behind visit and cost.
Michael Gibbons says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
There is a high level of interest in measuring because organizations need a way to determine is they are meeting their goals and objectives. It is a way to help define success or failure in terms everyone can understand. I do think it is overkill to try to measure everything. The organization should decide by starting with items that tie back to strategic objectives, financial goals, and customer goals. Once a solid repeatable process is in place, the organization can start integrating the measuring into other critical processes (Risk Management, Key Performance Indicators, etc.)
Michael Gibbons says
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
Vulnerability Management metrics – how many total vulnerabilities exist? How many vulnerabilities exist and are rated high and are exploitable? How many vulnerabilities are we remediating on a weekly, monthly, yearly basis? Help Desk – how many support desk calls are we receiving? How many of those calls are we resolving on the first attempt? What percentage of calls deals with password resets? It would be reasonable to keep going with as many as possible that provide meaningful information to IT and the organization. These metrics can show the value of IT and can used as the reason to upgrade/replace a system or process because there would be good data behind a recommendation.
Michael Gibbons says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
You could try to group the metrics at a higher level and provide higher level statistics. Another solution would be an electronic report with drill down capabilities for the levels of the organization that want to see the true details and not just a summary report.
Patrick DeStefano (tuc50677) says
I’ve seen several of these drill down scorecards. They can be very functional and educational. I was on a group which always used to have manually created scorecards in PowerPoint or Excel. One day, our management decided to build a dashboard scorecard. Once it was up and running, it was great. The only drawback I’ve seen so far is that it can take a lot of upkeep if the data changes a lot, but if done correctly and effectively, this type of scorecard can be a real winner.
Michael Gibbons says
4. How can measurements become obstacles to change?
Measurements can become obstacles to change by showing you are meeting the objectives of the organization while you know there are ways to improve a system/application/process etc. You could make the case for the change and the organization could look at it as why spend the money or put the resources into this when it is already meeting our expectations (if it isn’t broke, don’t fix it). They may wish to focus on area’s that are not meeting their key performance indicators.
Michael Gibbons says
5. What measures are being used in your organization? Do they make sense?
1st call resolution by the support desk. Total customer logins per channel (online, mobile). Virus/malware detections per month. Total discovered vulnerabilities per month, total remediated vulnerabilities per month.
They make sense from the standpoint that IT is able to show some areas of success. As certain processes mature across the organization, it will be interesting to see these metrics tied to specific business processes and key performance indicators.
Michael Gibbons says
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and post it on the class blog .
Consultant cost per month per department – minimal, meets budget, exceeds budget
Project status – on time, falling behind, at risk
Systems supported by IT – #
Systems not supported by IT – #
Issues reported on Systems support by IT – #
Average time to resolve –
Issues reported on Systems not supported by IT – #
Average time to resolve –
Patrick DeStefano (tuc50677) says
Why so much interest in measuring? Isn’t it overkill to try to measurre everything? How would you want your organization to decide?
It is overkill to try to measure everything. That being said, in order to be successful, any organization needs to be able to measure specific things which can help the governance team learn how the IT organization is doing. These metrics are called Key Performance Indicators (KPIs). You can measure everything you want, however a great deal of the measurements will not have any real meaning to the health of the organization overall. The key metrics will allow the governance team to make decisions on whether to divert resources to a struggling area or even, in some cases, pull the plug on a certain project. I would want my organization to look at what the overall business goals are and then select certain KPIs to help determine whether the changes IT is making are helping or hurting the goals.
Heiang Cheung says
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and post it on the class blog .
Star ambulance
Finance
IT expenses as a percent of total expense
Operating cost
IT expenses per department
Customers
Response time for request
Internal
Effectiveness of projects
Efficiency provided by projects
Jonathan Duani says
The Star Ambulance Case: Take Two
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and post it on the class blog.
I would first look at the project’s and get some information from them. I would try and keep track of their status. I would make sure that they are on time, on schedule and the budget is correct. Then I would keep track of the issues that were submitted to IT by different systems both supported by IT and not supported by IT. I would also keep track of the time it takes for customers to receive care. This way we can make sure that we are holding the correct standards. Finally, I would keep track of total money spent throughout the out the organization so that we can make sure we are on budget.
Patrick DeStefano (tuc50677) says
The Star Ambulance Case: Take Two
Reread the Star Ambulance Case and think about what metrics you would want on your BSC if you were the CIO. Mock up what your BSC would look like and post it on the class blog .
% Time Fixing Defects – Consultants vs. Full Time Employees
% Time Project Work – Consultants vs. Full Time Employees
Deviation from budgeted hours – By Project & Overall Consultants vs FTE
% Time External Vendors Called In To Assist – Defect Fixes vs Project Work
Forecasted Budget vs Actual Spend
I’m sure I’m missing a few, but these are just a few of what I would consider important KPIs for the STARS Ambulance case. They need to look at their metrics on having consultants vs full time employees to do the work. If the work is continuous over long periods of time, it may make more sense to hire a few more FTEs than to keep paying the higher costs of the consultants.
They should also be sure to look at budget deviations. In order to be able to run a business successfully, you need to be able to forecast efficiently and effectively. I just had a discussion today where an IT manager mentioned that he views a deviation of less than 15% very good forecasting. Looking at this case, I would assume that their forecasting can also use some work.
Brandan Mackowsky says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
Measuring attracts a lot of business interest because it allows an organization to understand its progress and effectiveness in conducting its business procedures. Measuring is extremely useful in allowing a business to view its past and current status to improve upon its future for each process. When asked if it is crucial to measure everything is determined by what resources the business has available. While it would be great to have measurements to backup reasons and methods as to how to improve a particular process, the amount of resources used to measure this would be far beyond that of a typical organization. The way the organization should decide what to measure should be based first on what resources they have available to dedicate to measuring metrics and what key functions exist that should be measured to ensure success, effectiveness, and further growth.
Brandan Mackowsky says
2. If your were a CIO, what metrics would you want? How many is reasonable to have?
As a CIO, the metrics that I would want to see would be both financial based and operational based. From a financial perspective, I would want to see how the company is performing by maintaining its budget and performance to appease shareholders. By ensuring IT remains on budget and shareholders are happy, the IT side of the business is contributing to the company’s success, and if it is not, I can develop methods to improve it to ensure that it does. By focusing on operational metrics, I am able to determine who is doing what, how it helps the company succeed, and if the positions are necessary and are performed effectively to ensure the company’s success in IT. With this, I know where I can add or subtract resources as well as determine what may be necessary to increase productivity and success.
Brandan Mackowsky says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
Since organizations, both small and large, can have many things that can be measured, it is key to set a foundation for the balanced score card to determine the measurements that closely resemble the goals of the organization. For example, if an organization wants to see rapid growth of customer relationships, key measurements it may want to see are how quickly its help desk answers support questions or how marketable its IT products are compared to the competition. As long as the scorecard includes vision, operational success, risk, and an IT story relating to the vision, a common theme will determine which metrics are necessary for the balanced scorecard at a given time.
Brandan Mackowsky says
4. How can measurements become obstacles to change?
Measurements can become obstacles to change by providing results from metrics that the company deems adequate in what it wants for its current performance. If an organization deems the metric results adequate, there is no drive to push it to change. With a positive view on the metrics that come back, an organization will happily continue performing its tasks the way that it was due to its satisfaction with the metrics results. With no reason or indication to improve, why would an organization want to alter something that works, thus leading measurements to become a change suppressor.
Brandan Mackowsky says
5. What measures are being used in your organization? Do they make sense?
My organization uses measurements such as time management and IT budget measurements to ensure projects are completed on time and under the hours budget. Through this, my organization gains an understanding of how quickly projects can be completed and how many resources are required to complete future projects. This allows my company to understand how efficient its employees are in completing tasks and how reasonable it is to assign resources to a particular project to guarantee completion by a specific date. They make sense because it allows the organization to set specific deadlines that should be able to be met based on its allocation of resources to the projects.
Jason M Mays says
1. Why so much interest in measuring? Isn’t it overkill to try to measure everything? How would you want your organization to decide?
I believe the need to justify IT in organizations that have poor IT governance and the continuous refinement and specialization of IT in the modern age lead to the desire to measure more than can be mentally digested.
The hype cycle and the potential economic graph in “The IT Balanced Scorecard Revisited” by Alec Cram and the potential economic graph in the Gartner research publication are great visuals of the uphill battle a CIO faces’ when they confront an E-board with a poor understanding of the effects of information system within the business process. CIO’s gather evidence with the hope that their efforts and department won’t have their budgets reduced, maintain interest and maintain the buy-in power necessary to build up IT governance. At the same time e-boards and business users can find themselves lagging in understanding the true scope of the relation of IT and business processes or questioning the value of IT with in their company despite seeing evidence of other organizations embracing IT.
Both sides act with urgency to explain or understand the value of IT in companies without a since of IT governance and. I think this leads to an overkill of information. Better yet I think its easy to fall in to the getting information for information sake. With so much to learn it can be tempting to not set limits on what you want to know.
I would want any organization I am apart of to take stock in its understanding where it ranks the value of the IT in relation to its business processes. From there, a strategy can bed develop to get the org from where it is to where it needs to be. Only then can you decide what would be necessary to track and what metrics are over kill.
Jason M Mays says
2. If you were a CIO, what metrics would you want? How many is reasonable to have?
I have found that the rule of 3 is effective when trying to relay a message or tell a story. Metrics are just a mass of numbers and buzzwords without a narrative. As a CIO, I would look to tell the story of how to choose goals that represent the right thing to do with 3 metrics that can provide evidence of progression in that story. To illustrate how to do those goals the right way I would choose up to 3 more metrics per goal that can support the efficiency of our operations.
Jason M Mays says
3. Assuming you have more metrics than can fit on one balanced scorecard what would you do? How would you handle it organizationally?
Tailor it the level of IT credibility of my organization. Metrics should be prioritized based on the IT governance, IT strategy, and business strategy aligning. For instance, I would remove metrics related to workplace innovation and strategic sourcing when my company has a high level of skepticism. The metrics are a measure of great goals, but those goals would be to advance for my company that is still trying to grasp IT value in the organization. It doesn’t make sense to track opportunities you can’t take advantage of. It’s a waste of resources to prioritize the metrics of threats that are not applicable to you even if they are applicable to your industry.
Jason M Mays says
4. How can measurements become obstacles to change?
If the measurements do not demonstrate IT’s progress and results relative to business goals. There are many red herrings that can prove both interesting and a waste of resources. Spending man power, money or time on a non-important metric means there is an important metric which didn’t receive the attention it deserved.
They can also become obstacles if the measurements are not relatable to the IT maturity of the audience. Spending time on metrics that are over the heads of IT immature stakeholders can give the perception of the IT department is not adding value. Spending time on metrics that don’t communicate more progressive goals wanted by stakeholders with a strong IT strategy will indicate that IT isn’t using resources responsibly.