How to Measure Scrum Success

Image of an agile manager or director celebrating success metrics for his Scrum team

It's Easier Than You Think

If you're contemplating making the investment in Agile processes, training, tools, and coaching, it's likely you have some goals in mind. For some, the priority is faster implementation, for others it's increased efficiency. Still others hope to be more competitive or spark more creative collaboration. In most cases, making customers happy by raising the quality of a project, product, or service is a key goal.

It's also natural, though, to worry about return on investment. How will you measure whether the switch to Scrum is bringing about the desired results? The answer, say executives who've navigated this transition, is simpler than you might imagine. "How did you measure the success of your last project? Whatever you've done in the past, that's what you'll do now," says Jim Starrett, a Certified Scrum Professional® and vice president of Bottomline Technologies. "What people worry about is whether they have to learn new measurement techniques, and the answer is no."

Tracking Metrics

Deciding just what to measure starts with determining the motivation for a change to Scrum, says Starrett, who over the past few years has transitioned a series of companies purchased by Bottomline. "Drivers can be one or many, such as the pressure of competitive forces, reduced revenue, quality not where you want it to be, or shareholder dissatisfaction with growth. I always try to articulate internally to teams why we want to transition them over."

Once your overall goal is defined, measuring progress is straightforward. "Success with Scrum is measurable by whatever metrics you would normally use," says Starrett. "Those might be velocity, however you define it; improved efficiency, such as fewer cycles in the production process; and improved quality, such as fewer defects — whatever key indicators you hope to see improvement in."

As Scrum spreads rapidly from software into other fields, it's clear many share the same goals, says Brad Swanson, a Certified Scrum Coach®, Certified Scrum Trainer®, and founder of Agility 11. "The same general categories of outcomes tend to be important regardless of industry," says Swanson.

Setting a Baseline

To make an accurate before-and-after comparison, you'll want a clear picture of how things are working today. If you haven't done this already, Swanson suggests setting some key metrics and applying them to past projects to obtain baseline data for comparison. "If you don't have good data, it's important to back up and look at recent projects and deliverables, and get baseline data as a starting point." Swanson suggests asking a few key questions:

What is our time to market? How long does it take from when we start a project until we deliver something valuable? "You might determine average time to market based on the minimum and maximum times it's currently taking to get a project deployed to production or released to customers," says Swanson.
How do we determine quality? "Look at how many defects are found after development, and the severity of those defects."
What is our customer satisfaction level, and how do we currently measure this? Polls can be a good measurement, Swanson says. "One popular strategy is to ask customers — and staff — to rate how well they perceive things are working according to a numeric scale. Then measure this over time to see how it's trending."

Once you have a set of indicators for speed, efficiency, quality, or any other crucial metrics, you can use exactly the same ones as you transition to Scrum.

Measuring Innovation and Competitiveness

There are a couple of metrics that most companies consider high on their list.

"The ability to innovate measures your organization's ability to sustain its value in the future," says Swanson.

"Gone are the days of two-, three-, four-year road maps," adds Starrett. "Today's executives and managers need to be able to innovate in response to constantly changing customer requirements. A measure of innovation and competiveness is costs to goods sold: How much did it cost to generate this sellable feature or product? Then ask, how marketable is it, how in demand, how unique from competitors? Lower costs and increased feature function are key to competitiveness against others."

Measuring Improved Performance

Companies that provide services rather than products often see the success of Scrum manifesting in better performance in areas such as sales, marketing, and output volume, says Swanson. "There's a pretty strong movement to apply [Scrum] techniques in the areas of online marketing, recruiting, consumer product design, sales, and business development. In these cases, companies are measuring success in terms of increased sales volume, better consumer response, or speedier turnarounds, such as the time needed to put together a marketing campaign."

Measuring Employee Engagement

One sometimes-unexpected outcome of the Agile process is a deeper sense of investment and engagement throughout the company — and that's a key metric indicating organizational health, for many boards of directors. "The board holds the CEO and the executive committee of the company accountable for employee engagement scores," says Lisa Hershman, herself a Scrum Alliance® board member and also the founder and CEO of the DeNovo Group and coauthor of Faster, Cheaper, Better: The Nine Levers for Transforming How Work Gets Done.

Employee surveys are the traditional vehicle for measurement, says Hershman, though sometimes less tangible signs of increased engagement also crop up. "One executive recently told me that after six months of Scrum, his employees had learned more about the product they were working on than they had in the previous ten years they'd been on it," says Starrett, underscoring Hershman's observation that Scrum gives employees a "line of sight" that helps them better understand how they're contributing to the common good.

"On a good team, everybody's aligned around the same goal. Everybody's participating," says Hershman. "As crazy as it sounds, you wouldn't believe how many companies can't get that done. Scrum gives them the methodology to bring a team together."

The result of this improved understanding of context and more global product concept, says Starrett, is often an increased motivation for success throughout the company.

Measuring Tactically

Hershman acknowledges that even with metrics in place, champions for change may face pushback as to whether Scrum is in fact the change that's responsible for improved performance within a busy organization. "There might be submeasures that are more analytical that can tell you how much is directly related to Scrum," she advises. "Think tactically. For example, do you want faster innovation to market? Cycle time is a more granular measure. Think about what Scrum is supposed to be doing for you — how you sell it to others in your company — and measure that."

Can Scrum provide your company with measurable improvements? Work with a Certified Scrum Coach® to find out more. But if you ask Jim Starrett, the answer is yes. "I've transitioned over 50 teams," he says, "and I've never had a single team tell me they wanted to go back to whatever methodology they used to use before agile."

Download the PDF

Related Article

Article: What is Scrum Team Velocity

RL_143_measure-scrum-success