How Can You Tell if an Agile Coach is Effective?

The key to success in agile coaching depends on the coach’s ability to measure impact

This article was originally published in Emergence Magazine.

The 2022 State of Agile Coaching Report focuses on one question: “How do agile coaches (and the organizations they serve) measure impact?” A joint effort of Scrum Alliance, ICAgile, and the Business Agility Institute, the second annual release of the State of Agile Coaching aims to shed light on the emerging profession of agile coaching. The intended audiences are agile coaches and the people who are looking to hire agile coaches. No matter where you fall on that spectrum, it’s important to understand how much of a difference agile coaching is making to your organization’s ultimate goal: to deliver value to your customers.

As one of the creators of the report, we’ve had a sneak peek at the data. We discovered a key fact that doesn’t surprise us, but does concern us: Not all coaches know where to begin when it comes to measuring success. In fact, 9.5% of team coaches and 5% of enterprise-level coaches reported struggling to measure their impact at an organizational level. The good news, though, is that 98% of respondents reported that coaching had a measurable impact on one or more factors in their organization. The report also shows a direct correlation between a coach’s ability to measure and articulate impact with that coach earning a higher income. So it’s clear that the ability to gauge effectiveness is pivotal to a coach’s success.

The ability to gauge effectiveness is pivotal to a coach’s success.

To understand what it means to be an impactful coach and what metrics are truly meaningful to an organization, we reached out to two people on opposite sides of an agile coaching engagement: an agile coach and a CEO.

Kemmy Raji, Scrum Alliance Certified Agile Coach

The Agile Coach

Nigerian-born and British, Kemmy Raji is currently a Scrum Alliance Certified Enterprise CoachSM based in Canada. Her background is in management and waterfall project management. She now works as a scrum master/agile coach and executive coach. Raji spends her spare time mentoring and coaching people who wish to explore ways to grow and also supporting children from disadvantaged backgrounds in Nigeria to give them access and support to stay in education.

Howard Sublett, CEO

The CEO

Howard Sublett has a tremendous breadth and depth of experience in leading companies and implementing a variety of agile practices. He most recently served as a CEO at Scrum Alliance. He also has a background as an agile coach and leader for several agile consultancies, including Solutions IQ/Accenture. Sublett is known for being a culture champion and community builder, and was the host of the popular Agile Amped podcast series.

How Do You Measure the Impact of Agile Coaching?

Organizations often hire coaches without having tangible goals in mind other than to “improve.” This ambiguity makes measurement difficult for both the coaches and company leaders. Raji explains, “Measuring the impact of agile coaching is subjective and can be tricky. I have often received no direction from the management aside from “‘Increase the teams’ velocity.’” Sublett agrees with the need for clear direction: “Companies should hire agile coaches with achievable goals in mind, or at a minimum, be able to point to areas in which they feel the organization needs improvement. (i.e., our cycle times are far too long).” Measurement and goal setting, however, are keystones for success. Sublett says, “ A failure to define the goals with a coach/client can lead to wasted time and capital as well as a lack of tangible results.”

Defining what to improve is just one way to measure the impact of agile coaching. Raji recommends shifting the collective language that the teams and organizations use to “center around common goals and how to collectively achieve those goals.” Sublett adds that goal-setting should be part of the coaching agreement: “Many times these goals are co-created with the coach as they enter an engagement. Measurement onf the success of the coaching engagement should be based on those agreed goals.”

“Measuring the impact of agile coaching is subjective and can be tricky.” – Kemmy Raji

What concrete metrics have you used to measure coaching success?

Most organizations measure effectiveness based on overall organization goals like Key Performance Indicators (KPIs), Objective Key Results (OKRs), and similar metrics. Raji says, “These kinds of ‘Concrete Metrics’ often originate from the vision of the organization.” Despite the prevalence of using these types of org-level goals as measures, this year’s State of Agile Coaching Report found that coaches who are measured on overall product improvement tend to have a greater impact on improving delivery or operational processes. Raji says that, whatever the coach is measuring, it’s imperative that measurements are taken before and after coaching takes place. “However, you might first define the ‘“why’” and/or “‘who’” you are measuring that will determine if the coaching is a success or not.

Raji says that, whatever the coach is measuring, it’s imperative that measurements are taken before and after coaching takes place. “However, you might first define the “why” and/or “who” you are measuring that will determine if the coaching is a success or not.”

“It's imperative that measurements are taken before and after coaching takes place.” – Kemmy Raji

Sublett echoes this approach. “In the beginning of any coaching engagement, there is usually a discovery phase where you capture all of the metrics the organization is currently using to measure success. You also measure all aspects of the goals of the engagement.”

As far as measurement tools go, both Sublett and Raji have had success with using surveys and tracking Net Promoter Score (NPS). Raji and Sublett can also rattle off a whole list of other potential measures, depending on what is of highest priority to the organization.

“I’ve seen organizations use cycle time, time to market, bug count reductions, employee engagement scores, predictable delivery, specific surveys on effectiveness of coaching, strategic alignment, employee retention, whole org strategic goals, and more,” says Sublett.

Raji adds to the list:

If the focus is from the organizational leaders’ stance, I use

360° Interviews
Post-Coaching Evaluations
Return on Investment (ROI) - % ROI = (Benefits Achieved – Coaching Costs) ÷ Coaching Costs X 100

If the focus is from the coach’s stance, I use

Coaching plans in place
Number of teams lift off
Number of people who access coaching
Maturity assessments (per pod)
Agile delivery metrics (per pod)
Final coaching reports

What kind of organizational improvements do leaders look for when they hire an agile coach?

There are a multitude of reasons to hire an agile coach, but a company's motive to do so may influence whether or not they’ll see the results they’re looking for. Raji explains that while some changes are market driven, the need for coaching can also be driven by ”regulatory changes, internal factors, or unique situations like the COVID-19 pandemic.” Sublett adds an unusual wrinkle to the usual list of reasons: “You also will see leaders hire agile coaches because it is the ‘in’ thing to do, or in order to increase perceived company value for a potential acquisition. This can be the most frustrating for a coach in terms of measuring results.”

Sublett has seen companies bring in agile coaches to address problems ranging from team-level issues such as “missing sprint goals and increasing bug counts,” to broader problems, such as “organizational design or lack of alignment between teams.”

Many leaders report hiring agile coaches to “improve culture.” How do you define cultural improvement?

Nearly 40% of respondents to the State of Agile Coaching Report cite difficulty changing company culture as a challenge for agile coaches. Yet 32% of respondents felt that the biggest impacts they have on organizations have to do with shifts in agile mindset and/or culture. Both Sublett and Raji offered solutions to this obstacle.

Company culture starts at the top. It’s nearly impossible to improve company culture without the buy-in and standard-setting from leadership. “Culture, they say, eats transformation for breakfast,” quips Raji. “Organizational agility isn’t possible unless the leader has an agile mindset,” cautions Sublett.

Often, a leader's early indication of a problem shows up in culture surveys. Once a problem is discovered, it takes some time to turn it around. Raji often has to remind leaders that cultures do not change overnight: “It takes time, patience, strong communication skills, and most importantly, trust between managers and their teams.” It’s also essential, Raji says, to have “a leadership team that actively seeks out opportunities to reduce waste, improve flow through the value stream, and increase the focus on the customer. What got you here won’t get you there.”

Sublett laments what he sees as misguided attempts to inject joy in the workplace. “Joy doesn’t come from foosball tables or new espresso machines. Joy happens when people understand what they should be doing, why they are doing it, and have autonomy in how to get it done.”

Raji has also been brought in to improve a variety of factors, including, she says:

Continuous delivery
Developers not able to handle frequent changes to product backlog items
Improvement in quality
Increase in the team’s velocity

“Joy doesn’t come from foosball tables or new espresso machines. Joy happens when people understand what they should be doing, why they are doing it, and have autonomy in how to get it done.” – Howard Sublett

Any organization-wide transformation effort is going to come with a hefty side of politics, says Raji, so your strategies have to adjust accordingly. Sublett agrees, adding, “Organizational coaching involves a much wider skillset and permissions for the coach [than team-level coaching]. To be successful at implementing organization-wide change, coaches need experience with systems thinking and organizational design.”

Where are the challenges and impediments to effective coaching?

Raji and Sublett are in complete agreement with this year’s State of Agile Coaching Report: The biggest impediment in agile coaching is leadership. Sublett points to the problem as being one of perception: “Leaders who see agility as only applying to others and not themselves can severely impact the success of any agile adoption.” Raji has experienced the same blindspot in leaders: “Leaders have this notion that the delivery teams are the only ones that need coaching in order for transformation to occur.” Leadership isn’t the only challenge to successful agile coaching. Raji lists several other factors that come into play:

"Leaders have this notion that the delivery teams are the only ones that need coaching in order for transformation to occur.” - Kemmy Raji

Pressure from management and time constraints:

It can be hard to coach a team when that team lacks knowledge or awareness on the benefits of coaching. It’s also tough to find adequate time to coach when teams are under huge pressure to deliver within a short timeframe.

Undefined roles and responsibilities:

When the roles and responsibilities are not well defined, it creates confusion, builds a blame culture, and prevents people from taking ownership. When this happens, it can be hard for people to open up to the coach and to build relationships that help transform the teams.

Lack of buy-in or understanding of organizational goals on a team level:

Often goals are set by top management, not the teams.

Some of the other challenges/impediments are:

A coach assigned too many teams
No coaching agreement or lack of buy-in from the team and/or organization
Frequent changes to coaching goals or organizational goals
Failing to provide adequate support for coaching and growth

Key Takeaways

An agile coach can impact many facets of an organization for the better, but coaches and leaders need to align at the beginning on exactly what the goals are.

In order to calculate impact, an agile coach must gather benchmark data before they begin coaching. Capturing a baseline is a critical first step in determining impact.

There are good reasons to hire an agile coach and there are less authentic reasons. If an organization wants to see improvements, they need to be honest about their motives. Measures can then reflect the outcomes they truly want to achieve.

Company culture and agile transformations must first happen at the top. Hiring a coach isn’t enough if leadership is not willing to adapt an agile mindset and create a positive environment themselves. Recognizing that agility is not just “a team thing” is often the first step towards a shift in the leader mindset.

Sublett sums it up this way, “In any coaching engagement, measure everything. Organizational systems are human systems and are dynamic in nature. As you engage, constantly measure and adapt your coaching plan accordingly. Your coaching plan, measurements, and results should be transparent to your key stakeholders. The best coaching engagement plans are co-created with your key stakeholders.”

Interested in the data? Read the full 2022 State of Agile Coaching report.