How do managers measure software development productivity?
This question has always been one asked by leaders in the tech industry. Today, though, in a post-covid world marked by a sudden and near complete shift to remote work, the question is more important than ever.
In an attempt to find a few strategies for identifying, creating and tracking productivity metrics, we turned to a trio of the brightest minds on the subject. Krishnar Kumar of Exathink, along with Mickey Mantle and Ron Lichty, co-authors of Managing the Unmanageable: Rules, Tools, and Insights for Managing Software People and Teams, joined us in the latest TechKnowCon session, sharing a wealth of wisdom. Here are a few takeaways from their fantastic presentation.
Defining productivity is trickier than it seems
Productivity can have a variety of definitions, depending on who is defining it and in what context.
“I find that whenever I talk to people, what they want to understand is whether their team is productive, but they all mean different things by it,” explained Kumar. “The CFO, for instance, has a valid reason to think about productivity, but their notion of productivity is probably from the lens of cost. They ask, ‘How big a team do I need to deliver what I need to do profitably? Engineering hiring is expensive now. If I want to add five engineers, it costs a lot more than it used to.’”
This, of course, stands in stark contrast to the productivity questions an engineering manager would ask. As a result, it’s often helpful to work backwards and ask what you are trying to achieve before deciding what and how you are going to measure.
“You want to understand what outcome you are looking for before you think about productivity,” said Kumar.
“There are two aspects of productivity. The first is deciding if it’s the output or the outcome that we’re measuring.”
Kumar goes on to add, “If you think of making software like making widgets, then you will focus on outputs – things being produced – and end up measuring things like amount of code produced, number of pull requests merged etc.. without considering whether that is creating any value. So how do you figure out the connection between the work an engineering team is doing and how they deliver value? The other part is the quantitative piece, which asks how you measure all this stuff. That’s a much less developed field, but my argument is that it is something that needs to happen much more rigorously now because we need better data to understand this in our new environment where people are disconnected from each other day-to-day.”
How does an organization maximize productivity?
The panelists offered a concise, 3-step framework to broadly consider this question:
- Ensure people, processes, and codebase are all working effectively to deliver value
- Minimize time and effort spent on work that does not add value
- Minimize the time and effort needed to do work that adds value
The importance of good data
With remote work now the norm rather than the exception for so much of the workforce, the old methods of gauging productivity no longer serve. “One of the things that you get when you’re walking around and managing everyone in one place is a sense of what is normal,” said Kumar. “There’s a lot of human judgment that comes in when we are in place, on a team, in a room, and everyone’s working, typing away at their keyboards.”
Good data can go some way to making up for the absence of this in-person benefit.
“I think many more people are going to have to start and learn how to sort of incorporate non-verbal, visual data oriented views of how work happens and form their intuitions. Becoming more data literate in engineering management can actually help us become better managers.”
What motivates programmers?
Mickey Mantle and Ron Lichty went deep on this topic, offering a surprising finding that upended a few of our set expectations.
“When we wrote our book,” said Mantle, “we studied different motivation theories. And we identified one, which we liked a lot, which was by Frederick Hertzberg. What he identified was the fact that there are a set of foundational factors that if they’re there, they don’t really motivate the team or the people, but if they’re not there, they hold it back.”
These foundational factors are:
- Status
- Security
- Relationship with subordinates
- Personal life
- Relationship with peers
- Salary
- Working conditions
- Relationship with supervisor
- Company policy and administration
- Supervision
Mantle was careful to clarify that these factors are separate from the actual motivators, which do, in fact, incentive productivity. Those are:
- Achievement,
- Recognition
- The work itself
- Responsibility
- Advancement
- Personal growth
“Google put out a really great book called Software Engineering at Google” clarified Kumar. “In it, they said that, yeah, we do a lot of measurement, but what we look for primarily is developer satisfaction. Once you have developer satisfaction, you will get productivity.”
“Remote is a strategy, not a tactic”
Kumar also noted that companies that have successfully set up remote work infrastructures have done so holistically. “If you look at companies like BaseCamp or GitLab… they are completely remote,” he explained. “They set everything up so that there is no HQ, and your whole way of working has to be different. If you look at remote as a kind of a “proxy by electronics” for doing things that you would have done in person, then it’s less effective. I think that a lot of people even now think that, okay, well we are doing remote now, but we plan to be back in the office as soon as we can. In that case, I don’t really consider them to be remote. They’re only remote because of circumstances. But a company like BaseCamp has always been remote. They don’t care where they hire from, and that’s a different skill set.
“I look at GitLab as kind of a model too; I think they do it right.” added Mantle. They are one hundred percent remote and remain a high-performing, transparent company. Everything they do (they have a “playbook”) is all accessible online.
Measuring productivity
When Kumar and Mantle finally addressed the question of productivity, they cautioned the audience to first tread carefully.
“If you start telling people you’re going to measure productivity and don’t do it carefully, you will lose a lot of people very quickly, especially engineers.”
Kumar adds, “I talk about productivity as three things. First, it’s an outcome that you want, but that outcome can be different. For instance, there’s a difference between developer productivity and development productivity. If you’re looking at development productivity, you might be looking at your whole product team and thinking about, okay, well, how effectively does that team as a whole deliver value? But if you’re a VP of engineering, looking at your engineering team, you may have a different set of things that you want to look at as your productivity outcome, such as how fast you deliver or what the quality is, etc. So you have to start with the desired outcome and be clear about what you’re measuring productivity for. And it may have different systems for measuring productivity at these different levels of granularity. And you want to have a framework of measurements and measures telling you what things you care about. You want to make that framework very transparent to everybody on the team so that they understand what is being measured and why. And then, finally, you need a measurement system with very accurate and actionable data that is trusted by the people working in the process.”
Key rules of productivity
Kumar and Mantle left the panel with a final, overarching rubric summarizing the keys to productivity.
- It’s not a number, it’s a framework
- Make your desired outcomes explicit
- Use a few, stable KPIs, and many supporting indicators
- Have a mix of leading and trailing indicators
- Include perceptual measures
- Make the framework transparent to your organization
Having laid out such helpful and critical information, Kumar and Mantle concluded by advising the attendees that the very best indicators will only take you so far. “It’s the results that count, it’s not the metrics,” offered Mantle.
Nevertheless, this TechKnowCon panel offered another helpful framework for considering the most critical organizational and operational issues in the industry. We learned that it’s critical to define outcomes before creating metrics, that good data can help bridge the transition to remote work, that foundational benefits are distinct from motivational factors, and that productivity metrics should be offered and explained to employees as transparently as possible.
