Rajon Tumbokon recently joined Splunk as their Director of Engineering Enablement and Engagement. His team is responsible for the technical onboarding of engineers, technical learning, internal documentation, and knowledge sharing. Before that, he worked at LinkedIn for nine years, where he also helped create their strong learning culture.
Through experience, Rajon knows that training can be one of the most expensive business solutions. It is important to keep in mind that the goal of training is to solve business problems.
“One key thing to remember is that the primary reason your company invests in you and your team is to solve a business problem. Yes, we need to design courses or learning experiences and write documentation, but all of those are just a means to an end of adding actual business value,” said Rajon. “As your programs grow and evolve, there comes a time when your leadership questions what kind of value you’re adding to the business, and they want to know if they’re making the right investment.”
In his talk, Rajon went through some practical ways that you can use metrics to demonstrate the value and impact of training.
“I cannot tell you silver bullet metrics that’ll make you successful in your company, but what I can share is practical methods that will help you identify how you can measure so that it aligns with your company’s expectations,” Rajon said.
Four types of learning measurements
There are four general ways you can measure the effectiveness of learning programs for software engineers:
- Usage and adoption: Is your solution being used? Examples include training attendance, video views, and document views.
- Acceptance: Do people like your program? Examples include net promoter score (NPS) and customer satisfaction (CSAT).
- Application: Are the skills and knowledge being applied on the job? Examples include NPS after 30 days of completing training and the adoption rate of a new tool.
- Value and Impact: How does your program impact the business? Examples include the overall CSAT of training programs and the cost of internal support overhead.
The first two types are tactical metrics. The data for these two types of metrics are easy and inexpensive to collect and can be acted on immediately. The last two types are strategic methods. The data is harder to collect, and it takes a longer time to see the results and act on them.
“If you decide to gather strategic metrics through surveys, there will always be a lagging indicator. That means you most likely will find results two or more quarters after you start collecting your data, and it’s because a change of perception and change of behavior is really hard. And for a large organization, it happens over a longer period of time,” said Rajon.
Knowing what and how to measure
While Rajon’s career has been focused on tech enablement, efficiency, and closing knowledge gaps, the same techniques to gather metrics can be applied to onboarding, collaborative learning, and peer learning initiatives.
Analyze the Audience
The first step is to analyze who your target audience is. You need to know how many people are in your organization, what your hiring and attrition rates are, how they are distributed around the globe, whether they are remote or not, and other factors so you can segment the audience into cohorts. This is crucial to understanding the data once you collect it.
Interview Stakeholders
Once you have a better understanding of the audience, the next step is to interview the stakeholders to discover what business problems need to be solved. Do they want to reduce support? Are there issues of productivity, efficiency, or accuracy? What kind of knowledge gaps do they see?
Interview the Audience
Even more important is interviewing different segments of your audience. Start by identifying employees from different levels, cohorts, locations, and teams to get a good cross-section and interview them. Find out what their biggest pain points are.
Create a Survey
You should start to notice patterns as you interview employees, which you will naturally form assumptions about. Now it’s time to validate those assumptions by sending out a survey to a larger group. Once you have the results, you can start slicing your data based on the segments you created in the audience.
Define Metrics
Now that you have contextual information from interviews and surveys, you can start defining metrics. Rajon suggests using the goals/signals/metrics framework, which is popular in product development and user experience:
- Goals: What is the business problem being solved?
- Signals: How do we know when the goal is reached?
- Metrics: How can we quantify signals in two or more measures?
Collect and Aggregate
The data you collect for your metrics will be incredibly noisy. To turn this data into real insights, you will need to aggregate this data using the segments you defined earlier as dimensions or filters to reduce the noise. This will let you know where to zoom in and what segment of the audience to focus on. To slice some of the data, you will need to turn to HR data for segments like tenure and location.
Next steps
Now that you have a baseline, the next step is to create data-driven hypotheses to solve the problem. Once you have a potential solution in place, you continue measuring the results with the same methods you used to find the pain points. It is also important to make your team accountable for its metrics. If you have a weekly team meeting, review your metrics as part of the agenda, and review them again on a monthly or quarterly basis with leadership.
