Overview
Jasmine Robinson, from the Netflix Developer Education (DevEd) team, shares how they empower their subject-matter-experts as part-time instructors to deliver exceptional workshops through the use of checklists.
This article reviews the challenges that the team set out to solve and how they managed to bottle their best-practices into easy-to-apply checklists for their army of internal presenters.
About Jasmine
Jasmine grew up in a tech-savvy family where she and her siblings built their own computers and argued over dinner which programming language was best. Her early career started in academia, where her roles ranged from instructional design and learning management systems administration to becoming a Director of Web Services. In 2019, she decided to enter the private sector and join the Developer Education team at Netflix, where she is thrilled to be surrounded by some of the most talented people in the industry. You can take a look at Jasmine’s blogs here.
The challenge
Before diving into how checklists can help learning facilitators be more successful, it’s worth understanding the context.
Netflix recently announced that they had over 200M subscribers across 190 countries. Netflix’s growth has resulted in increased hiring of engineers to build, support, and cater to the member experience.
Jasmine’s Developer Education (DevEd) team ensures those engineers have the knowledge they need to be successful. With only four members, the DevEd team is “small but scrappy” and as Jasmine says, “has a perfect combination of complementary skills.” The team is responsible for onboarding, continuous learning, and knowledge management, which would be a tall order even for a team many times their size.
To deal with this scale, the DevEd team recruits and supports Netflix engineers, as subject-matter-experts (SMEs), to facilitate most of their internal training workshops. Jasmine acknowledged that there are some advantages and challenges with this approach of democratized knowledge sharing:
Pros
-
- Aligns with their culture of openness and selflessness.
-
- Helps engineers build more in-depth product expertise.
-
- Allows them to share knowledge of specialized homegrown systems.
-
- Helps engineers build empathy with their users.
-
- Fosters cross-team relationships, which builds a community.
-
- Helps presenters enhance their overall communication skills.
-
- Increases the customer-first mentality.
-
- Enables a team of four to serve an audience of many thousands!
Cons
-
- It can be expensive as engineers are pulled from projects while doing training.
-
- SMEs are not always professional trainers.
This approach of having internal subject matter experts offer peer to peer training sessions and workshops has been a great success. Netflix uses PlusPlus as their internal training event registration system. The PlusPlus system is set up so that any employee can add an event. Removing event creation barriers has led to increased adoption of the event registration system providing a centralized location for internal company events and a 2x growth in the number of events created.
Jasmine pointed out that creating and providing training is expensive, and that they want to ensure they are maximizing the time of their training facilitators and learners. They needed an efficient way to ensure quality training for their engineers.
We are thrilled so many of our colleagues want to offer training, but we want to help ensure our volunteer trainers and learners are maximizing their time (because it can be expensive when it’s not done effectively and efficiently).
– Jasmine Robinson
The solution: checklist!
Jasmine acknowledged that there is no single solution to this challenge, but checklists have become an essential part of her team’s toolbox for several reasons:
-
- Saves time as they are efficient and can be automated.
-
- They are self-service, asynchronous, and self-paced, which is what DevEd’s customers had been requesting.
-
- Similarity to the familiar agile development methodologies such as Scrum/Kanban where you mark things as done as you go along.
-
- Easy to use as a reference by the engineers who do not teach every day and could be prone to forgetting the best practices.
On this last point, Jasmine reminded us of Atul Gawande’s The Checklist Manifesto, which describes checklists as a cognitive safety net so that you don’t have to remember everything.
Netflix follows the A.D.D.I.E. instructional system design model to systematize and optimize their approach to knowledge sharing. But, instead of teaching ADDIE to every part-time presenter, the DevEd team encoded the entire model into a series of simple yes/no checklists. As a result, the presenters get the best-practices right out of the box.
The first is their Analysis checklist, which includes 11 questions prompting for things like:
-
- Is building a workshop the most effective means of transferring this knowledge, and is it worth the effort? Would it be more efficient if this was an email memo?
-
- Have the learning objectives been clearly articulated?
-
- Has the success criteria been defined?
-
- Does your audience have biases?
Jasmine shared an example of a Netflix team creating the “Oops Training”, designed to help engineers become better at dealing with incidents, including producing write-ups (a.k.a. postmortems), which capture the lessons learned. The checklist prompted the facilitators to think about the impact of this training, so they set their goal to increase the overall quantity of the Oops write-ups as well as lower the recurrence of previously identified mistakes.
Next up is their Design checklist, with eight questions to help the content authors create engaging and retentive content. It prompts for things like:
-
- Has the content been organized in bite-sized chunks that flow well?
-
- Can it be made more concise?
-
- Does it leverage visuals effectively?
-
- Does it address top technology adoption resistance concerns?
-
- And so on.
Jasmine found that some of these questions needed a bit of additional motivation for them to sink in. For example, in order to get the passionate engineers to make their content more concise, Jasmine would introduce them to the Cognitive Load Theory or share a metaphor: “Think of your brain like your stomach. You get full and need time to digest.” To get them to use more visuals, she would show-case the Picture superiority effect. That worked, and these subject matter experts began to buy into these best practices.
Their Development checklists includes seven considerations, such as:
-
- Is the content designed to grab the attention of the participants?
-
- Does the workshop clearly state its objectives and provide credibility to the presenter(s)?
-
- Is the content based on the real-world examples?
-
- And so on.
Jasmine’s team found that the course content that is based on real-world examples, rooted in the company history/context, provides a real advantage over canned courses that can be found online. They also found that motivating each section with “Who’s done this before?”, “Who’s seen this problem before?”, or “Who hates this as much as I do?” helps draw people into the conversation. In order to make the whole learning experience feel more relevant and thus retentive, the team started recommending that the content be based on relatable but sanitized, data-sets.
Make it applicable. Retention of learning occurs best when participants can associate the new information with previous information.
– Jasmine Robinson
Their Implementation checklist has 18 questions, aimed to feel more like tips, and while that seems like a lot, Jasmine told us that they usually go very quickly through this one. This covers topics like:
-
- Did the presenter have a good location/setup (e.g., well-lit, clear audio, no distractions)?
-
- Did the presenter have an engaging start to get everyone’s attention?
-
- Did the presenter share the learning etiquette?
Some of these “tips” may seem obvious, but for first-time presenters, these tips help increase their workshops’ effectiveness.
Their final Evaluation checklist includes just four questions, primarily based on the Kirkpatrick model:
-
- Have the participants found the training favorable, engaging, and relevant?
-
- Have the participants acquired the intended knowledge, skills, attitude, confidence, and commitment?
-
- Have the participants applied what they learned?
-
- Was the intended goal of the workshop achieved?
Jasmine acknowledges that these questions are typically hard to objectively answer or automate.
The results
In practice, these checklists are tools in the presenters’ toolbox. The DevEd team purposely made them self-serve and non-mandated. Presenters have the freedom to use whatever works for them. When they use the checklists and get stuck, or have several “no” answers, they are encouraged to reach out to the DevEd team for support.
We empower people (at Netflix) to make their own choices, and rather than policies, we provide best practices and frameworks.
– Jasmine Robinson
The team’s strategy for checklist adoptions was to:
-
- Make them discoverable
-
- Dog-food them for their own presentations so that they are continually being improved.
-
- Encourage the presenters to co-develop and contribute to them, thereby increasing the sense of ownership.
Jasmine mentioned that in academia there was an extensive process for everything, but her DevEd team found that reducing the bureaucracy and workflows aligns better with Netflix culture.
The team has been happy with the results, which include:
-
- Standardizes how to evaluate workshops and support their audience.
-
- Provides better quality content and instruction, resulting in better experience for the learners.
-
- Positive feedback from the presenters, who have felt more confident, less taxed, and with a better sense of flow when presenting.
-
- Continued support and praise from happy leadership!
For Jasmine, well, she has been glad that they have not needed to teach a single ADDIE session since creating these checklists. She said that this new tool is akin to having “a virtual DevEd coordinator doing a consult” with every presenter. And that’s exactly what they hoped to accomplish when they set out to tackle the challenge of knowledge sharing at the Netflix scale.
The future
Going forward, Jasmine and the team plan to continue to iterate on the DevEd’s checklists, as she calls it “an open source initiative” and even encouraged the broader TechKnowCon community to help them with this effort.
In addition to the checklists, Jasmine is hoping to collaborate on the best practices for developer onboarding, knowledge management, and internal communication, as well as common learning assets, such as templates for self-paced learning.
She is also excited to partner with L&D teams within Netflix, as she said that these best-practices are applicable to most learning professionals. She praised the Netflix culture, which encourages folks to stretch themselves, explore, fail, and learn. We are excited to be able to learn alongside them.
