From worlds best pros to AI personas: The MasterClass journey
MasterClass has redefined personalized learning by providing access to the world's best minds – from culinary master Gordon Ramsay to fashion icon Anna Wintour. When they decided to create AI personas that could authentically represent these instructors, they faced a challenge that would make or break the entire experience.
"About 2.5 years ago, with GenAI, we discovered the opportunities to use this technology and capture the unique experiences, the voice, and the learnings our instructors have to offer and bring them to our users' homes", explains Aman Gupta, Senior Staff Engineer who leads AI Engineering and Research at MasterClass.
Their vision: MasterClass On Call — not just another AI chatbot, but authentic digital extensions of world-class experts that preserve their unique teaching methods and perspectives.
The stakes couldn't have been higher. A single hallucinated anecdote or off-brand response could damage the reputations instructors had spent their entire careers building, while simultaneously undermining MasterClass's position as a platform for authentic expertise.
"The most critical aspect for us was maintaining an authentic instructor persona, because that's what's most important to deliver amazing user engagement," Aman notes.
For MasterClass’s collaboration with Collinear, they chose Chris Voss—a former FBI hostage negotiator and internationally acclaimed negotiation expert—as the pilot AI instructor for On Call. Voss presented the perfect test case—his distinctive teaching style combines tactical empathy, calibrated questions, and specific negotiation techniques like "mirroring" that would be challenging for generic AI to capture authentically. An AI persona representing him would need to embody these unique approaches while avoiding fabricating FBI cases or personal experiences that never happened. This collaborative work on Voss would become the foundation for MasterClass's repeatable recipe for future instructor models.
When traditional approaches fall short
MasterClass quickly discovered that conventional AI implementation methods couldn't deliver the authenticity they required.
Traditional supervised fine-tuning would require prohibitive volumes of instructor-specific data – data that simply didn't exist in sufficient quantity. Relying on retrieval-augmented generation (RAG) improved factual accuracy but often failed to capture the nuanced teaching style that made the experts special in the first place.
Safety was a major challenge - generic safety filters frequently rejected perfectly valid instructor-style responses. When representing Chris Voss, his AI instructor persona would need to discuss negotiation tactics with hostage takers – content that generic safety models might flag as inappropriate.
"We needed to ensure our AI solutions had the right guardrails to remain faithful to that brand," explains Aman.
This was the last mile gap – the difference between a functional AI and one that truly delivered on the MasterClass promise of learning from the world's best.
Custom Judges: Enhancing AI persona authenticity
Collinear's approach revolutionized how MasterClass could represent their instructors through AI. Instead of relying on one-size-fits-all safety mechanisms, they implemented a framework using three key innovations:
1. Custom Quality Judges: Specialized evaluation models were trained on just 25-30 carefully selected examples that evaluated responses based on persona-specific criteria – understanding, for example, that Chris Voss's negotiation tactics needed to be presented in his distinct style with the right terminology and approach.
2. Knowledge Infusion: To meet high reliability standards, we generated datasets for continual pre-training to infuse persona-specific knowledge into the language models. Using a high rank for LoRA enabled the models to memorize relevant information and outperformed RAG on hallucination benchmarks.
3. Alignment Fine-tuning: Leveraging custom judges, high-quality training pairs were created that captured what made each persona unique. The custom judges evaluated these pairs, allowing for datasets that reflected distinctive teaching approaches without expensive human annotation. We fine-tuned models to instructor specifications while maintaining performance comparable to frontier models, striking the perfect balance between safety and authenticity.
"What stood out was Collinear's ability to build custom reliability and safety judges which we were able to customize per each instructor's individual needs," says Aman. "We worked very closely with Collinear to iterate and improve the quality of these judges as well as the On Call models."
Results That Transform the Learning Experience
The results weren’t just solid — they elevated MasterClass’s entire approach to AI. Here’s what our approach actually delivered:
Authenticity that users noticed: "The user feedback from our open beta has been extremely positive, and many users directly acknowledged the authenticity of the instructor conversation experience," Aman shares.
Significant safety and reliability improvements: The custom judges for safety and reliability showed extremely high alignment with MasterClass's policies, allowing them to filter out inappropriate content while preserving the instructors' unique voices.
Measurable performance gains: "Combining that data and Collinear's AutoAlign training process, we were able to drive over 15% improvement in safety and reliability scores compared to only doing supervised fine-tuning," notes Aman.
A repeatable recipe for growth: "This collaboration helped us bridge the gap between an alpha version and a production-ready experience. It also gave us a repeatable recipe to build new instructor models that are well aligned and also potentially new future AI experiences," Aman explains.
Ready to build an AI improvement flywheel?
Take a look at one of our CollinearSafe assessment reports. Uncover AI model weaknesses through our proprietary CollinearSafe red-teaming. Comprehensive assessment delivered in only one business day.
Schedule a live demo or a consultation today to see how we help enterprises drive performance and quality improvement for their AI solutions.