Senior LLM Quality Analyst

WHO WE ARE: MagicSchool is the premier generative AI platform for teachers. We're just over 2 years old, and more than 7 million teachers from all over the world have joined our platform. Join a top team at a fast growing company that is working towards real social impact. Make an account and try us out at our website and connect with our passionate community on our Wall of Love.

Senior LLM Quality Analyst

Role Description

As a Senior LLM Quality Analyst, you'll work with the Trust, Safety and Quality team to ensure MagicSchool's AI outputs meet the highest standards for educators and students. This role combines data analysis, prompt engineering, and educational experience. You will design and execute experiments to test prompt strategies, evaluate model performance, and maintain dashboards. You will bring a critical eye to AI quality, strong analytical skills, and a deep understanding of educational needs.

What You'll Do

• Design and Execute Prompt Engineering Experiments

  • Work with the product manager, data scientist and team lead to design prompt strategies and experiments to test them

  • Use our Evaluation Platform to improve and iterate on MagicSchool prompts from an educators perspective

  • Design experiments using our Evaluation Platform to compare models for different use cases

  • Analyze results and make data-driven recommendations for prompt improvements

• Maintain Response Quality Reporting and Dashboards

  • Work with the team lead to design reports and dashboards for performance-over-time reporting

  • Maintain evaluation dashboards and reporting output using Metabase

  • Ensure stakeholders have visibility into response quality metrics

• Evaluate and Improve AI Outputs

  • Work with the product manager and data scientist to design evaluators and test cases

  • Assess AI outputs from an educator's perspective to ensure they meet classroom needs

  • Identify quality issues and work cross-functionally with product managers, engineers, and the data scientist to address them

• Stay Current on LLM Best Practices

  • Keep up with the latest research and techniques in prompt engineering and LLM evaluation

  • Bring passion for solving education problems with technology

Qualifications, Competencies & Skills

• Strong analytical skills and comfort working in spreadsheets

• Experience with BI tools (we use Metabase, but experience with any similar tool is valuable)

• Basic statistics knowledge

• SQL

• Organized and detail-oriented: Maintains clear documentation and systematic approaches to testing

• Critical thinker: Can design meaningful experiments and draw insights from data

• Gets a lot done: Works hard, resourceful, does what it takes

• Adaptable: Smart, learns quickly, curious

• Builds relationships easily: Emotionally intelligent, warm communicator

• Strong communication skills: Team-first mindset, highly collaborative, can articulate decisions within team's context

• Nice to have: Analytical Python experience (huge plus - pandas, jupyter notebooks, data visualization)

Experience

• 5+ years of professional experience in data analysis

• Team Lead Experience

• Experience working with LLMs or AI products (prompt engineering, evaluation, quality analysis, or AI product work)

• Strongly preferred: K-12 teaching experience or EdTech industry experience

• Preferred: Startup experience

• Passionate about solving education problems with technology

Application Notice:

Notice: Priority Deadline and Review Start Date

Please note that applications for this position will be accepted until 12/14/25 - applications received after this date will be reviewed on an intermittent basis. While we encourage early submissions, all applications received by the priority deadline will receive equal consideration. Thank you for your interest, and we look forward to reviewing your application.

Why Join Us?

• Work on cutting-edge AI technology that directly impacts educators and students.

• Join a mission-driven team passionate about making education more efficient and equitable.

• Flexibility of working from home, while fostering a unique culture built on relationships, trust, communication, and collaboration with our team - no matter where they live.

• For full time employees:

• Unlimited time off to empower our employees to manage their work-life balance. We work hard for our teachers and users, and encourage our employees to rest and take the time they need.

• Choice of employer-paid health insurance plans so that you can take care of yourself and your family. Dental and vision are also offered at very low premiums.

• Every employee is offered generous stock options, vested over 4 years.

• 401k match & monthly wellness stipend.

Our Values:

• Educators are Magic: Educators are the most important ingredient in the educational process - they are the magic, not the AI. Trust them, empower them, and put them at the center of leading change in service of students and families.

• Joy and Magic: Bring joy and magic into every learning experience - push the boundaries of what’s possible with AI.

• Community: Foster community that supports one another during a time of rapid technological change. Listen to them and serve their needs.

• Innovation: The education system is outdated and in need of innovation and change - AI is an opportunity to bring equity, access, and serve the individual needs of students better than we ever have before.

• Responsibility: Put responsibility and safety at the forefront of the technological change that AI is bringing to education.

• Diversity: Diversity of thought, perspectives, and backgrounds helps us serve the wide audience of educators and students around the world.

• Excellence: Educators and students deserve the best - and we strive for the highest quality in everything we do.