The Community You Will Join:

Artificial Intelligence is at the heart of Airbnb’s product strategy. The Conversational Product team within Assistant Engineering is responsible for building and scaling Airbnb’s AI Assistant experience, expanding it beyond customer support to empower product domains across Airbnb.

Within the Conversational Product team, the AI Assistant Product Evaluation team builds the evaluation systems, data foundations, and observability tools that help Airbnb develop reliable, high-quality agentic AI products. Our work enables teams to measure AI product quality, understand model and agent behavior, identify regressions, and accelerate the feedback loop between model development and product iteration.

The Difference You Will Make:

As a Senior Software Engineer on the AI Assistant Product Evaluation team, you will design and build scalable data systems and evaluation platforms for Airbnb’s agentic AI products. You will work at the intersection of backend engineering, data systems, AI evaluation, and product quality.

In this role, you will help define how Airbnb measures, monitors, and improves the quality of agentic AI products at scale. You will build data-rich systems that transform product interactions, model outputs, human feedback, and evaluation results into reliable signals for product and model development. Your work will help teams move faster with greater confidence by making AI quality measurable, debuggable, and actionable across the development lifecycle.

You will also partner closely with modeling teams, product managers, data scientists, and operations teams to turn ambiguous AI evaluation needs into robust systems that help Airbnb ship trustworthy AI experiences.

A Typical Day:

• Design and productionize scalable data systems that support AI evaluation, metric computation, observability, and feedback loops for agentic AI products.
• Build data models, schemas, and processing pipelines for agentic AI interactions, supporting reliable logging, retrieval, metric computation, and long-term evaluation dataset management.
• Work closely with Core Modeling engineers to understand pain points in the LLM evaluation process, and develop LLM-as-a-judge solutions and data pipelines to address metric-related challenges in a scalable and efficient way.
• Collaborate with machine learning infrastructure engineering teams to evolve how we build and test evaluation framework for Airbnb Conversational AI products.
• Lead all phases of software development including architecture design, implementation and testing.
• Work collaboratively with cross-functional partners including product managers, operations and data scientists, identify opportunities for business impact, understand and prioritize requirements for machine learning systems and data pipelines, drive engineering decisions and quantify impact.
• Foster a culture of engineering excellence by supporting teammates in writing high-quality code, ensuring operational reliability, and sharing knowledge across the team.

Your Expertise:

• 5+ years of industry experience as a software engineer, backend engineer, platform engineer, or data-focused software engineer building production systems.
• BS, MS, or PhD in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
• Strong programming skills in Python, with experience building production-quality software, libraries, frameworks, or data processing systems.
• Experience designing and operating scalable data pipelines using Airflow or similar orchestration frameworks; experience with Spark, Flink, Kafka, Trino/Presto, Hive, Iceberg, or similar big data technologies is a strong plus.
• Strong understanding of data modeling, schema design, data quality, partitioning, indexing, storage formats, and tradeoffs for large-scale analytical and operational data systems.
• Experience building data layers, evaluation systems, feedback loops, experimentation platforms, observability tools, or AI/ML platform capabilities.
• Ability to analyze complex datasets, identify data quality issues, debug inconsistencies, and translate findings into actionable engineering or product decisions.
• Strong system design skills, including experience building reliable, extensible, maintainable systems with clear APIs, testing strategies, and operational ownership.
• Solid understanding of data structures and algorithms, with the ability to make practical engineering tradeoffs for performance, scalability, and maintainability.
• Familiarity with AI/ML system concepts such as model evaluation, offline evaluation, online monitoring, model quality metrics, human-in-the-loop workflows, experimentation, or model deployment.
• Proven ability to work cross-functionally with modeling engineers, product managers, data scientists, infrastructure teams, and operations partners to deliver end-to-end solutions.
• Excellent communication with the ability to drive alignment, set technical direction, and raise engineering quality across a team.

Your Location:

This position is US - Remote Eligible. The role may include occasional work at an Airbnb office or attendance at offsites, as agreed to with your manager. While the position is Remote Eligible, you must live in a state where Airbnb, Inc. has a registered entity. Click here for the up-to-date list of excluded states. This list is continuously evolving, so please check back with us if the state you live in is on the exclusion list . If your position is employed by another Airbnb entity, your recruiter will inform you what states you are eligible to work from.

Our Commitment To Inclusion & Belonging:

Airbnb is committed to working with the broadest talent pool possible. We believe diverse ideas foster innovation and engagement, and allow us to attract creatively-led people, and to develop the best products, services and solutions. All qualified individuals are encouraged to apply.

We strive to also provide a disability inclusive application and interview process. If you are a candidate with a disability and require reasonable accommodation in order to submit an application, please contact us at: [email protected]. Please include your full name, the role you’re applying for and the accommodation necessary to assist you with the recruiting process.

We ask that you only reach out to us if you are a candidate whose disability prevents you from being able to complete our online application.