As a Data Architect - Annotation, you’ll serve as the critical bridge between the Prompt Engineering team and the Data Labeling team, ensuring that the data feeding our AI systems is clean, consistent, and production-ready. You will own the workflows that generate, organize, and maintain high-quality datasets across multiple modalities, while using LLMs, automation, and statistical analysis to detect anomalies and improve data quality at scale.
Your work will directly influence the reliability of our VoiceAI and AI-driven products by ensuring that labeling pipelines, annotation standards, and evaluation data are robust enough to support high-stakes, real-world restaurant operations.
Essential Job Functions:
- Data Operations & Workflow Ownership
- Act as the transition point between Prompt Engineering and Data Labeling, translating model and product requirements into concrete data and annotation workflows.
- Design, implement, and maintain scalable data workflows for dataset generation, curation, and ongoing maintenance.
- Ensure data quality and consistency across labeling projects, with a focus on operational reliability for production AI systems.
- Annotation & Quality Management
- Create, review, and maintain high-quality annotations across multiple modalities, including text, audio, conversational transcripts, and structured datasets.
- Identify labeling inconsistencies, data errors, and edge cases; propose and enforce corrective actions and improvements to annotation standards.
- Utilize platforms such as Labelbox, Label Studio, or Langfuse to manage large-scale labeling workflows and enforce consistent task execution.
- Automation, Tooling & LLM-Assisted QA
- Use Python and SQL for data extraction, validation, transformation, and workflow automation across labeling pipelines.
- Leverage LLMs (e.g., GPT-4, Claude, Gemini) for prompt-based quality checks, automated review, and data validation of annotation outputs.
- Implement automated QA checks and anomaly-detection mechanisms to scale quality assurance for large datasets.
- Analysis, Metrics & Continuous Improvement
- Analyze annotation performance metrics and quality trends to surface actionable insights that improve labeling workflows and overall data accuracy.
- Apply statistical analysis to detect data anomalies, annotation bias, and quality issues, and partner with stakeholders to mitigate them.
- Collaborate with ML and Operations teams to refine labeling guidelines and enhance instructions based on observed patterns and error modes.
- Cross-Functional Collaboration & Documentation
- Work closely with Prompt Engineering, Data Labeling, and ML teams to ensure that data operations align with model requirements and product goals.
- Document data standards, annotation guidelines, and workflow best practices for use by internal teams and external labeling partners.
