Accountabilities:

The role focuses on ensuring the quality, reliability, and safety of advanced AI models through comprehensive evaluation and testing processes. You will collaborate on identifying weaknesses, improving evaluation methodologies, and creating actionable insights that enhance model performance.

• Evaluate large language models across areas such as factual accuracy, reasoning quality, hallucination detection, safety, and reliability.
• Design and execute test plans, regression suites, and evaluation frameworks to measure model performance.
• Identify and document model failures, including prompt vulnerabilities, incorrect outputs, bias issues, and reasoning inconsistencies.
• Develop clear evaluation rubrics, pass/fail criteria, and quality benchmarks for AI system assessment.
• Perform adversarial testing, red-teaming exercises, and robustness evaluations to identify potential risks.
• Verify grounding, retrieval-augmented generation outputs, tool usage accuracy, and workflow reliability.
• Analyze model behavior and provide recommendations for improving prompts, guardrails, and evaluation strategies.
• Support automation initiatives using tools such as Python or SQL to improve testing efficiency.
• Maintain detailed documentation of findings, root-cause analysis, and reproducible issue reports.
• Collaborate with technical teams to improve AI evaluation metrics, dashboards, and quality monitoring processes.

Requirements:

The ideal candidate brings strong experience in AI quality assurance, machine learning evaluation, software testing, or related technical disciplines. You should be comfortable analyzing complex AI behaviors and communicating technical findings clearly.

• Bachelor’s, Master’s, or PhD in Computer Science, Data Science, Computational Linguistics, Statistics, or a related technical field preferred.
• Experience testing, evaluating, or validating AI/ML systems, particularly large language models.
• Strong understanding of LLM evaluation concepts, including hallucination detection, factual consistency, safety testing, and reliability analysis.
• Experience designing QA frameworks, evaluation criteria, regression testing processes, or quality benchmarks.
• Familiarity with prompt engineering, system prompts, retrieval-augmented generation (RAG), and AI safety practices.
• Experience with adversarial testing, red teaming, bias assessment, or compliance-oriented AI reviews.
• Knowledge of test automation tools and programming languages such as Python or SQL.
• Familiarity with LLM evaluation tools, experiment tracking platforms, or AI testing frameworks.
• Strong analytical skills with the ability to identify patterns, investigate failures, and propose improvements.
• Excellent written and verbal communication skills, with the ability to clearly explain technical reasoning and findings.

Benefits:

• Competitive contractor compensation ranging from $6 to $65 per hour, depending on experience, expertise, and geographic location.
• Fully remote work environment with flexibility to contribute from anywhere.
• Opportunity to work on advanced AI systems and influence the development of future AI capabilities.
• Hands-on exposure to cutting-edge language models, evaluation methodologies, and AI safety practices.
• Ability to apply your technical expertise to impactful real-world AI improvement projects.
• Flexible contract-based opportunity designed for experienced AI and QA professionals.
• Contractors provide their own secure computer and high-speed internet connection.
• Company-sponsored benefits such as health insurance and paid time off are not included for this contract role.