How you will make an impact

• Design and build evaluation harnesses for agentic systems in Python — golden datasets, LLM-as-judge graders, multi-turn regression suites and trace-based assertions. In addition, develop framework to verify generated AI output.

• Author automated test suites for prompts, tools, structured outputs (Pydantic / JSON schema), retrieval pipelines (ETL Experience) and end-to-end agent workflows

• Validate guardrails around tool execution: auth scoping, input/output validation, PII and prompt-injection protections, and hallucination mitigation

• Wire evaluations into CI using Dataiku Evaluations, GitHub Actions or Jenkins so every change is graded against quality, safety and cost SLOs before it ships

• Build observability into testing by instrumenting traces with LangSmith, Langfuse, MLflow or OpenTelemetry and triaging production drift back into the eval harness

• Own quality end-to-end — define release criteria, run pre-prod and shadow tests, and partner with engineering to root-cause and fix regressions quickly

• Partner with data engineers on Snowflake-backed retrieval testing patterns (Cortex Analyst and Cortex Search Services) and with platform teams on observability, security and cost

• Help shape internal QA standards for AI & Data engineering as the stack evolves, contributing to design reviews and sharing knowledge across the India and U.S. teams

• Participate in a collaborative DevOps environment, working closely with developers, AI engineers, Data Engineers, DBAs and product partners across environments

In your first 90 days

By the end of your first 90 days, you will have stood up at least one production-grade evaluation harness — golden dataset, LLM-as-judge graders and regression suite — wired into CI for an internal agent. You will have automated trace-based assertions running against staging traffic, a clear quality scorecard for at least one shipped agent, and a clear opinion about what our next testing investment should be.

What you need to be successful

• 3+ years of professional QA / SDET experience, with production experience automating tests for backend services or data pipelines

• 1+ years of hands-on experience testing LLM or AI features in production: prompt regression, tool / function-call validation, structured outputs and RAG correctness

• Working knowledge of evaluation frameworks such as RAGAS, DeepEval, LangSmith, Langfuse or comparable LLM-as-judge tooling

• Strong Python and PyTest skills; solid SQL skills and comfort with at least one cloud platform (AWS, Azure or GCP)

• Fluency with Git, Docker, REST APIs and at least one CI tool (GitHub Actions, Jenkins, GitLab CI or CircleCI)

• Solid understanding of data security and responsible AI practices, particularly in PCI-compliant or regulated environments

• Proven ability to work independently and within a team, managing priorities across concurrent projects and time zones

• Strong written and verbal communication skills; able to work effectively with both technical and non-technical stakeholders

• A bachelor’s degree is not required — equivalent practical experience (including bootcamps, self-taught work, career changes or non-CS technical degrees) counts

• Bonus Skills:

• Hands-on experience with Dataiku DSS (Python / SQL recipes, scenarios, code environments, the dataiku and dataikuapi clients) or Dataiku Evaluations

• Experience with Dataiku LLM Mesh, Knowledge Banks, Prompt Studio, or Visual / Code Agents

• Experience with Snowflake, Snowpark, or Snowflake Cortex (Search, Analyst, Agents)

• Experience with red-teaming, prompt-injection testing or adversarial test generation for LLMs

• Familiarity with multi-agent patterns: supervisor / router, subagent / handoff, reflection, human-in-the-loop

• Experience with performance and load testing tools such as Locust, JMeter or k6

• ISTQB, AI Testing or comparable QA certification

• Experience in loyalty, martech, adtech or a comparable data-rich B2B domain