Intermediate/Senior Test Engineer Automation - OP01926 at Devpro

🟢 Are you in Brazil or Argentina? Join us as we actively recruit in these locations, offering a comfortable remote environment. Submit your CV in English, and we'll get back to you!

We invite an experienced and proactive Test Automation Engineer with strong Python and manual testing skills to join our team. In this role, you’ll design and implement robust testing solutions for cutting-edge AI products, focusing on improving RAG flows and testing LLM integrations to ensure reliability, safety, and performance.

🟩 What's in it for you:

Work hands-on on test automation for AI-powered solutions that have real-world impact
Grow your expertise in AI safety and RAG/LLM testing
Collaborate with a team of experienced engineers and QA professionals in a modern, innovation-focused environment

✅ Is that you?

3+ years of experience in automation testing
Strong programming skills in Python
Practical experience with pytest or similar Python testing frameworks
Experience developing and maintaining test automation frameworks
Solid understanding of test design, strategy, and QA best practices
Hands-on experience with CI/CD pipelines and tools (Jenkins, GitLab CI)
Understanding of RAG pipelines and testing AI/ML-based systems
Familiarity with AI safety testing (prompt validation, hallucination checks, response consistency)
Familiarity with API (REST, HTTP) testing and data validation
Good knowledge of Git, test reporting, and bug tracking tools
Strong analytical and problem-solving skills
Upper-Intermediate English level

Desirable:

Experience with major cloud platforms (GCP, AWS, Azure)
Knowledge of containerization tools (Docker, Kubernetes)

🧩Key responsibilities and your contribution

In this role, you'll develop and execute automated tests to ensure the quality and reliability of AI-driven products.

Design, develop, and maintain automation frameworks and test strategies for AI products
Evaluate and improve RAG flows, ensuring accuracy and performance
Test LLM integrations with prompt validation, hallucination checks, and response consistency checks
Prepare test data and clarify testing requirements as needed
Create and run automated test suites for APIs, data pipelines, and model integrations
Integrate automated tests into CI/CD pipelines to maintain quality and stability
Analyze test results, report defects, and work with teams to resolve issues
Participate in code reviews, sprint ceremonies, and QA planning sessions
Maintain clear test documentation and provide regular updates to stakeholders
Continuously improve testing tools, processes, and automation efficiency

🎾 What's working at Dev.Pro like?

Dev.Pro is a global company that's been building great software since 2011. Our team values fairness, high standards, openness, and inclusivity for everyone — no matter your background

🌐 We are 99.9% remote — you can work from anywhere in the world
🌴 Get 30 paid days off per year to use however you like — vacations, holidays, or personal time
✔️ 5 paid sick days, up to 60 days of medical leave, and up to 6 paid days off per year for major family events like weddings, funerals, or the birth of a child
⚡️ Partially covered health insurance after the probation, plus a wellness bonus for gym memberships, sports nutrition, and similar needs after 6 months
💵 We pay in U.S. dollars and cover all approved overtime
📓 Join English lessons and Dev.Pro University programs, and take part in fun online activities and team-building events

Our next steps:

✅ Submit a CV in English — ✅ Intro call with a Recruiter — ✅ Internal interview — ✅ Client interview — ✅ Offer

Interested? Find out more: