Job Title: Senior AI QA Automation Engineer Key Skills: Python, AI Testing, Test Automation, Playwright, Selenium, API Testing, Azure AI Foundry Evaluations, RAG Evaluation, Prompt Testing, CI/CD, Monitoring Experience: 5+ YOE. Location: Costa Rica, Peru, Colombia, Bolivia. Mode: Remote We at Coforge are hiring Senior AI QA Automation Engineer (#006fu00000bWsOCAA0-4) with the following skill set.
Key Responsibilities
- Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
- Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
- Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.
- Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability,



and recurring failure patterns.

Required Skills & Qualifications
- 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
- Strong Python skills with experience building reusable automated testing frameworks.
- Hands-on experience with Playwright and/or Selenium.
- Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
- Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
- Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
- Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
- Experience implementing CI/CD quality gates and production monitoring.
- Ability to translate business requirements, workflows, and AI behaviors into m

📌 Senior AI QA Automation Engineer (Perú)
🏢 encora10
📍 Perú