Automation QA Engineer
# Automation QA Engineer **Comrise** · Taguig · `On-site` 🕒 **Статус:** *Опубликовано: 12 дней назад* · *Источник: Indeed* --- ### About the Role The Automation QA Engineer will define, implement, and continuously improve quality assurance practices for AI-enabled and agent-based solutions. The role will ensure that AI workflows meet agreed standards for accuracy, reliability, performance, safety, compliance, and operational readiness through automated testing, evaluation, and continuous monitoring. Key Responsibilities - Define and execute QA and testing strategies for AI-enabled applications, agent workflows, and automated processes. - Establish quality criteria covering accuracy, latency, reliability, safety, compliance, and operational risk . - Build and maintain automated evaluation frameworks to assess agent performance, including: - Hallucination and response accuracy - Tool misuse and workflow errors - Policy or safety violations - Task completion and success rates - Regression and quality degradation - Develop and maintain automated test suites using Playwright for UI and end-to-end testing. - Create Python-based scripts for API, workflow, and agent validation. - Apply LLM evaluation frameworks such as DeepEval, RAGAS, and Evidently AI to measure output quality, regression, and system drift. - Implement continuous production monitoring to identify anomalies, quality degradation, and emerging safety or reliability concerns. - Develop dashboards and reports to communicate QA metrics, trends, defects, and quality insights to engineering teams and stakeholders. - Develop and maintain runbooks for common failure scenarios and support incident response activities. - Collaborate with developers, product teams, and other technical stakeholders to improve prompts, tool definitions, agent workflows, and overall system quality. - Ensure testing, logging, monitoring, and documentation practices support data privacy, auditability, risk management, and regulatory requirements . - Contribute to continuous improvement of QA processes, automation frameworks, and AI quality standards. Must-Have Qualifications - 3+ years of experience in QA, Test Automation, Software Testing, or DevOps. - 2+ years may be considered for candidates with direct hands-on experience testing AI/ML-enabled applications or agent-based systems. - Strong Python skills for test automation, evaluation frameworks, API testing, and basic data analysis. - Hands-on experience with Playwright or a comparable UI automation framework. - Experience with API testing and workflow validation using automated scripts. - Hands-on exposure to AI/LLM evaluation , including tools such as: - DeepEval - RAGAS - Evidently AI - Understanding of automated testing, regression testing, quality monitoring, and defect analysis. - Strong analytical and problem-solving skills with high attention to detail. - Ability to work effectively in a fast-changing technical environment and collaborate with engineering and product teams. Technical Skills Required: - Python – Test Automation & Data Analysis - Playwright – UI/E2E Automation - API & Agent Workflow Testing - LLM/AI Evaluation - Production Quality Monitoring - Regression & Performance Testing - Anomaly Detection Nice to Have: - Pytest or equivalent testing frameworks - SQL for querying logs, metrics, and evaluation datasets - Prometheus, Grafana , or similar monitoring platforms - Knowledge of hallucination detection and AI safety patterns - Experience with CI/CD pipelines and Git-based workflows - Exposure to cloud environments and observability tools Application Question(s): - When is your earliest availability to start? - What is your expected salary in Philippine Peso (PHP)? - Are you open to a 6-month project-based role, with a possibility of extension to 9–12 months, depending on your performance and business requirements? - Are you willing to work full onsite in Taguig? Education: - Bachelor's (Preferred) Experience: - QA / Test Automation: 3 years (Preferred) - Python automation: 2 years (Preferred) - Playwright / UI automation: 2 years (Preferred) - AI/LLM testing: 1 year (Preferred) - DeepEval, RAGAS, Evidently AI, or similar LLM: 1 year (Preferred) - CI/CD & Git-based workflows: 2 years (Preferred) Work Location: In person
Наблюдалась 2026-10-06, впервые 2026-09-24, источник — Indeed.