> Markdown version of [/jobs/ext/3429618-ai-test-automation-engineer](https://www.wearedevelopers.com/jobs/ext/3429618-ai-test-automation-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AI Test Automation Engineer

- **Company:** Globant
- **Location:** Egypt, AR, United States
- **Contract:** Permanent contract
- **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Automation of Tests, Continuous Integration, DevOps, JSON, Python (Programming Language), Mockito, System Testing, Large Language Models, Pytest, Gitlab-ci, Enterprise Integration
- **Published:** September 14, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=4a7f7b312fe57a2d

## About the Role

* Automation Mastery: 4+ years of test automation experience using Python and pytest (custom fixtures, mocking, and CI pipeline integration) .
 * LLM & Agentic System Testing: 1+ years of hands-on experience evaluating non-deterministic AI applications .
 * LLM Evaluation Engineering: Proven track record building LLM-as-a-judge scoring scripts and maintaining versioned evaluation datasets (golden paths, edge cases, regression suites) .
 * Agent Deep-Dives & Traces: Ability to validate tool selection and parameters, inspect multi-step agent execution traces (e.g., LangGraph, Microsoft Agent Framework), and mock agent endpoints for CI workflows .
 * Adversarial & Guardrail Testing: Experience designing negative-path test suites for prompt injections, hallucinated financial actions, and trust-level boundaries (Suggest / Approve / Act) .
 * Observability & CI Integration: Hands-on integration of test stages into GitLab CI (or equivalent) using LLM observability tools like OPIK or similar trace analysis platforms ., * Test Automation (Essential): 4+ years of test automation with strong Python and pytest . Experience testing APIs, contract testing, and strict JSON schema validation .
 * AI & LLM Testing (Essential): 1+ years of experience with behavioral acceptance criteria, evaluation datasets, and LLM-as-a-judge techniques . Practical understanding of managing non-determinism via statistical thresholds and automated judges .
 * Frameworks (Essential): Hands-on familiarity with agentic frameworks (LangGraph, Microsoft Agent Framework, LangChain, or LlamaIndex) sufficient to execute and mock agent loops .
 * Observability & Domain (Desirable): Experience with OPIK or equivalent trace analysis platforms . Prior experience testing in banking, fintech, or regulated financial environments .
 * Knowledge of Arabic language is plus.

## Description

* Agent Test Design & Automation: Design and implement deterministic pytest suites to validate agent behavior, intent handling, tool-call correctness, strict JSON schemas, and failure paths .
 * LLM Evaluation Engineering: Build and maintain LLM-as-a-judge evaluation scripts that score agent outputs for correctness, tone, and compliance, defining scoring thresholds and pass/fail release gates .
 * Mocking & Trace Analysis: Build mocks and fixtures for LLM endpoints and core APIs so agent loops run repeatably in CI pipelines . Execute, mock, and capture traces from LangGraph or Microsoft Agent Framework agent loops within test suites .
 * Guardrail & Security Testing: Continuously validate that agents refuse prompt injection attempts, do not hallucinate actions, and strictly follow compliance guardrails .
 * CI/CD Integration & Reporting: Integrate test suites into GitLab CI with DevOps, leverage OPIK for trace analysis, and deliver automated quality evidence gating weekly drops .