> Markdown version of [/jobs/ext/3782434-remote-ai-evaluation-engineer](https://www.wearedevelopers.com/jobs/ext/3782434-remote-ai-evaluation-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Remote AI Evaluation Engineer

- **Company:** Scopic Software
- **Location:** Phoenix, OR, United States (Remote available)
- **Contract:** Permanent contract
- **Skills:** AI Evaluation, Application Programming Interfaces (APIs), Encodings, Databases, Continuous Integration, Github, JSON, Python (Programming Language), Load Testing, Microsoft UI Automation, TypeScript, Openapi, Data Logging, LangChain, Evaluation Pipelines, Large Language Models, Llamaindex, Agentic-AI, LangSmith, CrewAI, AutoGen, Arize Phoenix, Evaluation of Large Language Models, Api Management
- **Published:** October 16, 2026
- **Apply:** https://www.careerjet.com/job/usea5edaa46fb5928569036729039557fe/eaa

## About the Role

* Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems

 * Hands-on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs

 * Experience integrating tests into GitHub Actions or similar CI systems using SDK-level execution

 * Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function-calling payloads

 * Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging

 * Ability to independently design and own an evaluation architecture from scratch

 * Strong written and spoken English

 * At least 3 hours of daily overlap with US Central Time business hours

 * Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI

 * Deep understanding of embedding models and vector databases

 * Experience load testing non-deterministic endpoints

## Description

Are you a QA or eval engineer who would rather build the testing system than run someone else's? We're looking for a founding AI Evaluation Engineer to create the evaluation, testing, and observability infrastructure for production-grade AI agents. These agents handle complex reasoning, multi-turn conversations, and direct tool execution (APIs, code execution, database calls). You'll go beyond UI automation and manual prompt checks to build headless, statistical, and deterministic evaluation pipelines for non-deterministic LLM behavior.
Responsibilities

 * Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment

 * Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency

 * Implement fast, low-cost code-based checks before relying on LLM-as-a-judge. Examples include JSON schema validation, regex matching, API status checks, and embedding similarity.

 * Design test suites for multi-turn conversations, validating context retention, state drift, and safeguards against infinite tool-calling loops

 * Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to keep sandboxes and guardrails secure

 * Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools, and reduce LLM API costs through caching and mocking