Remote AI Evaluation Engineer

Are you a QA or eval engineer who would rather build the testing system than run someone else’s? We’re looking for a founding AI Evaluation Engineer to create the evaluation, testing, and observability infrastructure for production-grade AI agents. These agents handle complex reasoning, multi-turn conversations, and direct tool execution (APIs, code execution, database calls). You’ll go beyond UI automation and manual prompt checks to build headless, statistical, and deterministic evaluation pipelines for non-deterministic LLM behavior. Responsibilities

• Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment

• Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency

• Implement fast, low-cost code-based checks before relying on LLM-as-a-judge. Examples include JSON schema validation, regex matching, API status checks, and embedding similarity.

• Design test suites for multi-turn conversations, validating context retention, state drift, and safeguards against infinite tool-calling loops

• Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to keep sandboxes and guardrails secure

• Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools, and reduce LLM API costs through caching and mocking

• Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems

• Hands-on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs

• Experience integrating tests into GitHub Actions or similar CI systems using SDK-level execution

• Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function-calling payloads

• Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging

• Ability to independently design and own an evaluation architecture from scratch

• Strong written and spoken English

• At least 3 hours of daily overlap with US Central Time business hours

• Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI

• Deep understanding of embedding models and vector databases

• Experience load testing non-deterministic endpoints

• Your growth is our growth. We invest in your future with paid training and other professional opportunities.

• We’re industry innovators. You’ll work with modern technologies and a team that values strong engineering and continuous improvement.

• Your location. Your schedule. Pick your time-zone, choose your preferred hours, and work from where you feel most productive.

• Flexibility and freedom are in our DNA. As long as you have a stable internet connection and the drive to thrive, you can work from anywhere.

• A workload you can rely on. We ensure a consistent flow of engaging, challenging product work.

• Recognition and reward. We acknowledge diligence and strong performance with annual pay increases.

Down to business!

• Rate Range: Your starting rate is negotiable depending on your skills and experience.

• This is an hourly, independent contractor position. Compensation is based solely on hours worked. As an independent contractor role, this position does not include PTO.

• Compensation is paid monthly via wire transfer.