> Markdown version of [/jobs/ext/4621385-ai-engineer-generative-ai-engineer](https://www.wearedevelopers.com/jobs/ext/4621385-ai-engineer-generative-ai-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# AI Engineer / Generative AI Engineer

- **Company:** TMS LLC
- **Location:** Los Angeles, CA, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Amazon Web Services, Cloud Engineering, Continuous Integration, Github, Python (Programming Language), Software Engineering, Retrieval-Augmented Generation, Large Language Models, AgentCore, Generative AI, Langfuse — LLM Observability and Analytics Platform, Agentic-AI, RAGAS (Retrieval Augmented Generation Assessment), AWS Data Analytics, Machine Learning Operations, Claude, Agent SDK, Service Stack
- **Published:** November 7, 2026
- **Apply:** https://www.juju.com/job/28_25303_1811f250b81554e174fa5cf21e3d609d

## About the Role

* Experience in Python and agentic AI development.
 * Hands- on work with AWS Bedrock AgentCore Evaluations (including Bedrock evaluation).
 * Ability to create, curate, and use golden datasets.
 * Experience designing LLM-as-a-Judge systems and calibrating them against human labels.
 * Automated evaluation implementation and integration into CI gates.
 * Experience with Langfuse and/or OpenTelemetry.

Technology stack

 * Python, Claude Agent SDK
 * AWS Bedrock AgentCore, AWS Bedrock AgentCore Runtime
 * LLM-as-a-Judge, CI/CD
 * Langfuse, OpenTelemetry
 * RAG, AWS data processing workflows
 * Bedrock evaluation, * RAGAS or equivalent evaluation frameworks.
 * Trajectory evaluation and agentic red-teaming.
 * Bedrock Guardrails, Strands Agents SDK, and GitHub Actions., * 15+ years of experience (minimum).
 * 6-8+ years of software engineering, cloud engineering, or AI/ML experience.
 * Hands-on experience with production AI agents and continuous evaluation.
 * Strong problem-solving and troubleshooting skills.

## Description

* Develop AI agents using Python and Claude Agent SDK.
 * Deploy agents using AWS Bedrock AgentCore Runtime.
 * Design and maintain golden datasets for reliable evaluation coverage.
 * Implement LLM-as-a-Judge evaluations, including calibration against human labels.
 * Integrate automated evaluation workflows into CI/CD pipelines as quality gates.
 * Set up Langfuse and/or OpenTelemetry for AI observability, tracing, and monitoring.
 * Evaluate agent outcomes including accuracy, task completion, tool execution, and performance.
 * Support RAG pipelines and AWS data processing workflows.