AI Engineer
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+10 more
Job description
We are seeking a hands-on AI Engineer to build and enhance an internal AI-powered developer platform. The primary focus is developing stateful, human-in-the-loop AI agent workflows using LangGraph and Python, along with designing and optimizing automated evaluation frameworks to measure agent performance., The internal developer platform consists of two major components:
- Agent Plane: AI agent workflows built using LangGraph and Python.
- Collaborative UI: A user interface built using Node.js to support human-agent interaction., * Design, develop, and debug stateful AI agent workflows using LangGraph.
- Implement agent state management, checkpointing, interrupts, and tool-calling capabilities.
- Extend and maintain automated LLM evaluation frameworks.
- Conduct A/B testing across different agent workflows and prompt variations.
- Optimize prompts through measurable improvements and regression testing.
- Evaluate AI agent performance based on response quality, cost, and latency.
- Develop and enhance human-agent interactive UI capabilities.
- Ensure reliability, consistency, and performance of AI-driven workflows.
Requirements
The ideal candidate should have strong experience building production-grade AI agent workflows, implementing deterministic state management, and conducting automated regression testing.
Important: We are not looking for candidates whose experience is limited to basic RAG implementations or simple chatbot development. Candidates should demonstrate strong engineering fundamentals and hands-on experience with LangGraph, agent orchestration, and LLM evaluation frameworks., * Strong hands-on experience with LangGraph, including state graphs, checkpointing, interrupts, and tool-calling agents.
- Experience with LLM evaluation frameworks, including golden datasets, LLM-as-a-Judge, and A/B testing.
- Strong prompt engineering skills, including structured outputs and prompt optimization.
- Strong programming experience with Python.
- Experience developing and testing production-ready AI agent workflows.
- Ability to measure and improve LLM performance through automated evaluation and regression testing.
Preferred Qualifications
- Experience with Claude and AWS Bedrock.
- Front-end development experience using JavaScript/TypeScript.
- Experience with Tiptap or Y.js.
- Familiarity with LLM observability tools such as Arize and OpenTelemetry.
- Experience with Node.js and AWS.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦