> Markdown version of [/jobs/ext/4789539-ai-engineer](https://www.wearedevelopers.com/jobs/ext/4789539-ai-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Engineer - **Company:** Innovative IT Solutions Inc - **Location:** Malvern, PA, United States - **Salary:** $104,000 - **Contract:** Contract - **Skills:** LangGraph Framework, JavaScript (Programming Language), A/B Testing, Artificial Intelligence, Amazon Web Services, Computer Programming, Software Debugging, Human-Computer Interaction, Python (Programming Language), Node.Js, Regression Testing, TypeScript, Retrieval-Augmented Generation, Large Language Models, Grafana, Multi-Agent Systems, Prompt Engineering, Front End Software Development, Claude, Virtual Agents, Evaluation of Large Language Models, Human in the Loop - **Published:** November 19, 2026 - **Apply:** https://www.dice.com/job-detail/9c54de48-5f08-4742-bc7a-2b74996256f8 ## About the Role The ideal candidate should have strong experience building production-grade AI agent workflows, implementing deterministic state management, and conducting automated regression testing. Important: We are not looking for candidates whose experience is limited to basic RAG implementations or simple chatbot development. Candidates should demonstrate strong engineering fundamentals and hands-on experience with LangGraph, agent orchestration, and LLM evaluation frameworks., * Strong hands-on experience with LangGraph, including state graphs, checkpointing, interrupts, and tool-calling agents. * Experience with LLM evaluation frameworks, including golden datasets, LLM-as-a-Judge, and A/B testing. * Strong prompt engineering skills, including structured outputs and prompt optimization. * Strong programming experience with Python. * Experience developing and testing production-ready AI agent workflows. * Ability to measure and improve LLM performance through automated evaluation and regression testing. Preferred Qualifications * Experience with Claude and AWS Bedrock. * Front-end development experience using JavaScript/TypeScript. * Experience with Tiptap or Y.js. * Familiarity with LLM observability tools such as Arize and OpenTelemetry. * Experience with Node.js and AWS. ## Description We are seeking a hands-on AI Engineer to build and enhance an internal AI-powered developer platform. The primary focus is developing stateful, human-in-the-loop AI agent workflows using LangGraph and Python, along with designing and optimizing automated evaluation frameworks to measure agent performance., The internal developer platform consists of two major components: * Agent Plane: AI agent workflows built using LangGraph and Python. * Collaborative UI: A user interface built using Node.js to support human-agent interaction., * Design, develop, and debug stateful AI agent workflows using LangGraph. * Implement agent state management, checkpointing, interrupts, and tool-calling capabilities. * Extend and maintain automated LLM evaluation frameworks. * Conduct A/B testing across different agent workflows and prompt variations. * Optimize prompts through measurable improvements and regression testing. * Evaluate AI agent performance based on response quality, cost, and latency. * Develop and enhance human-agent interactive UI capabilities. * Ensure reliability, consistency, and performance of AI-driven workflows.