AI Engineer

Innovative IT Solutions Inc
Malvern, PA, United States
1 day ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Contract
Employment type
Full-time (> 32 hours)
Compensation
$104,000
Working hours
Regular working hours
Job source

Tech stack

LangGraph Framework JavaScript (Programming Language) A/B Testing Artificial Intelligence Amazon Web Services Computer Programming Software Debugging Human-Computer Interaction Python (Programming Language) Node.Js Regression Testing TypeScript
+10 more
Retrieval-Augmented Generation Large Language Models Grafana Multi-Agent Systems Prompt Engineering Front End Software Development Claude Virtual Agents Evaluation of Large Language Models Human in the Loop

Job description

We are seeking a hands-on AI Engineer to build and enhance an internal AI-powered developer platform. The primary focus is developing stateful, human-in-the-loop AI agent workflows using LangGraph and Python, along with designing and optimizing automated evaluation frameworks to measure agent performance., The internal developer platform consists of two major components:

  • Agent Plane: AI agent workflows built using LangGraph and Python.
  • Collaborative UI: A user interface built using Node.js to support human-agent interaction., * Design, develop, and debug stateful AI agent workflows using LangGraph.
  • Implement agent state management, checkpointing, interrupts, and tool-calling capabilities.
  • Extend and maintain automated LLM evaluation frameworks.
  • Conduct A/B testing across different agent workflows and prompt variations.
  • Optimize prompts through measurable improvements and regression testing.
  • Evaluate AI agent performance based on response quality, cost, and latency.
  • Develop and enhance human-agent interactive UI capabilities.
  • Ensure reliability, consistency, and performance of AI-driven workflows.

Requirements

The ideal candidate should have strong experience building production-grade AI agent workflows, implementing deterministic state management, and conducting automated regression testing.

Important: We are not looking for candidates whose experience is limited to basic RAG implementations or simple chatbot development. Candidates should demonstrate strong engineering fundamentals and hands-on experience with LangGraph, agent orchestration, and LLM evaluation frameworks., * Strong hands-on experience with LangGraph, including state graphs, checkpointing, interrupts, and tool-calling agents.

  • Experience with LLM evaluation frameworks, including golden datasets, LLM-as-a-Judge, and A/B testing.
  • Strong prompt engineering skills, including structured outputs and prompt optimization.
  • Strong programming experience with Python.
  • Experience developing and testing production-ready AI agent workflows.
  • Ability to measure and improve LLM performance through automated evaluation and regression testing.

Preferred Qualifications

  • Experience with Claude and AWS Bedrock.
  • Front-end development experience using JavaScript/TypeScript.
  • Experience with Tiptap or Y.js.
  • Familiarity with LLM observability tools such as Arize and OpenTelemetry.
  • Experience with Node.js and AWS.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…