AI Engineer

You will work cross-functionally with data engineering, platform, product, and clinical teams to identify high-impact opportunities for AI automation-from data source onboarding to clinical decision support to evidence synthesis. Your focus will be on building production-grade agentic systems with rigorous validation, clear confidence scoring, and human-in-the-loop oversight to ensure reliability in a regulated healthcare environment. You will establish patterns, best practices, and tooling that allow the organization to scale AI-driven automation across multiple domains. You will measure agent performance and impact-tracking accuracy, hallucination rates, and real-world clinical outcomes. You will be part of a growing AI team that values both cutting-edge AI capabilities and deep healthcare domain understanding., * Design, build, and maintain agentic systems and LLM-powered applications that automate healthcare workflows, data pipelines, and clinical decision support - from conception through production deployment

• Build and orchestrate agents using LLM APIs (OpenAI, Anthropic, etc.) and agentic frameworks (LangChain, LangGraph, CrewAI, or custom orchestration) to solve complex, multi-step healthcare problems
• Develop prompt libraries, agent instructions, and reusable "skills" that improve agent accuracy, consistency, and reliability across different use cases and data domains
• Build validation and confidence-scoring layers that flag low-confidence agent decisions for human review before production deployment; establish guardrails and review workflows for agent-authored code and outputs
• Own end-to-end delivery of AI-automated systems - from problem scoping and requirements gathering through agent development, testing, and validated production deployment
• Implement rigorous evaluation and QA frameworks for agentic systems - including golden datasets, test cases, output validation, hallucination detection, and regression testing
• Establish and maintain evaluation metrics for agent performance, reliability, and clinical appropriateness; measure agent accuracy, hallucination rates, clinical validity, and real-world impact
• Implement observability, evaluation, and regression testing frameworks specific to agentic systems - decision tracing, lineage logging, and performance tracking
• Collaborate with data engineering and platform teams to integrate agent-built outputs (dbt models, transformation logic, recommendations) into existing data architectures and clinical workflows
• Ensure all agentic systems comply with healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices - including explainability, auditability, and clinician trust
• Continuously evaluate new LLM models, agent frameworks, prompt engineering techniques, and tooling; recommend adoption or migration based on healthcare-specific requirements (accuracy, cost, latency, regulatory alignment)
• Partner with data engineering to establish robust data validation and input validation layers for agents - agents are only as good as the data they operate on
• Lead experimentation and measurement of AI-automated systems impact on speed, quality, compliance, and cost across healthcare workflows
• Document agent architectures, prompt strategies, evaluation frameworks, and best practices for both technical and non-technical stakeholders
• Mentor AI Connector Engineers and other team members on agentic development patterns, LLM-powered application design, and responsible AI practices
• Work on-call as needed to support production agentic systems, troubleshoot agent issues, and respond to performance degradation or hallucination detection

• 3+ years of professional experience in data engineering, backend engineering, machine learning, or a related field
• 1+ years of hands-on experience building with LLM APIs and agentic orchestration frameworks - not just using AI coding assistants, but architecting agentic systems
• Strong Python and SQL proficiency
• Experience with cloud data platforms (AWS, Databricks)
• Solid understanding of data modeling, ETL/ELT patterns, and medallion architecture (Bronze/Silver/Gold)
• Experience building and consuming APIs
• Demonstrated experience with prompt engineering, agent evaluation, and validating LLM outputs
• Experience designing evaluation frameworks, test cases, and quality assurance for AI/ML systems
• Demonstrated ability to measure and track AI system performance through metrics and KPIs (accuracy, precision, recall, hallucination rates)
• Strong debugging and analytical skills, especially in ambiguous or novel technical territory
• Excellent written and verbal communication skills - this role requires documenting agent reasoning, decisions, and limitations clearly for both technical and non-technical audiences
• Comfortable working in a fast-moving environment with incomplete information and rapidly evolving AI/ML capabilities

Strong Pluses

• Experience with dbt or similar data transformation frameworks
• Familiarity with orchestration tools (Airflow, Databricks Workflows) and workflow automation
• Experience with agent evaluation and observability tooling (LangSmith, Langfuse, or custom frameworks)
• Background in healthcare, fintech, or another regulated/high-stakes domain where AI reliability is critical
• Experience building internal developer tooling, platform capabilities, or developer-facing products
• Hands-on experience with RAG (retrieval-augmented generation) or other grounding techniques for LLMs
• Familiarity with healthcare data formats and standards (FHIR, HL7, claims data, clinical NLP)
• Experience with model evaluation, fairness assessment, or bias detection in ML/AI systems
• Understanding of healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices
• Experience establishing QA frameworks, test plans, and quality metrics for ML/AI systems
• Startup or high-growth environment experience with rapid iteration and learning
• Published research, open-source contributions, or demonstrated thought leadership in AI/agentic systems