AI Eval / Testing (Eval Engineer)

We are seeking a skilled AI Engineer to build, integrate, and operationalize AI/ML models and agent workflows on AWS and Azure as the core AI foundation, with Microsoft Copilot as the primary user experience layer. The role involves collaborating with AI Architects and data teams to deploy scalable, production-grade AI solutions that are grounded in enterprise data, governed responsibly, and optimized for real-world performance. The candidate should be able to own the full AI engineering lifecycle, from prototyping and integration through to production deployment and ongoing optimization. Key Responsibilities LLM & Agent Development

• Build, integrate, and iterate on LLM-powered agent experiences for enterprise knowledge access and workflow automation.

• Own prompt engineering, orchestration logic, and multi-agent workflow design using AWS Bedrock and Azure AI services.

• Implement grounding, citation enforcement, and refusal behavior patterns aligned with enterprise governance standards.

• Build structured triage and escalation logic within agent workflows to support robust, production-grade AI systems.

• Own the AI engineering layer end to end, from prototype through pilot validation and production deployment.

RAG & Retrieval Engineering

• Implement RAG pipelines using structured and unstructured enterprise data on AWS and Azure cloud-native services.

• Tune retrieval quality through vector search, re-ranking strategies, and context window optimization.

• Work with embedding models, chunking strategies, and hybrid retrieval approaches to improve answer relevance.

• Integrate vector databases such as Azure AI Search and Amazon OpenSearch to support enterprise RAG systems.

Evaluation & Quality Assurance

• Define and run evaluation frameworks to measure answer accuracy, hallucination rates, and response quality.

• Ensure relevance, freshness, observability, and security of AI outputs across production environments.

• Implement monitoring and drift detection for deployed AI models and LLM-based workflows.

Cloud & Platform Integration

• Build and deploy AI solutions on AWS (Bedrock, SageMaker, Lambda) and Azure (Azure AI Foundry, Azure ML, Azure Functions) as the primary cloud platforms.

• Integrate Microsoft Copilot as the enterprise user experience layer, connecting AI capabilities to end-user workflows.

• Integrate AI components with enterprise applications, APIs, and data sources to enable scalable, end-to-end AI workflows.

• Ensure compliance with security, privacy, and responsible AI guidelines across all AI deployments.

Collaboration & Delivery

• Collaborate with AI Architects and data teams to translate architectural designs into production-ready AI implementations.

• Work within cross-functional delivery teams to align AI solutions with business and product requirements.

• Support POCs and pilot programmes, demonstrating the value and feasibility of AI solutions before full-scale deployment.

• 6-10 years of overall experience with 3+ years building LLM-powered applications in production or near-production environments.

• Hands-on experience with AWS AI/ML services (Bedrock, SageMaker) and Azure AI services (Azure AI Foundry, Azure OpenAI Service, Azure ML).

• Experience integrating with Microsoft Copilot or building Copilot extensibility solutions (plugins, connectors, or agents).

• Hands-on experience with RAG architectures: vector search, embedding models, chunking strategies, and hybrid retrieval.

• Strong understanding of grounding techniques, hallucination mitigation, and AI evaluation methodologies.

• Experience with agent orchestration frameworks and patterns: multi-agent routing, workflow chaining, and context management.

• Strong Python skills; familiarity with LangChain, Semantic Kernel, or equivalent agent orchestration frameworks preferred.

• Ability to work autonomously and own the full AI engineering stack within a cross-functional delivery team.

Line of Business: Service & Support Pay Range: $95,540.00 - $127,379.99 About Heidelberg Materials Heidelberg Materials is one of the world's largest suppliers of building ma…