> Markdown version of [/jobs/ext/4762998-director-ai-systems-engineering-evaluation](https://www.wearedevelopers.com/jobs/ext/4762998-director-ai-systems-engineering-evaluation). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Director, AI Systems Engineering & Evaluation - **Company:** Novartis Institutes For Biomedical Research, Inc. - **Location:** London, United Kingdom - **Experience:** Expert - **Salary:** £100,240 - **Contract:** Permanent contract - **Skills:** LangGraph Framework, Application Programming Interfaces (APIs), Agile Methodology, Artificial Intelligence, Computer Programming, Data Governance, Data Visualization, Software Design Patterns, Knowledge-Based Systems, Machine Learning, Natural Language Processing, Software Engineering, Management of Software Versions, Digital Twin, Retrieval-Augmented Generation, Large Language Models, Deep Learning, Model Validation, Agentic-AI, Data Strategy, Build Management, AgentOps, CrewAI, AutoGen, Machine Learning Operations, Human in the Loop, GXP - **Published:** November 18, 2026 - **Apply:** https://startup.jobs/director-ai-systems-engineering-evaluation-u175-fcrs-us175-novartis-ins-10802143 ## About the Role · 10+ years in Data Science or AI, with demonstrable experience building and deploying production AI/ML systems · Hands-on experience with agentic frameworks (e.g. LangGraph, AutoGen, CrewAI, or equivalent) and with turning prototypes into reusable, production-grade capabilities · Proven experience defining evaluation approaches, benchmarks and production-readiness criteria for LLM or agentic systems · Experience leading senior technical teams and external partners in complex, regulated environments; healthcare or life sciences experience preferred, Agile (Programming Methodology) (Inactive), Artificial Intelligence (AI), Business Value Creation, Change Management, Curious Mindset, Data Governance, Data Literacy, Data Quality, Data Science, Data Strategies, Data Visualization, Deep Learning, Learning Agility, Machine Learning (ML), Machine Learning Algorithms, Mentorship, Nlp (Neuro-Linguistic Programming) And Genai (Inactive), Organization Awareness (Inactive), Stakeholder Engagement, Statistical Analysis, Time Series Analysis, Vision (Inactive) ## Description This is a senior AI leader role within Novartis Global Drug Development (GDD), responsible for designing, building and productionising the shared foundations that every Development AI system runs on. Think reusable agentic architectures, agent-accessible tools and knowledge (e.g. via MCP), evaluation harnesses, Physical AI patterns, and AgentOps pipelines, so that AI systems reach production faster, behave reliably and accelerate drug development timelines and time-to-patient. The role sits at the intersection of deep technical expertise (agentic frameworks, RAG, evaluation science, data and knowledge foundations) and strategic influence, setting the technical direction for how AI systems across Development are built, tested and trusted, in close collaboration with DSAI domain leaders, Product teams, Strategy & Governance and DDIT. Beyond hands-on design and delivery, the position carries accountability for evaluation and reliability: agent and model evaluation frameworks (reliability, hallucination risk, task completion, regulatory-grade auditability), red teaming, monitoring and production-readiness criteria, and the technical evidence that links each AI system to the business outcomes it is meant to deliver. The leader coaches and develops data scientists and AI engineers at the frontier of applied AI in pharma, and builds external visibility through publications and conference presentations. It requires 10+ years of production AI/ML experience with specific fluency in agentic design patterns, evaluation and responsible-AI practices. The role owns reusable AI foundations and the technical evidence of AI quality. Product roadmaps and value realisation sit with Product teams, policy and approvals with Strategy & Governance, and platforms and infrastructure operations with DDIT., · Develop and execute a roadmap for the shared AI foundations of Development, including reusable agentic architectures, retrieval and knowledge systems, simulation and digital twins, so that each new system builds on the last · Architect, prototype and productionise reusable agentic components: agent orchestration, memory and context management, tool integration and human-in-the-loop patterns, delivered as templates and reference implementations that domain teams can extend · Define agent and model evaluation frameworks covering reliability, hallucination risk, grounding, task completion rate and regulatory-grade auditability, and build the shared benchmark suites, evaluation datasets and test harnesses that apply them · Establish red teaming and adversarial testing for agentic systems (prompt injection, jailbreaks, retrieval attacks, tool misuse) in partnership with cybersecurity · Set production-readiness criteria and monitoring for AI systems in Development, including drift, degradation and failure-mode analysis, and support technical validation of GxP-relevant systems · Define MLOps, LLMOps and AgentOps practices (versioning, orchestration, tracing, observability) and partner with DDIT so the required signals are built into the platforms they run · Design and implement the data and knowledge foundations that agentic systems depend on for grounded, trustworthy decision-making; integrate internal/external data sources, APIs and knowledge bases as agent-accessible tools with appropriate access controls and governance · Connect technical performance evidence to the business-value KPIs each AI system is meant to move, working with Product teams so that decisions to scale, fix or retire rest on objective evidence · Turn proven project-level solutions into reusable capabilities, reducing duplication and time-to-build across domains · Strategically coordinate and prioritise team and partner resources across initiatives; proactively manage risks, stakeholder expectations and competing priorities · Proactively evaluate the evolving AI landscape (e.g. reasoning models, agent frameworks, MCP, evaluation tooling) and guide technology selection · Publish and present at conferences and academic institutions; serve as a Novartis ambassador for applied and agentic AI · Coach and develop data scientists and AI engineers in agentic design patterns, evaluation and responsible AI practices