Managing Sovereign AI Infrastructure: MLOps and LLMOps in Highly Regulated Banking IT

AI only becomes strategically relevant when it can be operated reliably, securely, and sustainably. In highly regulated environments such as the financial industry, this creates specific challenges around infrastructure, automation, and governance. This workshop provides practical insights into the design and operation of a sovereign AI platform and how MLOps and LLMOps are implemented in this context. We show how open-source LLMs are operated on-premises, how models are versioned, monitored, and automatically deployed and what challenges arise during inference operations. Using a production use case supporting around 200,000 workplaces, we demonstrate how AI assistant systems and AI agents can be built and operated under strict compliance and regulatory requirements. This session is a hands-on experience report for organizations that do not want to consume AI as a cloud API, but instead aim to operate AI as a controllable, sovereign infrastructure.