Senior Software Engineer - AI Infrastructure

• Design and implement highly scalable, reliable, and performant distributed systems that power Azure AI traffic management and scheduling.

• Drive architecture and technical direction for routing, capacity allocation, and workload optimization services.

• Develop algorithms and control-plane systems that intelligently distribute AI workloads across global infrastructure while meeting latency, availability, and cost objectives.

• Collaborate with partner teams to onboard new models, regions, and customer offerings while maintaining operational excellence.

• Build observability and operational tooling to monitor service health, capacity utilization, traffic patterns, and customer impact in near real-time.

• Lead investigations into complex production issues and drive systematic improvements to reliability, resiliency, and platform automation.

• Identify and execute opportunities to improve fleet efficiency, reduce operational overhead, and optimize utilization of premium AI infrastructure.

• Mentor engineers, review designs and code, and raise engineering standards across the organization.

• Drive engineering excellence through testing, automation, documentation, and operational best practices.

• Influence long-term platform strategy and contribute to roadmap planning for next-generation AI infrastructure.

• Bachelor’s Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience., * Master’s Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor’s Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.

• Experience with telemetry, metrics pipelines, or resource scheduling systems.

• Familiarity with cloud platforms (Azure, AWS, Google Cloud Platform) and container orchestration (Kubernetes, Service Fabric).

• Experience operating large-scale services with stringent reliability and availability requirements.

• Experience with distributed storage systems, telemetry platforms, and real-time data processing.

• Familiarity with AI/ML infrastructure, GPU fleets, inference serving systems, or large-scale model deployment platforms.

• Experience leading cross-team initiatives and driving projects from design through production deployment and operation.

Other Requirements:Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.

Join our team within AI Infrastructure, where we are building the AI data-plane that powers all LLM inferencing workloads across Microsoft and Azure customers-from cutting-edge startups to Fortune 500 enterprises. Our converged AI fabric delivers inference capabilities for all LLMs in Microsoft catalog, including OpenAI, Anthropic, Mistral, Cohere, Llama, and more.

As a Senior Software Engineer, you will shape the future of one of the largest and fastest-growing services in Azure, foundational to Microsoft’s AI strategy. Our mission is to serve models at scale-reliably, efficiently, and with ultra-low latency-enabling a rich set of AI-powered product experiences. This is a rapidly evolving space with immense opportunities to learn, innovate, and drive industry-wide impact!