> Markdown version of [/jobs/ext/3017248-principal-ai-software-engineer](https://www.wearedevelopers.com/jobs/ext/3017248-principal-ai-software-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Principal AI Software Engineer

- **Company:** Microsoft
- **Location:** Redmond, WA, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** C (Programming Language), Java (Programming Language), JavaScript (Programming Language), Artificial Intelligence, C Sharp (Programming Language), C++ (Programming Language), Computer Clusters, Nvidia CUDA, Data Deduplication, Extract Transform Load (ETL), Distributed Systems, Dynamic Random-Access Memory, Memory Management, Python (Programming Language), Linux Kernel, Peer-To-Peer (P2P), Performance Tuning, Remote Direct Memory Access, Data Driven Tests, Software Engineering, Subsystems, Systems Architecture, System Software, Virtualization Technology, Large Language Models, Information Technology, TensorRT
- **Published:** September 20, 2026
- **Apply:** https://www.careerjet.com/job/use52b14ba8c2f3701e96b96730dfdfdfc/eaa

## About the Role

tiers, and hardware/software co-designed approaches for reducing inference TCO. Design and execute workload characterization studies for agentic, multi-turn, coding, reasoning, and long-context AI workloads to quantify memory consumption, latency, throughput, token efficiency, and system utilization. Analyze end-to-end data movement across GPU, CPU, storage, and networking subsystems, identifying optimization opportunities within GPU Direct Storage (GDS), GPU Direct RDMA (GDR), peer-to-peer memory transfers, and distributed inference pipelines. Develop software prototypes, framework extensions, and instrumentation to evaluate KV Cache offload, prefetching, migration, compression, deduplication, and memory-overcommit techniques. Build performance models and simulation frameworks to predict the impact of memory hierarchy innovations on large-scale inference deployments. Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding
in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience. 12+ years of experience in systems software (OS kernel, memory management, I/O stacks, Virtualization) with demonstrated track record of success, guiding architecture and software enabling. 10+ years of experience leading significant hardware/software co-design projects involving CPU and/or systems architecture and influencing technical direction. Deep expertise in Linux kernel internals, memory management, I/O subsystems, NUMA, DMA, and GPU/CPU/storage/network data paths. Hands-on experience with NVIDIA GPU software stacks including CUDA, NCCL, GPUDirect Storage (GDS), and GPUDirect RDMA (GDR). Understanding of AI inference infrastructure, large GPU clusters, inference serving architectures, and workload performance optimization. Experience characterizing and optimizing KV Cache intensive workloads including long-context, agentic, multi-turn, coding, and reasoning workloads.
Experience with inference frameworks such as vLLM, SGLang, and TensorRT-LLM. Familiarity with KV Cache technologies including LMCache, SGLang HiCache, cache offload, cache sharing, and memory tiering approaches. Experience designing or extending inference runtimes, scheduling systems, memory management components, or KV Cache subsystems. Understanding of disaggregated prefill/decode architectures, distributed inference serving, and multi-node cache-sharing topologies. Experience with CXL memory expansion, memory pooling, and memory tiering solutions in large-scale deployments. Software development skills in C/C++, Python, CUDA, and distributed systems. Skilled in partnering and influencing architects, hardware engineers, and software leads Ability to manage through ambiguity, bringing clarity and results orientation to engage and energize collaborators and stakeholders Collaboration skills, teamwork, and sense of presumed responsibility Verbal and written communication skills, and ability
to articulate and engage with both technical and non-technical stakeholders at all levels. Experience leading and driving complex projects with respect and integrity, including those with multiple workstreams spanning different business and technical disciplines. Intellectual curiosity and passion about learning and deploying new technologies. Problem-solving skills, analytical capabilities, and attention to detail

## Description

Lead full system software prototyping to develop capable proof-of-concepts to evaluate hardware/software co-designed capabilities for memory TCO reduction such as through memory tiering/pooling and overcommit solutions for Azure usages and deployment scenarios. Develop deep insights through workload characterization and correlation to identify systems optimization opportunities. Influence and shape hardware architecture and industry alignment, targeting three-to-six-year timeframe, with data-driven analysis, insights and recommendations. Lead characterization and optimization of Large Language Model (LLM) inference workloads with focus on KV Cache capacity, placement, migration, and utilization across GPU HBM, host DRAM, CXL memory expansion/pooling, SSD, and emerging memory tiers. Develop proof-of-concepts and evaluation frameworks to assess memory-tiering architectures for AI inference, including CXL pooled memory, memory expansion solutions, context-memory platforms, SSD-backed cache


## Related Videos

- [Building the Nervous System of AI - Michael Kagan (NVIDIA)](https://www.wearedevelopers.com/videos/2133-building-the-nervous-system-of-ai-michael-kagan-nvidia)
- [Coffee with Developers - Stephen Jones - NVIDIA](https://www.wearedevelopers.com/videos/1303-coffee-with-developers-stephen-jones-nvidia)
- [Tour de Force: Open-Source LLM Inference Optimization from Simple to Sophisticated](https://www.wearedevelopers.com/videos/100099-tour-de-force-open-source-llm-inference-optimization-from-simple-to-sophisticated)
- [10M Data Records Lost, Underwater Computing, and Psychedelic Fish - Matthias Geniar](https://www.wearedevelopers.com/videos/1908-10m-data-records-lost-underwater-computing-and-psychedelic-fish-matthias-geniar)
- [Trends, Challenges and Best Practices for AI at the Edge](https://www.wearedevelopers.com/videos/630-trends-challenges-and-best-practices-for-ai-at-the-edge)
- [Efficient deployment and inference of GPU-accelerated LLMs​](https://www.wearedevelopers.com/videos/929-efficient-deployment-and-inference-of-gpu-accelerated-llms)

## Related Articles

- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence)
- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)
- [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud)