> Markdown version of [/jobs/ext/1305549-ai-inference-infrastructure-software-engineer](https://www.wearedevelopers.com/jobs/ext/1305549-ai-inference-infrastructure-software-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # AI Inference Infrastructure Software Engineer - **Company:** NIO - **Location:** San Jose, CA, United States - **Experience:** Expert - **Salary:** $192,100.0 - $249,600.0 - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Systems Engineering, C++ (Programming Language), Program Optimization, Nvidia CUDA, Computer Programming, Computer Engineering, Linux Kernel, Language Modeling, Open Source Technology, Tensorflow, Software Engineering, System Programming, Systems Integration, Load Balancing, Cloud Platform System, Pytorch, Large Language Models, Deep Learning, Caching, Information Technology, Optimization Algorithms, Hardware Acceleration, GPT - **Published:** July 17, 2026 - **Apply:** https://www.dice.com/job-detail/0c76e77a-9bbe-45a2-8d3f-3782b86c9d23 ## About the Role * 5+ years of hands-on software development experience in building and optimizing AI inference systems at scale. * Direct experience in LLM/VLM model internals, including Transformer-based architectures, inference bottlenecks, and optimization techniques. * Strong expertise in performance engineering: kernel development, parallelism strategies, memory optimization, and distributed inference systems. * Proficiency with GPU/NPU programming (CUDA, or vendor-specific SDKs), compiler toolchains, and deep learning frameworks (PyTorch, or TensorFlow). * Strong programming skills in C/C++, with a track record of delivering high-performance, production-grade software. * Solid foundation in computer architecture, systems programming (CPU/GPU pipelines, memory hierarchy, scheduling), and embedded systems. * BS/MS in Computer Science, Computer Engineering, or related technical field. * Excellent communication and collaboration skills, with the ability to work across cross-functional teams., * Master's or PhD degree in Computer Science, Electrical/Computer Engineering, or related fields, plus 5 years industry experience * Experience building inference serving systems for large models, including batching, scheduling, caching, and load balancing. * Expertise in hardware-aware model optimization (e.g., kernel fusion, mixed precision, quantization, pruning). * Familiarity with edge and embedded AI, including real-time constraints and limited-resource optimization. * Contributions to widely used AI frameworks, libraries, or performance-critical software (open source or proprietary). ## Description We are looking for a senior AI Inference Infrastructure Software Engineer with strong hands-on experience building, optimizing, and deploying high-performance, scalable inference systems. This position is focused on designing, implementing, and delivering production-grade software that powers real-world applications of Large Language Models (LLMs) and Vision-Language Models (VLMs). This is an exciting opportunity for an engineer who thrives at the intersection of AI systems, hardware acceleration, and large-scale robust deployment, and who wants to see their contributions ship in production, at scale. In this role, you will directly shape the architecture, roadmap and performance of AI capabilities of our AIOS platform, driving innovations that make LLM/VLM systems fast, efficient, and scalable across cloud, edge, and hybrid edge-cloud environments. You will work closely with system, hardware, and product teams to deliver high-performance inference kernels for hardware accelerators, design scalable inference serving systems, and integrate optimizations such tensor parallelism and custom kernels into production pipelines. Your work will have immediate impact, powering intelligent automotive systems in the next generation of electric vehicles. Roles and Responsibilities: * Design and implement high-performance, scalable inference systems for LLMs and VLMs across cloud, edge, and edge-cloud hybrid platforms. * Develop and optimize custom kernels and operators for specific hardware accelerators (GPU, NPU, DSP, etc.), improving throughput, latency, and memory efficiency. * Integrate advanced optimization techniques such as KV-cache management, tensor/model parallelism, quantization, and memory-efficient execution into production inference systems. * Partner with system and hardware teams to ensure tight hardware-software integration and optimal performance across diverse compute environments. * Translate architectural requirements into robust, maintainable, production-ready software that meets performance, safety, and reliability standards. * Define and drive the evolution roadmap for LLM/VLM inference in the AIOS stack, ensuring scalability and adaptability to new workloads. * Stay ahead of industry trends and competitor solutions, applying best practices from both AI and large-scale systems engineering. ## Related Videos - [ Evaluating AI models for code comprehension](https://www.wearedevelopers.com/videos/1462-evaluating-ai-models-for-code-comprehension) - [HTTP headers that make your website go faster](https://www.wearedevelopers.com/videos/1676-http-headers-that-make-your-website-go-faster) - [Photonic Computing: Programming a New Class of AI Accelerators (incl. Live Coding)](https://www.wearedevelopers.com/videos/100196-photonic-computing-programming-a-new-class-of-ai-accelerators-incl-live-coding) - [Efficient deployment and inference of GPU-accelerated LLMs​](https://www.wearedevelopers.com/videos/929-efficient-deployment-and-inference-of-gpu-accelerated-llms) - [Developing an AI.SDK](https://www.wearedevelopers.com/videos/198-developing-an-ai-sdk) - [Speak, Code, Deploy: Transforming Developer Experience with Voice Commands](https://www.wearedevelopers.com/videos/1159-speak-code-deploy-transforming-developer-experience-with-voice-commands) ## Related Articles - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it) - [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence)