> Markdown version of [/jobs/ext/3962106-principal-ai-inference-systems-software-architect](https://www.wearedevelopers.com/jobs/ext/3962106-principal-ai-inference-systems-software-architect). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Principal AI Inference Systems Software Architect

- **Company:** CyberCoders
- **Location:** San Jose, CA, United States
- **Experience:** Expert
- **Salary:** $250,000.0 - $300,000.0
- **Contract:** Permanent contract
- **Skills:** Computer-Aided Design, Artificial Intelligence, C++ (Programming Language), Compilers, Nvidia CUDA, Computer Programming, Computer Engineering, Data Centers, Extract Transform Load (ETL), Memory Management, Python (Programming Language), Performance Tuning, Software Product Management, Software Architecture, Software Engineering, Systems Architecture, AI Infrastructure, Pytorch, Retrieval-Augmented Generation, Large Language Models, Distributed Inference, Non-Uniform Memory Access, Agentic-AI, Discretization, Information Technology, Hardware Acceleration, ROCm, Machine Learning Operations, TensorRT, VLLM, Model Inference, Decoding
- **Published:** October 27, 2026
- **Apply:** https://www.thejobnetwork.com/job/c8874b06-aacd-498a-927f-3b167c78cce7/principal-ai-inference-systems-software-architect

## About the Role

* BS/MS in Computer Science, Computer Engineering, Electrical Engineering, or related field (PhD preferred).
 * 10+ years of experience in systems software, AI infrastructure, computer architecture, inference systems, runtimes, or compilers.
 * Deep expertise in AI/ML inference workloads and large-scale AI systems.
 * Strong programming skills in C/C++ and/or Python.
 * Strong understanding of computer architecture, operating systems, memory systems, and data movement.
 * Experience translating AI workload requirements into scalable system architectures.
 * Expertise in performance analysis, modeling, and benchmarking.

Preferred Experience

 * LLM inference, transformers, KV-cache optimization, vLLM, PyTorch, CUDA, Triton, TensorRT, ROCm.
 * Quantization, FlashAttention, speculative decoding, distributed inference, and MoE architectures.
 * GPU systems, AI accelerators, HBM, DDR, CXL, NUMA, and memory disaggregation.
 * Hardware/software co-design for AI infrastructure and data center systems.

## Description

This is a highly technical role for an individual who deeply understands AI data center and edge AI inference workloads, model execution, systems software, memory management, and hardware/software co-design. The successful candidate will work closely with the Chief AI Systems Architect, AI Product Strategist, AI Hardware / Memory Systems Architect, and R&D engineering teams to understand how AI workloads execute on current infrastructure, identify software bottlenecks, and architect software that enables new hardware and memory technologies to deliver measurable customer value. We are particularly interested in investigating opportunities involving AI workload, software stack, and inference runtime usage of the hardware to deliver the required performance and efficiency., * Analyze and optimize AI inference workloads including LLMs, GenAI, Agentic AI, RAG, MoE, multimodal, and edge AI.

 * Architect AI runtime and systems software for efficient utilization of compute, memory, and accelerator resources.
 * Design software for advanced memory architectures, including HBM, DDR, CXL, memory pooling, disaggregated memory, and KV-cache management.
 * Collaborate with hardware architects on CXL, memory systems, AI accelerators, compression engines, and near-memory computing technologies.
 * Develop performance optimization strategies focused on throughput, latency, memory efficiency, power, and total cost of ownership.
 * Define software architectures enabling practical deployment of CXL-based memory expansion, pooling, and disaggregation.
 * Build prototypes, performance models, benchmarks, and emulation environments to validate new architectural concepts before silicon availability.