> Markdown version of [/jobs/ext/620383-principal-software-engineer](https://www.wearedevelopers.com/jobs/ext/620383-principal-software-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Principal Software Engineer - **Company:** Amgen - **Location:** Thousand Oaks, CA, United States - **Experience:** Experienced - **Salary:** $157,021.0 - $212,440.0 - **Contract:** Permanent contract - **Skills:** A/B Testing, Artificial Intelligence, Airflow, Amazon Web Services, Microsoft Azure, Big Data, Cloud Computing, Cloud Engineering, Code Review, Databases, Continuous Integration, Data Architecture, Information Engineering, Data Integrity, DevOps, Distributed Computing Environment, Distributed Systems, Apache Hadoop, Python (Programming Language), Machine Learning, Online Analytical Processing, Online Transaction Processing, Performance Tuning, Tensorflow, Azure Machine Learning, Search Technologies, User-Centered Design, Google Cloud, Pytorch, Large Language Models, Multi-Agent Systems, Apache Spark, Deep Learning, Generative AI, Kubernetes, Information Technology, HuggingFace, Data Management, Machine Learning Operations, Virtual Agents, Data Pipelines, Docker, Databricks, Data Generation - **Published:** June 12, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=d03282f61e1c9490 ## About the Role Do you have experience in Team development?, Do you have a Master's degree?, Let's do this. Let's change the world. In this vital role you will play a pivotal role in building and scaling our machine learning models from development to production. Your expertise in both machine learning and operations will be essential in creating efficient and reliable ML pipelines. A background in data engineering, including experience with data pipelines and distributed data processing, is a strong plus., Doctorate degree and 2 years of experience OR Master's degree and 4 years of experience OR Bachelor's degree and 6 years of experience OR Associate's degree and 10 years of experience OR High school diploma / GED and 12 years of experience * Deep expertise in machine learning, deep learning, and Generative AI (LLMs, transformers, embeddings, fine-tuning techniques). * Proven track record of leading and delivering production-grade ML/GenAI systems end-to-end with measurable business impact with strong experience in designing scalable system architectures for ML and GenAI, including distributed systems and high-throughput pipelines. * Expertise in MLOps/LLMOps ecosystems (MLflow, Kubeflow, Airflow, CI/CD, Docker, Kubernetes). * Strong system design, architecture, and problem-solving skills with the ability to operate independently and lead large initiatives. * Demonstrated proficiency in leveraging cloud platforms (AWS, Azure, GCP) for data engineering solutions. Strong understanding of cloud architecture principles and cost optimization strategies. * * Proven ability to mentor and guide junior and mid-level engineers (L4/L5). Good-to-Have Skills: Degree in computer science, Statistics, and Data Science preferred. Master's degree and 6+ years experience Or Bachelor's degree and 8+ years' experience * Cloud Computing certificate preferred * Experience with big data ecosystems (Spark, Hadoop) and large-scale data processing. * Strong background in data engineering and building scalable data platforms. * Advanced proficiency in Python and modern ML/AI frameworks (PyTorch, TensorFlow, Hugging Face, LangChain or similar). * Experience designing robust evaluation and validation systems, including automated evals, human-in-the-loop, safety testing, and monitoring frameworks. * Extensive experience with RAG architectures, vector databases, and knowledge-grounded systems. * Strong understanding of agentic AI frameworks, including orchestration, planning, memory, and tool use. * Knowledge of advanced statistical modeling, experimentation design, and causal inference. * Experience with NLP, semantic search, embeddings, and vector search systems. * Familiarity with Responsible AI practices, including fairness, explainability, governance, and regulatory considerations. * Experience with cloud-native AI/ML services (AWS, Azure, GCP) and cost/performance optimization. * Experience with Databricks platform for enterprise-scale ML and GenAI workloads. * Exposure to advanced evaluation techniques, including red-teaming, adversarial testing, and synthetic data generation. * Experienced with data modeling and performance tuning for both OLAP and OLTP databases * Experienced with Apache Spark, Apache Airflow and Databricks platform ## Description * Lead the end-to-end design, development, and delivery of machine learning and Generative AI (GenAI) solutions, from problem framing to production deployment and business impact realization. * Act as the technical owner for large-scale ML/GenAI initiatives, driving architecture decisions, scalability, reliability, and long-term maintainability. * Design and implement advanced agentic AI systems, including multi-agent architectures, reasoning workflows, tool integration, and autonomous decision-making systems. * Define and institutionalize evaluation, validation, and governance frameworks for ML/GenAI systems, including model performance, prompt evaluation, safety guardrails, hallucination mitigation, and compliance. * Partner directly with business stakeholders and product leaders to understand objectives, translate them into AI/ML solutions, and ensure measurable value delivery. * Establish and enforce best practices in MLOps, LLMOps, and DevOps, including CI/CD, monitoring, observability, reproducibility, and cost optimization. * Architect and oversee scalable cloud-based ML/GenAI platforms leveraging AWS, GCP, or Azure. * Drive experimentation strategy, including A/B testing, prompt optimization, and iterative improvement of models and agent workflows. * Provide technical leadership and mentorship to L4 and L5 engineers, including design reviews, code reviews, and career guidance. * Lead cross-functional collaboration across data science, engineering, product, and business teams to deliver integrated AI solutions. * Stay at the forefront of advancements in machine learning, Generative AI, and agentic systems, and drive adoption of new technologies and approaches. * Design, develop, and implement robust data architectures and platforms to support ML Operation. * Ensuring data integrity, accuracy, and consistency through rigorous quality checks and monitoring. ## Related Videos - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [Photonic Computing: Programming a New Class of AI Accelerators (incl. Live Coding)](https://www.wearedevelopers.com/videos/100196-photonic-computing-programming-a-new-class-of-ai-accelerators-incl-live-coding) - [Navigating the AI Revolution in Software Development](https://www.wearedevelopers.com/videos/1266-navigating-the-ai-revolution-in-software-development) - [Geometric deep learning for drug discovery](https://www.wearedevelopers.com/videos/264-geometric-deep-learning-for-drug-discovery) - [Docker build without Docker](https://www.wearedevelopers.com/videos/100114-docker-build-without-docker) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it) - [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models) - [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)