> Markdown version of [/jobs/ext/1768191-principal-software-engineer](https://www.wearedevelopers.com/jobs/ext/1768191-principal-software-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Principal Software Engineer

- **Company:** CVS Health
- **Location:** Raleigh, NC, United States
- **Experience:** Experienced
- **Salary:** $144,200.0 - $288,400.0
- **Contract:** Permanent contract
- **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Optical Character Recognition (OCR), Audit Trail, BigQuery, Program Optimization, Continuous Integration, Distributed Systems, Fault Tolerance, Identity and Access Management, Python (Programming Language), Machine Learning, Performance Tuning, Regression Testing, Site Reliability Engineering Practices, Search Technologies, Software Engineering, Data Streaming, Google Cloud, Modern Ui, ReactJS, Fast Healthcare Interoperability Resources, Delivery Pipeline, Large Language Models, Concurrency, Prompt Engineering, Backend, AngularJS, Information Technology, Google Cloud Functions, Production Code, Health Level Seven International, Machine Learning Operations, Api Design, GPT, Microservices
- **Published:** July 13, 2026
- **Apply:** https://dejobs.org/x/x/09B397F6F99F40B98ADA4D00DB1ABB28/job/

## About the Role

* 10+ years of software engineering experience with a strong record of delivering production systems at scale
 * 5+ years as a hands-on Staff or Principal-level engineer (actively coding, not removed from implementation)
 * Expert proficiency in Python including performance tuning, concurrency, and system optimization
 * 5+ years building real-time distributed systems (event-driven, streaming, Pub-Sub) with clear latency and reliability requirements
 * 5+ years working in a major public cloud (GCP preferred: Vertex AI, GKE, Cloud Run, Pub/Sub, BigQuery, IAM, etc.)
 * Deep experience with microservices architecture, API design (REST), and resilient systems (retries, backpressure, tracing)
 * 3+ years building and operating production ML systems (MLOps, monitoring, retraining, deployment pipelines)
 * 3+ years developing LLM-powered applications in production, including prompt design, RAG/vector search, evaluation frameworks, and performance optimization
 * Experience building modern UI components with React or Angular
 * Strong computer science fundamentals, including distributed systems, algorithms, and concurrency, * Experience operating LLM systems in regulated environments (PII/PHI, governance, security, auditability)
 * Familiarity with vector databases (e.g., Vertex Vector Search, Pinecone, Weaviate, pgvector) and advanced retrieval techniques
 * Experience designing evaluation frameworks for LLM systems (gold datasets, regression testing, LLM-as-judge)
 * Strong background in CI/CD, SRE practices, and SLO-driven engineering
 * Experience with healthcare data standards (FHIR, HL7) and compliance frameworks (HIPAA, HITRUST)
 * Strong communication skills, with the ability to explain complex systems to both technical and non-technical audiences
 * Proven track record of mentoring engineers into senior and staff roles

Education:

 * Bachelor's degree in Computer Science or a related field (or equivalent experience)
 * Master's or PhD in Computer Science, Machine Learning, or a related discipline is a plus

## Description

We are hiring a Principal Software Engineer - a deeply hands-on technical leader who thrives on solving complex problems, writing production code, and elevating the engineers around them.

You will serve as the senior engineering voice across a portfolio of mission-critical clinical decision-support applications. These systems combine low-latency distributed architectures with advanced LLM, OCR, and ML pipelines running on Google Cloud.

This is not an "architect-from-the-sidelines" role. You will actively design systems, write critical code, ship to production, support live environments, and mentor engineers across UI, backend, and ML. Success in this role is measured by the quality, performance, reliability, and clinical impact of what you deliver-not by presentations.

What You'll Do

 * Own architecture and execution Lead the technical direction of a real-time, multi-service clinical platform on GCP, while personally building the most critical and performance-sensitive components.
 * Build production-grade LLM systems Design and operate scalable LLM pipelines (e.g., Claude, Gemini, GPT-4-class) including prompt engineering, structured outputs, evaluation frameworks, retrieval/vector search, OCR-based document understanding, grounding, guardrails, and hallucination mitigation.
 * Deliver real-time, high-performance systems Architect distributed, event-driven, streaming services that support sub-second clinician experiences under heavy load, with strict SLOs and strong fault tolerance.
 * Operationalize machine learning Lead deployment and lifecycle management of models on Vertex AI, including feature pipelines, model registries, drift detection, A/B and shadow testing, and CI/CD-integrated quality controls.
 * Design for scalability and change Build flexible APIs, microservices, and data contracts that support rapid iteration, backed by strong testing, observability, and progressive delivery practices.
 * Raise the engineering bar Drive high-quality code reviews, establish standards across testing, CI/CD, security, and observability, and mentor engineers across full-stack and ML disciplines.
 * Own outcomes end-to-end Translate ambiguous clinical and business needs into delivered solutions, monitor real-world performance, and continuously iterate based on data.
 * Partner across disciplines Collaborate with Data Science, Product, Clinical SMEs, AI Governance, and Security teams to ensure solutions meet enterprise standards for compliance, privacy, and safety.


## Related Videos

- [Watch Tests Go Brrrr! : Getting Started with Cypress in ReactJS](https://www.wearedevelopers.com/videos/282-watch-tests-go-brrrr-getting-started-with-cypress-in-reactjs)
- [Developing the Backend with Stefan Lingler, CTO at Shpock](https://www.wearedevelopers.com/videos/100360-developing-the-backend-with-stefan-lingler-cto-at-shpock)
- [ Evaluating AI models for code comprehension](https://www.wearedevelopers.com/videos/1462-evaluating-ai-models-for-code-comprehension)
- [Coffee with Developers - Maria Apazoglou](https://www.wearedevelopers.com/videos/1209-coffee-with-developers-maria-apazoglou)
- [Nest.js - TypeScript in the backend can also be clean](https://www.wearedevelopers.com/videos/1033-nest-js-typescript-in-the-backend-can-also-be-clean)
- [Slopquatting, API Keys, Fun with Fonts, Recruiters vs AI and more - The Best of LIVE 2025 - Part 2](https://www.wearedevelopers.com/videos/1765-slopquatting-api-keys-fun-with-fonts-recruiters-vs-ai-and-more-the-best-of-live-2025-part-2)

## Related Articles

- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)
- [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud)
- [Résumé-Driven Development: How IT trends affect the job market for software developers](https://www.wearedevelopers.com/magazine/59-resume-driven-development-how-it-trends-affect-the-job-market-for-software-developers)