> Markdown version of [/jobs/ext/1184098-senior-software-engineer-spark-platform](https://www.wearedevelopers.com/jobs/ext/1184098-senior-software-engineer-spark-platform). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Software Engineer, Spark Platform - **Company:** DOORDASH, INC. - **Location:** San Francisco, CA, United States (Remote available) - **Experience:** Expert - **Salary:** $130,600.0 - $192,000.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Apache HTTP Server, Information Engineering, Distributed Systems, Network Topologies, Python (Programming Language), Azure Machine Learning, Scala (Programming Language), SQL Databases, Apache Spark, Data Lakes, Kubernetes, Information Technology, Amazon Elastic Mapreduce (EMR), Databricks - **Published:** July 4, 2026 - **Apply:** https://www.juju.com/job/00000000gdmrtb ## About the Role + B.S., M.S., or PhD in Computer Science or equivalent. + 6+ years of industry experience designing and operating distributed systems at scale. + Deep, hands-on experience with Apache Spark - internals, query execution, shuffle, the executor/driver model - at platform scale on Amazon EMR, Databricks, or an in-house deployment, with a focus on platform operations (runtime upgrades, cluster lifecycle, shuffle, observability, multi-tenant scheduling) rather than authoring individual Spark jobs. + Production experience with one or more of: remote/external shuffle systems (Celeborn, Magnet, Cosco, or similar), batch/big-data schedulers (YuniKorn, Volcano, Kueue, or the Spark-on-Kubernetes operator), or the observability and SRE patterns that make distributed compute platforms operable. + Strong fluency operating workloads on Kubernetes in production - operator patterns, executor pod lifecycle, network topology, and the multi-tenant failure modes that show up at scale. + Familiarity with data lake table formats such as Apache Iceberg or Delta Lake, and with the query and SQL engines that read them. + Track record of acting as a technical leader on a platform team - setting direction, mentoring, and partnering with management on roadmap and hiring. + Professional experience with Scala, Java, Python, or Go; strong SQL. ## Description As a Senior Software Engineer on Spark Platform, you will set the technical direction for our in-house Spark deployment and shape the architecture that will run DoorDash's data, analytics, and ML compute for the next five years and beyond. You will own the deep, cross-cutting problems that span the runtime, the shuffle service, the scheduler, and the overall service reliability - making the architectural calls that compound across the platform's lifetime. You will partner with the Engineering Manager on technical roadmap, hiring, and team shape, and act as the senior technical voice in cross-team partnerships with Data Engineering, ML Platform, and product engineering teams that depend on the platform. You must be located in San Francisco, Sunnyvale, Seattle, or New York City for this hybrid position. You will report into the Engineering Manager on our Spark Platform team. You're excited about this opportunity because you will… + Set the multi-year technical direction for an in-house Spark-on-Kubernetes platform - runtime, shuffle, scheduler, reliability - and make the architectural calls that compound for years. + Own the deepest distributed-systems problems on the team: shuffle architecture, multi-tenant scheduling, runtime performance, and the failure modes that only show up at scale. + Partner with the Engineering Manager on technical roadmap, hiring, interview design, and team shape as the team continues to grow. + Uplevel the rest of the team through design reviews, mentorship, and raising the bar on what we ship. + Represent Spark Platform in cross-team architecture forums and shape how data, analytics, and ML workloads land on the platform. ## Related Videos - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Flex your Energy: Building a Cloud-Native Platform for Renewable Energy Communities](https://www.wearedevelopers.com/videos/1990-flex-your-energy-building-a-cloud-native-platform-for-renewable-energy-communities) - [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) - [Understanding Kubernetes in a visual way](https://www.wearedevelopers.com/videos/100085-understanding-kubernetes-in-a-visual-way) - [Enabling intelligent logistics automation: home-grown Industrial IoT platform at Austrian Post](https://www.wearedevelopers.com/videos/2018-enabling-intelligent-logistics-automation-home-grown-industrial-iot-platform-at-austrian-post) - [Parquet, Delta, Iceberg & Ducklake - An introduction for developers](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Fully Remote Software Engineer Jobs](https://www.wearedevelopers.com/magazine/447-fully-remote-software-engineer-jobs) - [Résumé-Driven Development: How IT trends affect the job market for software developers](https://www.wearedevelopers.com/magazine/59-resume-driven-development-how-it-trends-affect-the-job-market-for-software-developers) - [React Developer Salary [2023]](https://www.wearedevelopers.com/magazine/198-react-developer-salary-2023) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline)