> Markdown version of [/jobs/ext/3361496-lead-scala-data-engineer](https://www.wearedevelopers.com/jobs/ext/3361496-lead-scala-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Scala Data Engineer - **Company:** ESG - **Location:** United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Agile Methodology, Amazon Web Services, Application Frameworks, Cloud Computing, Data Infrastructure, IBM InfoSphere DataStage, Linux, Distributed Data Store, Drools, Apache Hadoop, Hadoop Distributed File System, Apache Hive, Cloudera, Scala (Programming Language), Systems Integration, Apache Yarn, Cloudera Manager, Apache Spark, Git, Atlassian Tools, Software Version Control, Service Stack - **Published:** September 30, 2026 - **Apply:** https://www.dice.com/job-detail/79290cce-0d97-42ca-af8c-bcc78b4acbc9 ## About the Role * 7+ years of experience developing enterprise-scale distributed data-processing applications.\\n * Strong hands-on Scala development experience, preferably 4-6+ years.\\n * 4-6+ years of Apache Spark and Hive development experience.\\n * Significant experience developing and maintaining applications on Cloudera Data Platform (CDP) 7.x or equivalent enterprise Hadoop environments.\\n * Hands-on experience implementing business rules using the Drools Rules Engine, preferably 4-6 years.\\n * Strong SQL development and query-optimization skills.\\n * Experience supporting Linux-based production environments.\\n * Strong experience troubleshooting distributed Spark applications in production.\\n * Experience with Git and modern version-control practices.\\n * Demonstrated experience taking technical ownership of production applications, including incidents, defects, enhancements, releases, deployments, and performance issues.\\n, * Medicaid or healthcare industry experience.\\n * Experience with Medicaid Encounter Processing.\\n * Experience with Cloudera Manager, HDFS, and YARN.\\n * Experience integrating with IBM DataStage.\\n * Familiarity with AWS infrastructure supporting Cloudera.\\n * Experience working in Agile environments using Jira and Confluence.\\n, n We are specifically looking for a Scala/Spark Data Engineer with strong Cloudera/CDP experience, rather than a Cloudera Administrator. The successful candidate should be capable of independently owning a production application across the complete Scala + Spark + Hive + Drools + Cloudera technology stack. \\n ## Description We are seeking an experienced Lead Scala Data Engineer to serve as the primary technical owner of an enterprise data-processing framework supporting Medicaid Encounter Processing and enterprise data ingestion. \\n \\n This is a hands-on technical ownership position responsible for maintaining and enhancing a mission-critical production application built with Scala, Apache Spark, Hive, Drools, and Cloudera Data Platform (CDP). The ideal candidate will have extensive experience developing distributed data-processing applications and supporting them in production. \\n \\n Key Responsibilities \\n \\n * Serve as the primary technical owner of the Scala/Spark application framework supporting Medicaid Encounter Processing.\\n * Maintain and enhance production applications developed using Scala, Spark, Hive, and Drools.\\n * Own Drools business-rule implementation and ongoing rule maintenance.\\n * Support enterprise ingestion and processing of provider, member, reference, eligibility, and encounter data.\\n * Own Spark and Hive batch-processing workflows.\\n * Troubleshoot production issues, identify root causes, and resolve application defects.\\n * Implement business, regulatory, and application changes.\\n * Manage production releases, version control, and deployment coordination.\\n * Perform Spark performance tuning and optimization.\\n * Monitor and provide basic operational support for the Cloudera Data Platform (CDP).\\n * Maintain technical documentation, operational procedures, and knowledge-transfer materials.\\n * Coordinate with infrastructure, cloud operations, QA, business, and other technical teams.\\n ## Related Videos - [Docker network without Docker](https://www.wearedevelopers.com/videos/1418-docker-network-without-docker) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Implementing continuous delivery in a data processing pipeline](https://www.wearedevelopers.com/videos/73-implementing-continuous-delivery-in-a-data-processing-pipeline) - [Docker exec without Docker](https://www.wearedevelopers.com/videos/1094-docker-exec-without-docker) - [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews) - [Flex your Energy: Building a Cloud-Native Platform for Renewable Energy Communities](https://www.wearedevelopers.com/videos/1990-flex-your-energy-building-a-cloud-native-platform-for-renewable-energy-communities) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Résumé-Driven Development: How IT trends affect the job market for software developers](https://www.wearedevelopers.com/magazine/59-resume-driven-development-how-it-trends-affect-the-job-market-for-software-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)