> Markdown version of [/jobs/ext/4805279-senior-lead-data-engineer-hybrid-san-francisco-bay-area-ca-w2-only](https://www.wearedevelopers.com/jobs/ext/4805279-senior-lead-data-engineer-hybrid-san-francisco-bay-area-ca-w2-only). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior / Lead Data Engineer - Hybrid - San Francisco Bay Area, CA - W2 only - **Company:** Sarum LLC - **Location:** San Francisco, CA, United States (Remote available) - **Experience:** Expert - **Contract:** Contract - **Skills:** Data Analysis, Microsoft Azure, Big Data, Code Review, Computer Programming, Data Architecture, Information Engineering, Data Infrastructure, Extract Transform Load (ETL), Data Transformation, Data Systems, Data Warehousing, Distributed Computing Environment, SQL Databases, Data Ingestion, Azure Data Factory, Sql Optimization, Microsoft Fabric, Pyspark, Information Technology, Data Management, Data Pipelines, Serverless Computing, Databricks - **Published:** November 20, 2026 - **Apply:** https://www.dice.com/job-detail/4ab1f5b3-41d0-4ecb-8840-52b30a180609 ## About the Role Client is seeking a highly skilled and experienced Senior / Lead Data Engineer to join its data engineering team for challenging, large-scale, and multi-year data transformation initiatives. The ideal candidate will bring extensive experience in designing, developing, and implementing scalable data engineering solutions using Azure Native Services, PySpark, SQL, Microsoft Fabric, Azure Data Factory (ADF), and Databricks. This role offers an excellent opportunity to work on modern cloud-based data platforms, solve complex engineering challenges, and contribute to enterprise-level data transformation projects. The candidate should possess strong technical expertise, excellent problem-solving skills, and the ability to work independently while collaborating with cross-functional teams., * 10+ years of overall IT experience. * 8+ years of hands-on Data Engineering experience. * 4+ years of experience working with Azure cloud technologies. * Strong hands-on experience with Azure Data Factory (ADF), Azure Databricks, and Azure Functions. * 1-2 years of hands-on experience with Microsoft Fabric. * Strong programming and development expertise in PySpark. * Advanced SQL skills, including querying, manipulating, and analyzing large datasets. * Strong understanding of ETL/ELT processes and data warehousing fundamentals. * Experience designing and implementing large-scale data pipelines and cloud data solutions. * Ability to work independently and deliver in a fast-paced environment. * Excellent communication, analytical, and problem-solving skills. Good-to-Have Skills * Databricks Certification. * Strong understanding of Data Modeling techniques. * Azure Architect Certification. * Experience with modern cloud data architecture and enterprise data transformation initiatives. ## Description We are seeking a Senior / Lead Data Engineer in San Francisco Bay Area, CA., 1. Data Engineering & Pipeline Development * Design, develop, and implement scalable data pipelines using PySpark, SQL, Azure, and Databricks. * Build and maintain robust ETL/ELT processes for large-scale data processing. * Develop efficient data ingestion, transformation, and integration solutions. * Optimize data pipelines for performance, scalability, and reliability. 2. Azure Cloud & Microsoft Fabric * Leverage Azure Native Services to develop modern cloud-based data solutions. * Design and implement data engineering workflows using Azure Data Factory (ADF). * Utilize Azure Databricks for large-scale distributed data processing. * Work with Microsoft Fabric to build and enhance modern data platforms. * Integrate multiple data sources and services across the Azure ecosystem. 3. Data Warehousing & Data Management * Design and implement scalable enterprise data warehousing solutions. * Apply data warehousing fundamentals and best practices. * Develop efficient SQL queries for data transformation, analysis, and reporting. * Ensure data quality, consistency, integrity, and reliability. 4. Collaboration & Technical Leadership * Collaborate with business stakeholders, architects, and cross-functional engineering teams. * Translate business requirements into technical solutions and data architecture. * Provide technical guidance and mentorship to junior data engineers. * Conduct code reviews and promote best practices in development and implementation. * Troubleshoot complex data engineering challenges and production issues. 5. Continuous Improvement * Stay updated with emerging technologies and trends in cloud data engineering. * Identify opportunities to improve data architecture, pipeline performance, and engineering processes. * Contribute to continuous improvement initiatives across the data platform.