Senior / Lead Data Engineer - Hybrid - San Francisco Bay Area, CA - W2 only
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+12 more
Job description
We are seeking a Senior / Lead Data Engineer in San Francisco Bay Area, CA., 1. Data Engineering & Pipeline Development
- Design, develop, and implement scalable data pipelines using PySpark, SQL, Azure, and Databricks.
- Build and maintain robust ETL/ELT processes for large-scale data processing.
- Develop efficient data ingestion, transformation, and integration solutions.
- Optimize data pipelines for performance, scalability, and reliability.
-
Azure Cloud & Microsoft Fabric * Leverage Azure Native Services to develop modern cloud-based data solutions. * Design and implement data engineering workflows using Azure Data Factory (ADF). * Utilize Azure Databricks for large-scale distributed data processing. * Work with Microsoft Fabric to build and enhance modern data platforms. * Integrate multiple data sources and services across the Azure ecosystem.
-
Data Warehousing & Data Management * Design and implement scalable enterprise data warehousing solutions. * Apply data warehousing fundamentals and best practices. * Develop efficient SQL queries for data transformation, analysis, and reporting. * Ensure data quality, consistency, integrity, and reliability.
-
Collaboration & Technical Leadership * Collaborate with business stakeholders, architects, and cross-functional engineering teams. * Translate business requirements into technical solutions and data architecture. * Provide technical guidance and mentorship to junior data engineers. * Conduct code reviews and promote best practices in development and implementation. * Troubleshoot complex data engineering challenges and production issues.
-
Continuous Improvement * Stay updated with emerging technologies and trends in cloud data engineering. * Identify opportunities to improve data architecture, pipeline performance, and engineering processes. * Contribute to continuous improvement initiatives across the data platform.
Requirements
Client is seeking a highly skilled and experienced Senior / Lead Data Engineer to join its data engineering team for challenging, large-scale, and multi-year data transformation initiatives.
The ideal candidate will bring extensive experience in designing, developing, and implementing scalable data engineering solutions using Azure Native Services, PySpark, SQL, Microsoft Fabric, Azure Data Factory (ADF), and Databricks.
This role offers an excellent opportunity to work on modern cloud-based data platforms, solve complex engineering challenges, and contribute to enterprise-level data transformation projects.
The candidate should possess strong technical expertise, excellent problem-solving skills, and the ability to work independently while collaborating with cross-functional teams., * 10+ years of overall IT experience.
- 8+ years of hands-on Data Engineering experience.
- 4+ years of experience working with Azure cloud technologies.
- Strong hands-on experience with Azure Data Factory (ADF), Azure Databricks, and Azure Functions.
- 1-2 years of hands-on experience with Microsoft Fabric.
- Strong programming and development expertise in PySpark.
- Advanced SQL skills, including querying, manipulating, and analyzing large datasets.
- Strong understanding of ETL/ELT processes and data warehousing fundamentals.
- Experience designing and implementing large-scale data pipelines and cloud data solutions.
- Ability to work independently and deliver in a fast-paced environment.
- Excellent communication, analytical, and problem-solving skills.
Good-to-Have Skills
- Databricks Certification.
- Strong understanding of Data Modeling techniques.
- Azure Architect Certification.
- Experience with modern cloud data architecture and enterprise data transformation initiatives.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦