Senior / Lead Data Engineer - Hybrid - San Francisco Bay Area, CA - W2 only

Sarum LLC
San Francisco, CA, United States
1 day ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
1 year minimum
Working hours
Regular working hours
Job source

Tech stack

Data Analysis Microsoft Azure Big Data Code Review Computer Programming Data Architecture Information Engineering Data Infrastructure Extract Transform Load (ETL) Data Transformation Data Systems Data Warehousing
+12 more
Distributed Computing Environment SQL Databases Data Ingestion Azure Data Factory Sql Optimization Microsoft Fabric Pyspark Information Technology Data Management Data Pipelines Serverless Computing Databricks

Job description

We are seeking a Senior / Lead Data Engineer in San Francisco Bay Area, CA., 1. Data Engineering & Pipeline Development

  • Design, develop, and implement scalable data pipelines using PySpark, SQL, Azure, and Databricks.
  • Build and maintain robust ETL/ELT processes for large-scale data processing.
  • Develop efficient data ingestion, transformation, and integration solutions.
  • Optimize data pipelines for performance, scalability, and reliability.
  1. Azure Cloud & Microsoft Fabric * Leverage Azure Native Services to develop modern cloud-based data solutions. * Design and implement data engineering workflows using Azure Data Factory (ADF). * Utilize Azure Databricks for large-scale distributed data processing. * Work with Microsoft Fabric to build and enhance modern data platforms. * Integrate multiple data sources and services across the Azure ecosystem.

  2. Data Warehousing & Data Management * Design and implement scalable enterprise data warehousing solutions. * Apply data warehousing fundamentals and best practices. * Develop efficient SQL queries for data transformation, analysis, and reporting. * Ensure data quality, consistency, integrity, and reliability.

  3. Collaboration & Technical Leadership * Collaborate with business stakeholders, architects, and cross-functional engineering teams. * Translate business requirements into technical solutions and data architecture. * Provide technical guidance and mentorship to junior data engineers. * Conduct code reviews and promote best practices in development and implementation. * Troubleshoot complex data engineering challenges and production issues.

  4. Continuous Improvement * Stay updated with emerging technologies and trends in cloud data engineering. * Identify opportunities to improve data architecture, pipeline performance, and engineering processes. * Contribute to continuous improvement initiatives across the data platform.

Requirements

Client is seeking a highly skilled and experienced Senior / Lead Data Engineer to join its data engineering team for challenging, large-scale, and multi-year data transformation initiatives.

The ideal candidate will bring extensive experience in designing, developing, and implementing scalable data engineering solutions using Azure Native Services, PySpark, SQL, Microsoft Fabric, Azure Data Factory (ADF), and Databricks.

This role offers an excellent opportunity to work on modern cloud-based data platforms, solve complex engineering challenges, and contribute to enterprise-level data transformation projects.

The candidate should possess strong technical expertise, excellent problem-solving skills, and the ability to work independently while collaborating with cross-functional teams., * 10+ years of overall IT experience.

  • 8+ years of hands-on Data Engineering experience.
  • 4+ years of experience working with Azure cloud technologies.
  • Strong hands-on experience with Azure Data Factory (ADF), Azure Databricks, and Azure Functions.
  • 1-2 years of hands-on experience with Microsoft Fabric.
  • Strong programming and development expertise in PySpark.
  • Advanced SQL skills, including querying, manipulating, and analyzing large datasets.
  • Strong understanding of ETL/ELT processes and data warehousing fundamentals.
  • Experience designing and implementing large-scale data pipelines and cloud data solutions.
  • Ability to work independently and deliver in a fast-paced environment.
  • Excellent communication, analytical, and problem-solving skills.

Good-to-Have Skills

  • Databricks Certification.
  • Strong understanding of Data Modeling techniques.
  • Azure Architect Certification.
  • Experience with modern cloud data architecture and enterprise data transformation initiatives.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…