Data Engineer - Databricks

BuzzClan

Beaverton, OR, United States

3 days ago

Apply on www.wayup.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

5 years minimum

Working hours

Regular working hours

Job source

www.wayup.com

Tech stack

Query Performance Airflow Amazon Web Services Microsoft Azure Computer Programming Continuous Integration Data as a Services Information Engineering Data Integration Extract Transform Load (ETL) Data Warehousing DevOps

+16 more

Apache Hive Python (Programming Language) Performance Tuning SQL Databases Data Streaming Workflow Management Systems Enterprise Data Management Data Processing Azure Data Factory Git Data Lakes Pyspark Information Technology Data Management Data Pipelines Databricks

Job description

Design, develop, and maintain scalable ETL/ELT data pipelines and enterprise data solutions.
Develop and optimize data processing solutions using Databricks, PySpark, Spark SQL, and Delta Lake.
Build reliable and high-performance data pipelines using Python and SQL.
Implement data integration and transformation processes across various enterprise data sources.
Work with cloud platforms such as Azure, AWS, or GCP , with preference for Azure.
Develop and manage data workflows using orchestration tools such as Azure Data Factory (ADF) and/or Apache Airflow.
Apply data modeling, data warehousing, and performance optimization best practices.
Implement data quality, scalability, reliability, and performance improvements across data platforms.
Use Git, CI/CD, and DevOps practices to support automated and reliable data deployments.
Collaborate with business stakeholders, data architects, developers, and other cross-functional teams to deliver enterprise data solutions.
Troubleshoot complex data pipeline and processing issues and provide effective technical solutions.
Contribute to modern Databricks capabilities such as Delta Live Tables (DLT), Unity Catalog, and Structured Streaming where applicable.

Requirements

We are seeking an experienced Data Engineer with strong Databricks expertise to design, develop, and maintain scalable enterprise data pipelines and data engineering solutions. The ideal candidate will have hands-on experience with Databricks, PySpark, Spark SQL, Delta Lake, Python, and SQL , along with experience in cloud platforms and modern data engineering practices. Candidates must be local to or able to work onsite in Beaverton, Oregon 3 days per week., + 6+ years of overall IT experience , including 5+ years of Data Engineering experience.

Strong hands-on experience with Databricks.
Strong expertise in PySpark, Spark SQL, and Delta Lake.
Strong programming skills in Python and SQL.
Proven experience building scalable ETL/ELT pipelines.
Experience with Azure, AWS, or GCP , preferably Azure.
Experience with Azure Data Factory (ADF) and/or Airflow.
Strong understanding of data modeling and data warehousing.
Experience with data pipeline and query performance optimization.
Hands-on experience with Git, CI/CD, and DevOps practices.
Strong analytical, problem-solving, collaboration, and communication skills. Preferred Skills
Delta Live Tables (DLT)
Unity Catalog
Structured Streaming
Azure cloud data services
Experience working with large-scale enterprise data platforms.