Databricks Data Engineer

Trebecon LLC

Charlotte, NC, United States

6 days ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

8 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Application Programming Interfaces (APIs) Amazon Web Services Microsoft Azure Cloud Storage Computer Programming Databases Data Architecture Information Engineering Data Governance Extract Transform Load (ETL) Data Transformation Data Warehousing

+13 more

Python (Programming Language) SQL Databases Data Processing Google Cloud Sql Optimization Apache Spark Git Data Layers Data Lakes Pyspark Software Version Control Data Pipelines Databricks

Job description

Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
Build and optimize data processing solutions using Apache Spark, Python, and SQL.
Implement Delta Lake tables and support schema evolution, data quality, and ACID transactions.
Develop Bronze, Silver, and Gold data layers following the Medallion/Lakehouse architecture.
Build batch and near-real-time data pipelines using Spark and Databricks.
Develop and manage Databricks Jobs and Workflows.
Integrate data from databases, APIs, cloud storage, and other enterprise data sources.
Perform data transformation, cleansing, validation, and enrichment.
Optimize Spark applications, SQL queries, and data pipelines for performance and cost.
Implement data governance, security, and access controls using Unity Catalog.
Monitor production pipelines, troubleshoot failures, and resolve data-quality issues.
Collaborate with Data Architects, Analysts, Developers, and business stakeholders.
Follow development best practices for version control, testing, deployment, and documentation.

Requirements

8+ years of experience in Data Engineering.
Strong hands-on experience with Databricks.
Strong proficiency in PySpark / Apache Spark.
Advanced SQL skills.
Strong programming experience with Python.
Hands-on experience with Delta Lake.
Experience with ETL/ELT pipeline development.
Knowledge of Lakehouse and Medallion architecture.
Experience with Databricks Workflows/Jobs.
Experience with Unity Catalog and data governance.
Strong understanding of data modeling and data warehousing concepts.
Experience working with cloud platforms such as AWS, Azure, or Google Cloud Platform.
Experience with Git and CI/CD practices.