Databricks Data Engineer

Robotics Technologies LLC
Atlanta, GA, United States
1 day ago
Apply on www.indeed.com
Prepare application

Role details

Contract type
Temporary contract
Employment type
Full-time (> 32 hours)
Compensation
$158,080
Working hours
Regular working hours
Job source

Tech stack

Amazon Web Services Microsoft Azure Cloud Storage Computer Programming Databases Continuous Integration Data Architecture Data Validation Data Governance Extract Transform Load (ETL) Data Security Data Systems
+19 more
Data Warehousing Apache Hive Python (Programming Language) Cloud Services SQL Databases Data Streaming Workflow Management Systems Data Processing Apache Spark Git Data Lakes Pyspark Infrastructure Automation Frameworks Apache Kafka Machine Learning Operations Terraform Stream Processing Data Pipelines Databricks

Job description

  • We are looking for a skilled Databricks Data Engineer to develop and manage enterprise-scale data processing solutions using Databricks, Apache Spark, PySpark, and Delta Lake.
  • The ideal candidate should have strong experience in building data pipelines, implementing lakehouse architecture, optimizing Spark workloads, and integrating cloud data platforms. The candidate will work with data architects, analysts, and engineering teams to deliver scalable and reliable data solutions., * Design and develop scalable data engineering solutions using Databricks.
  • Build batch and streaming data pipelines using Apache Spark.
  • Develop data transformations using PySpark and Spark SQL.
  • Implement lakehouse architecture using Delta Lake.
  • Manage data access, governance, and permissions through Unity Catalog.
  • Build and maintain automated workflows using Databricks Workflows.
  • Implement incremental data processing and CDC pipelines.
  • Optimize Spark jobs and improve processing performance.
  • Integrate Databricks with cloud storage and enterprise databases.
  • Develop data validation and quality monitoring processes.
  • Collaborate with data scientists and analytics teams.
  • Troubleshoot production issues and ensure pipeline reliability.

Requirements

  • Strong hands-on experience with Databricks.
  • Advanced programming skills in Python and PySpark.
  • Strong knowledge of Apache Spark and Spark SQL.
  • Experience with Delta Lake and lakehouse architecture.
  • Knowledge of Unity Catalog and data governance.
  • Experience with Delta Live Tables / Lakeflow.
  • Experience building ETL and ELT pipelines.
  • Knowledge of AWS, Azure, or GCP cloud platforms.
  • Experience with SQL, data modeling, and data warehousing.
  • Familiarity with Git, CI/CD, and workflow orchestration., * Experience with Databricks Lakeflow and Lakeflow Connect.
  • Knowledge of Apache Kafka and real-time data processing.
  • Experience with Terraform and cloud infrastructure automation.
  • Knowledge of MLflow and machine learning workflows.
  • Experience with enterprise data migration projects.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.indeed.com
Prepare application

Good distractions

Loading talks and stories from around this role…