Databricks Engineer

eSolutionsFirst LLC

Vienna, VA, United States

11 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

3 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Airflow Amazon Web Services Amazon S3 Microsoft Azure Big Data Cloud Storage Computer Programming Continuous Integration Data Architecture Information Engineering Data Governance Extract Transform Load (ETL)

+22 more

Data Security Data Systems DevOps Apache Hive Python (Programming Language) Performance Tuning SQL Databases Data Streaming Azure Data Factory Snowflake Apache Spark Git Microsoft Fabric Data Lakes Pyspark Apache Kafka Data Management Video Streaming Azure Synapse Analytics Data Pipelines Jenkins Databricks

Job description

We are seeking an experienced Senior Databricks Engineer to design, develop, and optimize enterprise-scale data platforms on Databricks. The ideal candidate will have strong expertise in building scalable data pipelines, cloud data engineering, and modern Lakehouse architectures. This role requires working closely with data architects, analytics teams, and business stakeholders to deliver high-performance data solutions., * Design and develop scalable data pipelines using Databricks.

Build and optimize Spark applications for large-scale data processing.
Develop ETL/ELT workflows using PySpark and SQL.
Implement Delta Lake best practices for data quality and performance.
Collaborate with architects and business teams to define data solutions.
Optimize Databricks clusters, jobs, and notebooks.
Implement CI/CD pipelines and DevOps best practices.
Monitor production workloads and troubleshoot performance issues.
Ensure data security, governance, and compliance standards are met.
Document technical solutions and mentor junior engineers.

Requirements

7+ years of experience in Data Engineering.
3+ years of hands-on experience with Databricks.
Strong experience with Apache Spark (PySpark/Scala/Spark SQL).
Experience building ETL/ELT pipelines using Databricks.
Strong knowledge of Delta Lake, Unity Catalog, and Lakehouse architecture.
Experience with Azure Databricks (preferred) or AWS Databricks.
Strong SQL programming skills.
Experience with Python and/or Scala.
Experience with cloud storage technologies (ADLS, S3, Azure Blob).
Knowledge of CI/CD, Git, Azure DevOps, or Jenkins.
Experience with workflow orchestration tools such as Airflow or Databricks Workflows.
Understanding of data modeling, performance tuning, and optimization.
Experience with streaming technologies is a plus (Kafka, Structured Streaming).
Strong communication and stakeholder management skills., * Azure Data Engineer or Databricks certifications.
Experience with Snowflake, Azure Synapse, or Microsoft Fabric.
Experience implementing data governance and security best practices.
Financial Services or Federal domain experience is a plus.