Lead Data Engineer - Databricks

TechDigital Corporation
Bolingbrook, IL, United States
1 day ago
Apply on www.careerbuilder.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
3 years minimum
Working hours
Regular working hours

Tech stack

Unity 3d Big Data Cloud Computing Cyber Security Data Architecture Information Engineering Data Infrastructure Data Security Python (Programming Language) SQL Databases Data Streaming Scripting
+8 more
Google Cloud Cloud Platform System Apache Spark IT Architecture Data Lakes Data Management Cisco Databricks

Job description

Mandatory Skills: Strong expertise in Apache Spark, Delta Lake, Databricks SQL, Python, SQL and GCP Platform services., Architect and implement scalable Lakehouse data platforms using Databricks and Delta Lake. Design robust batch and streaming data pipelines leveraging Apache Spark, structured streaming, and modern ELT patterns. Lead migration of Jobs from other cloud data platform to Databricks. Implement secure data governance, access control, and lineage using Unity Catalog. Architect integrations with cloud platforms such as Google Cloud Platform. Optimize performance and manage compute costs through efficient cluster configuration and Spark workload tuning. Collaborate with data engineers, analytics teams, and ML engineers to enable scalable data products and analytics. Define best practices for data quality, reliability, and observability across the data platform. Provide technical leadership, architecture guidance, and mentorship to data engineering teams.

Requirements

Big Data, Apache Spark, Python Programming/Scripting Language, SQL (Structured Query Language), GCP (Good Clinical Practices), Data Management, Cloud Computing, Access Control, Information/Data Security (InfoSec), Cisco Unity, 10+ years of experience in data engineering or data architecture in Big Data platforms. 3+ years hands-on experience with Databricks platform architecture Strong expertise in Apache Spark,Delta Lake,Databricks SQL,Python, SQL and GCP Platform services.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerbuilder.com
Prepare application

Good distractions

Loading talks and stories from around this role…