DataBricks Data engineer

Cognizant Technology Solutions Corporation
Atlanta, United States of America
4 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English

Job location

Atlanta, United States of America

Tech stack

Amazon Web Services (AWS)
Amazon Web Services (AWS)
Azure
Data as a Services
Data Architecture
Information Engineering
Data Governance
ETL
Data Systems
Distributed Computing Environment
Meta-Data Management
Performance Tuning
Cloud Services
Software Deployment
Data Streaming
Data Processing
Cloud Platform System
Data Lake
PySpark
Data Management
Cloud Migration
Databricks

Job description

We are seeking a highly skilled Lead Data Engineer with deep expertise in Databricks SQL, PySpark, and modern cloud data platforms. The ideal candidate will have hands-on experience delivering cloud modernization and migration initiatives, along with the ability to design and optimize scalable data architectures that support analytics, data engineering, and enterprise-scale processing.

This role involves handling the Databricks to Pyspark Migration design, guiding engineering teams, and ensuring data solutions meet performance, security, and governance standards., Design and implement scalable, high-performance data migrations using Databricks Lakehouse, Delta Lake, and distributed processing frameworks.

Develop, optimize, and productionize Databricks SQL and PySpark pipelines supporting large-scale ETL, streaming, and advanced analytics workloads.

Lead end-to-end architecture for cloud modernization and migration projects, ensuring best practices in scalability, performance, governance, and cost efficiency.

Collaborate with business partners, product owners, and engineering teams to convert business requirements into robust technical solutions.

Provide technical leadership and mentorship to data engineers and developers, ensuring adherence to coding, design, and architectural standards.

Review and improve solution designs, ensuring alignment with enterprise governance, security, and compliance frameworks.

Support production deployments, troubleshooting, and performance tuning of data workflows running on Databricks and cloud-native platforms.

Drive continuous improvement and adoption of emerging technologies, tools, and best practices within the data engineering ecosystem.

Requirements

Strong hands-on expertise with Databricks SQL

Advanced development experience with PySpark for ETL and data processing

good understanding of cloud-native data platforms (AWS, Azure, or GCP)

strong foundation in data modeling, performance optimization, and governance

Preferred Skills:

Experience with AWS data services (Glue, S3, Lambda, Lakehouse technologies)

Exposure to Iceberg/Delta Lake, data quality frameworks, and data cataloging

Strong communication skills and ability to work with global teams

Apply for this position