Databricks Data Engineer

INSYSTECH, INC.

Richardson, TX, United States

2 days ago

Apply on www.careerjet.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

4 years minimum

Compensation

$85,500.0 - $130,000.0

Working hours

Regular working hours

Job source

www.careerjet.com

Tech stack

Airflow Amazon Web Services Amazon S3 Big Data Code Review Computer Programming Continuous Integration Information Engineering Data Infrastructure Data Integration Extract Transform Load (ETL) Data Transformation

+15 more

Data Systems Data Warehousing Database Queries Distributed Systems Performance Tuning Scala (Programming Language) Data Logging Data Processing Apache Spark Caching Git Data Lakes Pyspark Data Pipelines Databricks

Job description

Design, develop, and maintain scalable enterprise data pipelines using Databricks, Spark, PySpark, Scala, and AWS.
Develop Databricks notebooks and Spark applications for large-scale data transformation and processing.
Build and optimize ETL/ELT pipelines integrating data from multiple enterprise sources.
Develop reusable data-processing frameworks using Scala and PySpark.
Optimize Spark jobs, cluster configurations, partitioning, caching, and data-processing performance.
Implement data-quality checks, monitoring, logging, and exception-handling mechanisms.
Develop and maintain Delta Lake-based data solutions.
Troubleshoot data pipeline failures and production performance issues.
Collaborate with architects, analysts, and engineering teams to translate business requirements into scalable data solutions.
Participate in code reviews, testing, deployments, and production support.

Requirements

8+ years of overall experience in Data Engineering, Big Data, ETL, or Data Platform development.
4+ years of strong hands-on experience with Databricks.
Strong hands-on development experience with Apache Spark and PySpark.
Strong programming experience with Scala.
Hands-on experience building data solutions on AWS.
Strong experience developing scalable ETL/ELT and batch data-processing pipelines.
Experience processing large-scale datasets using distributed computing technologies.
Strong SQL skills for data transformation, validation, and performance optimization.
Experience with Delta Lake / Lakehouse architectures.
Experience troubleshooting and optimizing Spark and Databricks workloads.
Strong understanding of data warehousing, data modeling, and data integration concepts., * AWS S3, Glue, EMR, Lambda, Redshift, Delta Lake, Unity Catalog, Airflow, Git and CI/CD experience is preferred.