Data Engineer - Databricks / Scala

Cyma Systems Inc
Quincy, MA, United States
4 days ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
8 years minimum
Working hours
Regular working hours
Job source

Tech stack

Java (Programming Language) Apache HTTP Server Microsoft Azure Big Data Cloud Engineering Information Engineering Extract Transform Load (ETL) Software Debugging R (Programming Language) Performance Tuning Scala (Programming Language) Data Streaming
+14 more
Data Processing Data Ingestion Sql Optimization Delivery Pipeline Apache Spark Containerization Data Lakes Pyspark Azure AKS Spark Streaming Data Pipelines Serverless Computing Docker Databricks

Job description

Design and develop scalable Databricks data engineering solutions.

Build and optimize batch and streaming data pipelines.

Implement data ingestion, transformation and Lakehouse solutions.

Troubleshoot production Databricks jobs and DevOps pipelines.

Optimize Spark workloads and ensure data processing performance.

Requirements

8+ years of Data Engineering experience.

Strong hands-on expertise in Azure Databricks, Scala and Apache Spark.

Scala / PySpark development and performance tuning.

Advanced SQL, ETL/ELT and data pipeline development.

Spark DataFrames, Datasets and Notebooks.

Large-scale data processing, streaming and transformations.

Delta Lake, Unity Catalog and Auto Loader.

Lakehouse and Medallion Architecture.

Spark Structured Streaming.

Databricks clusters, sizing, serverless and job orchestration.

Azure, DevOps pipelines and AKS troubleshooting.

Production monitoring, debugging and performance optimization.

Nice-to-Have Skills:

Apache Big Data ecosystem experience.

Java or R programming.

Docker and Kubernetes.

Containerization and cloud-native deployments.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…