Senior Data Engineer ( Databricks, Kafka, Spark, AWS)
PEN SOFTWARE INC.
Atlanta, United States of America
1 month ago
Role details
Contract type
Temporary contract Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
English Experience level
SeniorJob location
Atlanta, United States of America
Tech stack
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Data Architecture
Information Engineering
ETL
Hive
Identity and Access Management
Operational Databases
SQL Databases
Data Streaming
Sql Optimization
Spark
Data Lake
PySpark
Kafka
Data Pipelines
Databricks
Job description
PEN Software is seeking a Senior Data Engineer with strong hands-on experience in Databricks-based lakehouse platforms. The role involves building and supporting scalable batch and real-time data pipelines using Spark, Kafka, and AWS.
This is a hands-on engineering role focused on production-grade data pipelines and modern cloud data architecture.
Requirements
Do you have experience in Spark?, Minimum 5+ years in Data Engineering, * Strong experience in Databricks (Jobs, Workflows, Notebooks)
- Strong Apache Spark (PySpark / Spark SQL)
- Delta Lake and lakehouse architecture
- Delta Live Tables (DLT) pipelines
- Kafka streaming data experience
- Advanced SQL
- AWS (S3, IAM, etc.)
- ETL/ELT pipeline development
- Production data engineering experience
Experience:
- Data Engineering: 6 years (Required)
- Spark: 4 years (Required)
- Kafka: 3 years (Required)
- AWS: 3 years (Required)
- Data lake: 3 years (Required)
- SQL: 4 years (Required)