AWS Data Engineer

Blue Space Technologies

Malvern, PA, United States

5 days ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

8 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Airflow Amazon Web Services Amazon S3 Computer Programming Continuous Integration Data as a Services Data Architecture Information Engineering Extract Transform Load (ETL) Data Transformation Data Migration Data Warehousing

+16 more

Python (Programming Language) SQL Databases Data Storage Management Sql Optimization Apache Spark Git Data Lakes Pyspark AWS Data Analytics Apache Kafka Video Streaming Cloudwatch Terraform Stream Processing Data Pipelines Databricks

Job description

Design, develop, and maintain scalable data pipelines on AWS.
Build and optimize data engineering solutions using Databricks and PySpark.
Develop ETL/ELT pipelines for batch and near-real-time data processing.
Work with AWS services such as S3, Glue, Lambda, Redshift, EMR, and CloudWatch.
Develop data transformations using Python, SQL, and PySpark.
Work with Databricks Delta Lake and implement efficient data storage and processing solutions.
Optimize Spark jobs, Databricks notebooks, and data pipelines for performance and scalability.
Implement data quality, validation, monitoring, and error-handling processes.
Collaborate with data architects, analysts, application teams, and business stakeholders.
Participate in data migration, modernization, and cloud transformation initiatives.
Follow enterprise security, governance, and development standards.

Requirements

We are looking for an experienced AWS Data Engineer with strong hands-on experience in Databricks, PySpark, and AWS data services to support a large-scale enterprise data engineering project., * 8+ years of experience in Data Engineering.

Strong hands-on experience with AWS data services.
Strong Databricks experience in an enterprise environment.
Strong PySpark / Apache Spark experience.
Strong programming experience with Python.
Advanced SQL skills.
Experience building and supporting ETL/ELT data pipelines.
Strong experience with AWS S3, Glue, Lambda, Redshift, and/or EMR.
Hands-on experience with Delta Lake and Databricks data engineering.
Experience with data warehousing and dimensional data modeling.
Experience with Git and CI/CD processes., * Experience with Databricks Workflows, Unity Catalog, and Delta Live Tables (DLT).
Experience with AWS Lakehouse architecture.
Experience with Terraform or infrastructure-as-code.
Experience with Airflow or other workflow orchestration tools.
Experience with streaming technologies such as Kafka/Kinesis.
Financial services or banking domain experience is a plus.
Experience working in large enterprise environments.