Data Engineer - AWS & PySpark

Cliff Services Inc

Dallas, TX, United States

12 days ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Amazon Web Services Amazon S3 Big Data Data as a Services Extract Transform Load (ETL) Data Transformation Data Systems Distributed Computing Environment Python (Programming Language) Cloud Services Standard Sql Data Processing

+3 more

Cloud Platform System Pyspark Data Pipelines

Job description

Design, develop, and maintain scalable data pipelines using PySpark and AWS.
Develop and optimize large-scale data processing and ETL workflows.
Work with AWS cloud services to build reliable and high-performance data solutions.
Perform data transformation, cleansing, validation, and integration.
Optimize PySpark jobs for performance, scalability, and cost efficiency.
Collaborate with data architects, analysts, developers, and business stakeholders.
Troubleshoot data pipeline issues and ensure data quality and reliability.
Follow engineering best practices for code development, testing, deployment, and documentation.

Requirements

We are looking for an experienced Data Engineer with strong hands-on expertise in AWS and PySpark. The ideal candidate must have prior professional experience working with Banking domain and be capable of developing scalable data pipelines and data processing solutions in a cloud environment., * Strong hands-on experience with PySpark.

Strong experience with AWS cloud services.
Experience developing ETL/data pipelines and processing large datasets.
Strong Python and SQL skills.
Experience with data transformation, integration, and data quality.
Mandatory: Prior Capital One project/client experience.
Strong communication and problem-solving skills.

Preferred

Experience with AWS data services such as S3, Glue, EMR, Lambda, Redshift, or similar.
Experience with distributed data processing and cloud-based data platforms.
Financial services/banking domain experience.