Data Engineer - PySpark & Python

CIS Technologies Inc.

Charlotte, NC, United States

about 1 month ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Clean Code Principles Artificial Intelligence Amazon Web Services Amazon S3 Code Review Data Validation Information Engineering Extract Transform Load (ETL) Data Transformation Data Mining Database Queries Software Design Patterns

+9 more

Python (Programming Language) Object-Oriented Software Development Performance Tuning Standard Sql Freeform SQL Backend Pyspark Apache Kafka Data Pipelines

Job description

Design, develop, and optimise large-scale data pipelines using PySpark and Python.
Implement and adhere to best practices in object-oriented programming to build reusable, maintainable code.
Write advanced SQL queries for data extraction, transformation, and loading (ETL).
Collaborate closely with data scientists, analysts, and stakeholders to gather requirements and translate them into technical solutions.
Troubleshoot data-related issues and resolve them in a timely and accurate manner.
Leverage AWS cloud services (e.g., S3, EMR, Lambda, Glue) to build and manage cloud-native data workflows (preferred).
Participate in code reviews, data quality checks, and performance tuning of data jobs.

Requirements

Relevant experience in a data engineering or backend development role.
Experience working with AWS cloud ecosystem (S3, Glue, EMR, Redshift, Lambda, etc.). Kafka
Strong hands-on experience with PySpark and Python, especially in designing and implementing scalable data transformations.
Solid understanding of Object-Oriented Programming (OOP) principles and design patterns.
Proficient in SQL, with the ability to write complex queries and optimise performance.
Strong problem-solving skills and the ability to troubleshoot complex data issues independently.
Excellent communication and collaboration skills.
Hands-on experience with AI Tools.