AWS Data Engineer

Purple Drive Technologies LLC

Charlotte, NC, United States

about 2 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

8 years minimum

Working hours

Regular working hours

Job source

dice.com

Tech stack

Amazon Web Services Amazon S3 Data Analysis Cloud Database Cloud Engineering Data Governance Data Integrity Extract Transform Load (ETL) Data Transformation Data Security DevOps Fraud Prevention and Detection

+22 more

Python (Programming Language) Enterprise Messaging Systems Meta-Data Management Performance Tuning E2e Testing Cloud Services Data Streaming Amazon Connect Parquet AWS Lambda Servicebus Event Driven Architecture Pyspark Infrastructure Automation Frameworks Deployment Automation AWS Glue Data Analytics AWS Data Analytics Apache Kafka Terraform Stream Processing Data Pipelines

Job description

AWS Lambda
Amazon S3
Amazon EventBridge
Amazon Kinesis (Data Streams & Firehose)
AWS Glue (ETL & Crawlers)
AWS Step Functions
Amazon Athena
Amazon Connect
Amazon Macie
Experience designing and implementing AWS Data Lake architectures, * Design and implement scalable AWS-based data engineering solutions
Develop real-time event-driven architectures using EventBridge, Kafka, and Kinesis
Build and optimize ETL pipelines using AWS Glue, Python, and PySpark
Configure and manage AWS Glue Crawlers for automated schema discovery and metadata management
Design and maintain data lakes using Amazon S3, Parquet, Glue Catalog, and Athena
Develop integrations between Customer Profiles, messaging platforms, and downstream systems
Build automated workflows using AWS Step Functions
Provision and manage AWS infrastructure using Terraform
Implement CI/CD pipelines for automated deployment and infrastructure management
Perform end-to-end testing, monitoring, troubleshooting, and performance tuning
Optimize cloud architecture for scalability, reliability, cost efficiency, and security
Maintain technical documentation, architecture diagrams, and operational runbooks

Requirements

Strong proficiency in:
Python
PySpark
Experience developing:
ETL/ELT Pipelines
Data Transformation Workflows
Real-Time Data Processing
Hands-on experience with:
Parquet File Format
Glue Crawlers
AWS Glue Data Catalog

Streaming & Event-Driven Architecture

Experience with:
Event-Driven Architecture
Publish/Subscribe (Pub/Sub) Models
Apache Kafka
Amazon EventBridge
Kinesis Data Streams
Kinesis Firehose

DevOps & Infrastructure

Hands-on experience with:
Terraform (Infrastructure as Code)
CI/CD Pipelines
Cloud Automation
Experience deploying and managing AWS infrastructure

Data Analytics & Security

Experience using:
Amazon Athena
AWS Glue Catalog
Amazon Macie
Strong understanding of:
Data Governance
Data Security
Data Integrity
Performance Optimization, * Experience in fraud detection or financial services platforms
AWS Certifications (Solutions Architect, Data Engineer, or Developer Associate)
Experience with:
Lakehouse Architectures
Streaming Analytics
Cloud-Native Data Platforms
Knowledge of DevOps best practices and Infrastructure Automation