AWS Data Engineer

Lightning Minds Inc.
Cary, United States of America
3 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Cary, United States of America

Tech stack

Agile Methodologies
Airflow
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Big Data
Cloud Engineering
Code Review
Computer Programming
Continuous Integration
Data Architecture
Information Engineering
ETL
Data Warehousing
DevOps
Distributed Computing Environment
Python
Performance Tuning
Cloud Services
Workflow Management Systems
Data Processing
Cloud Platform System
Snowflake
Spark
AWS Lambda
GIT
Cloudformation
Data Lake
PySpark
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Kafka
Video Streaming
Terraform
Data Pipelines
Amazon Web Services (AWS)
Redshift
Databricks

Job description

LPL Financial is seeking a Senior AWS Data Engineer to design, develop, and optimize scalable cloud-based data platforms and ETL pipelines using AWS services. The ideal candidate should have extensive experience in big data technologies, cloud engineering, and modern data architectures., * Design and develop cloud-native data pipelines on AWS.

  • Build and maintain ETL/ELT workflows.
  • Optimize data processing performance and scalability.
  • Integrate data from multiple enterprise sources.
  • Develop reusable data engineering frameworks.
  • Ensure data quality, security, and governance.
  • Collaborate with Architects, Data Scientists, and Business teams.
  • Troubleshoot production issues and optimize existing pipelines.
  • Participate in code reviews and mentor junior engineers.

Requirements

  • 8+ years of Data Engineering experience.

  • 4+ years of hands-on AWS experience.

  • Strong experience with:

  • AWS Glue

  • Amazon S3

  • Amazon Redshift

  • AWS Lambda

  • Amazon EMR

  • Amazon Athena

Strong Python and PySpark programming skills.

Advanced SQL development experience.

Experience building scalable ETL/ELT pipelines.

Experience with Spark and distributed data processing.

Strong knowledge of Data Lakes and Data Warehousing.

Experience with Apache Airflow or other workflow orchestration tools.

Experience with Git, CI/CD, and DevOps practices.

Understanding of data modelling and performance optimization.

Experience working in Agile environments.

Preferred Skills

  • Financial Services or Banking domain experience.
  • Knowledge of Snowflake.
  • Experience with Kafka or streaming technologies.
  • Terraform or CloudFormation experience.
  • Databricks experience is a plus.

Apply for this position