Lead AWS Glue Developer

Insight Global
Chesterfield, United States of America
2 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Chesterfield, United States of America

Tech stack

API
Artificial Intelligence
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Automation of Tests
Cloud Computing
Cloud Engineering
Code Review
Information Engineering
Data Governance
Data Integration
ETL
Software Debugging
DevOps
File Systems
Amazon DynamoDB
Github
JSON
Python
Meta-Data Management
Metadata Repositories
Performance Tuning
Regression Testing
Amazon Web Services (AWS)
Software Engineering
Systems Integration
S3 Bucket
Data Processing
File Transfer Protocol (FTP)
GitHub Copilot
Spark
Software Troubleshooting
AWS Lambda
Infrastructure as Code (IaC)
Cloudformation
Event Driven Architecture
PySpark
Integration Tests
Infrastructure Automation Frameworks
Deployment Automation
Amazon Web Services (AWS)
Maintaining Code
Integration Frameworks
Amazon Web Services (AWS)
Functional Programming
Cloudwatch
Amazon Web Services (AWS)
Software Version Control
Data Pipelines
Serverless Computing

Job description

We are seeking a Lead AWS Glue Developer to design, build, and maintain scalable cloud-native data integration and ETL solutions within AWS. This role will lead the development of serverless data pipelines, automate cloud infrastructure deployment, and build robust data processing frameworks leveraging AWS Glue, AWS Lambda, DynamoDB, S3, and related AWS services. The ideal candidate will have deep expertise in AWS data engineering, Infrastructure as Code (IaC), Python development, and modern DevOps practices. Experience with AWS Glue, Lambda, CloudFormation, S3, and DynamoDB aligns closely with capabilities referenced in internal AWS engineering profiles.

Responsibilities

AWS Data Engineering & ETL Development

Design, develop, and optimize AWS Glue ETL jobs using Python and Apache Spark.

Build scalable batch and event-driven data processing solutions.

Develop and maintain data pipelines that ingest, transform, validate, and load data into AWS data stores.

Create and manage AWS Glue Data Catalogs, crawlers, and metadata management processes.

Implement data quality controls, monitoring, and exception handling throughout ETL workflows.

Serverless Application Development

Develop AWS Lambda functions to automate file processing and event-driven workflows.

Build solutions that process files dropped into S3 buckets and orchestrate movement of data into DynamoDB tables and downstream systems.

Implement integrations between S3, Lambda, DynamoDB, and other AWS services.

Troubleshoot, optimize, and enhance serverless applications for performance and reliability.

Cloud Infrastructure & DevOps

Create and maintain AWS CloudFormation templates to automate infrastructure provisioning and deployment.

Configure and manage AWS CodeCommit repositories and CI/CD pipelines.

Establish and maintain AWS environments following Infrastructure-as-Code best practices.

Set up and administer S3 buckets, SFTP services, access controls, encryption, and lifecycle management policies.

Collaborate with DevOps teams to automate deployments and improve platform reliability.

Technical Leadership

Provide technical leadership and architectural guidance to development teams.

Lead code reviews and establish development standards and best practices.

Mentor junior developers and data engineers.

Collaborate with architects, business stakeholders, and product teams to define technical solutions.

Drive performance tuning, scalability improvements, and cloud modernization initiatives.

Testing & Quality Engineering

Write comprehensive unit, integration, and regression tests for AWS Lambda functions and AWS Glue jobs.

Implement automated testing strategies within CI/CD pipelines.

Ensure high code quality, security compliance, and operational excellence.

Requirements

8+ years of software engineering or data engineering experience.

5+ years of hands-on AWS development experience.

Strong expertise with

AWS Glue

AWS Lambda

Amazon S3

Amazon DynamoDB

AWS CloudFormation

AWS CodeCommit

Python

Apache Spark / PySpark

Experience building serverless and event-driven architectures.

Strong understanding of JSON data structures and APIs.

Experience developing and deploying Infrastructure-as-Code solutions.

Experience building and supporting ETL and data integration pipelines.

Strong troubleshooting, debugging, and performance optimization skills.

Experience writing automated unit and integration tests.

Preferred Qualifications

AWS Certified Data Engineer, Solutions Architect, or Developer certification.

Experience with additional AWS services such as Step Functions, EventBridge, SNS, SQS, CloudWatch, or Redshift.

Experience working in Agile/Scrum environments.

Familiarity with GitHub and modern source control practices.

Knowledge of data governance, security, and compliance requirements.

Experience leading development teams and technical projects.

AI & Productivity Expectations

Comfortable using GitHub Copilot to accelerate software development and improve developer productivity.

Ability to leverage AI-assisted coding tools while maintaining code quality, security, and performance standards.

Interest in applying emerging AI-enabled development practices to cloud engineering and data engineering workflows.

Apply for this position