Data Engineer

ETeam Inc

Glasgow, UK

30 days ago

Apply on www.careerboard.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Compensation

£104,520.0

Working hours

Regular working hours

Job source

www.careerboard.com

Tech stack

Unity 3d Artificial Intelligence Airflow Amazon Web Services Amazon S3 Data Analysis Unit Testing Cloud Computing Code Review Continuous Integration Information Engineering Extract Transform Load (ETL)

+20 more

Data Warehousing DevOps Distributed Computing Environment Identity and Access Management Python (Programming Language) SQL Databases Data Processing Cloud Platform System Apache Spark Amazon Virtual Private Cloud (VPC) Gitlab Cloudformation Data Lakes Pyspark Real Time Data Apache Kafka Machine Learning Operations Video Streaming Data Pipelines Databricks

Job description

To design, build, and maintain scalable data pipelines, data lakes, and data warehouse solutions on AWS. The role focuses on developing high-performance data engineering solutions using PySpark, Spark, Python, and AWS services, enabling secure, reliable, and efficient data processing and analytics across enterprise platforms., * Design, develop, and maintain scalable batch and Real Time data pipelines using PySpark, Spark, Python, and AWS services.

Build and optimize data lakes and data warehouse solutions ensuring data quality, security, and accessibility.
Develop reusable, production-grade ETL/ELT frameworks and data processing solutions.
Implement orchestration workflows using AWS Step Functions, Airflow, and other automation tools.
Develop and maintain cloud infrastructure using AWS CloudFormation.
Collaborate with business stakeholders to understand requirements and translate them into scalable technical solutions.
Optimize data processing performance, monitoring, and operational support.
Implement unit testing, code reviews, and CI/CD best practices using GitLab.
Support platform modernization and migration initiatives leveraging Spark-based architectures.
Work closely with Data Scientists and Analytics teams to enable AI/ML use cases.

Requirements

Strong hands-on experience in Data Engineering with delivery of production-grade solutions.
Expertise in PySpark, Apache Spark, Python, and SQL.
Strong experience designing and optimizing complex data pipelines and ETL/ELT frameworks.
Hands-on experience with AWS services including:
S3
Glue
Lambda
Step Functions
Athena
ECS
IAM
KMS
VPC
SageMaker (preferred)
Experience with AWS CloudFormation for Infrastructure as Code.
Strong understanding of data lakes, data warehouses, and distributed data processing.
Experience with GitLab, CI/CD, Unit Testing, and DevOps practices.
Excellent problem-solving skills and ability to work independently.
Strong stakeholder management and communication skills.

Nice to Have

Experience with Databricks, Delta Lake, Unity Catalog, and migration projects.
Knowledge of AI/ML and MLOps frameworks.
Experience with streaming technologies such as Kafka or Kinesis.

Ideal Candidate: A hands-on Data Engineer with strong expertise in Spark, PySpark, AWS, and CloudFormation, capable of building scalable enterprise data solutions while driving modernization and cloud transformation initiatives