Lead Data & AI Engineer

EPAM Systems, Inc.
London, UK
11 days ago
Apply on www.indeed.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
8 years minimum
Working hours
Regular working hours
Job source

Tech stack

Artificial Intelligence Microsoft Azure Computer Programming Continuous Integration Data Architecture Information Engineering Data Governance Extract Transform Load (ETL) Distributed Computing Environment Github Python (Programming Language) Machine Learning
+13 more
Performance Tuning Azure Active Directory Data Streaming Azure Data Factory Data Build Tool (dbt) Prompt Engineering Containerization Data Lakes Pyspark Machine Learning Operations Software Version Control Data Pipelines Databricks

Job description

  • Design and implement robust data architectures using cloud-native technologies
  • Build large-scale ETL/ELT workflows to process heterogeneous datasets
  • Create data models optimized for AI/ML pipelines and advanced analytics
  • Develop streaming and batch pipelines leveraging tools like Azure Data Factory and Databricks
  • Operationalize ML solutions, integrating feature stores, model registries and inference endpoints
  • Collaborate with data scientists to deploy and monitor AI models using enterprise MLOps frameworks
  • Integrate GenAI-assisted development methods into data workflows for automation and efficiency
  • Ensure data governance, lineage, cataloging and quality frameworks across platforms
  • Optimize platform performance, manage compute cost efficiency and enable observability for critical workloads
  • Contribute to best practices, mentoring engineering teams in advanced data and AI engineering

Requirements

  • Minimum 8+ years working in data engineering, with proven architecture and leadership experience
  • Advanced proficiency in SQL for performance tuning at large scale
  • Strong programming skills in Python, with applied experience in data engineering workflows
  • Expertise in PySpark for distributed data processing
  • Hands-on experience with Databricks, including Delta Lake and performance optimization
  • Knowledge of Azure Data Factory for orchestration of pipelines (ETL/ELT)
  • Familiarity with AI/ML pipeline development, including integration of models into production
  • Demonstrated exposure to Gen AI-assisted development workflows for accelerating data engineering tasks
  • Strong understanding of CI/CD processes for data and ML pipelines such as GitHub Actions, Azure DevOps
  • Ability to manage enterprise-scale solutions across global environments with compliance in mind

Nice to have

  • Prompt Engineering knowledge and experience building RAG workflows
  • Familiarity with Microsoft Foundry platforms
  • Version control and CI/CD experience with GitHub
  • Understanding of ETL/ELT optimization patterns beyond Azure stack
  • Knowledge of data mesh or lakehouse architectural concepts
  • Hands-on exposure to dbt (data build tool), MKDocs and similar developer productivity tooling

Benefits & conditions

  • EPAM Employee Stock Purchase Plan (ESPP)
  • Protection benefits including life assurance, income protection and critical illness cover
  • Private medical insurance and dental care
  • Employee Assistance Program
  • Competitive group pension plan
  • Cyclescheme, Techscheme and season ticket loans
  • Various perks such as free Wednesday lunch in-office, on-site massages and regular social events
  • Learning and development opportunities including in-house training and coaching, professional certifications, and courses
  • If otherwise eligible, participation in the discretionary annual bonus program
  • If otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program

About the company

We’re looking for a Lead Data & AI Engineer to join our team in the UK in a hybrid working mode. In this role, you will design and deliver advanced data solutions and AI-assisted capabilities on modern cloud platforms. You’ll build scalable data pipelines, optimize data quality and governance and enable ML feature engineering for analytics and intelligent applications. The role requires hands-on technical expertise in data engineering and AI integration while driving reliability, performance and security for enterprise-scale systems.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.indeed.com
Prepare application

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

6:36 min

Funding open source through GitHub Accelerator and Sponsors

Stormy Peters · World Congress 2023

4:32 min

Harnessing Spark with Python using PySpark and Py4J

Ayon Roy · LIVE

5:14 min

Executing Databricks jobs with built-in Airflow operators

Alan Mazankiewicz · LIVE

3:09 min

Balancing data science skillings alongside systems engineering rigor

Nico Schmidt · LIVE

2:40 min

Using GitHub primitives for internal documentation and corporate operations

Kyle Daigle · Coffee With Developers

Videos

See all

Related articles

See all