> Markdown version of [/jobs/ext/3284244-lead-data-ai-engineer](https://www.wearedevelopers.com/jobs/ext/3284244-lead-data-ai-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data & AI Engineer

- **Company:** EPAM Systems, Inc.
- **Location:** London, UK
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Microsoft Azure, Computer Programming, Continuous Integration, Data Architecture, Information Engineering, Data Governance, Extract Transform Load (ETL), Distributed Computing Environment, Github, Python (Programming Language), Machine Learning, Performance Tuning, Azure Active Directory, Data Streaming, Azure Data Factory, Data Build Tool (dbt), Prompt Engineering, Containerization, Data Lakes, Pyspark, Machine Learning Operations, Software Version Control, Data Pipelines, Databricks
- **Published:** September 22, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=6226ec29de6fa2fb

## About the Role

* Minimum 8+ years working in data engineering, with proven architecture and leadership experience
 * Advanced proficiency in SQL for performance tuning at large scale
 * Strong programming skills in Python, with applied experience in data engineering workflows
 * Expertise in PySpark for distributed data processing
 * Hands-on experience with Databricks, including Delta Lake and performance optimization
 * Knowledge of Azure Data Factory for orchestration of pipelines (ETL/ELT)
 * Familiarity with AI/ML pipeline development, including integration of models into production
 * Demonstrated exposure to Gen AI-assisted development workflows for accelerating data engineering tasks
 * Strong understanding of CI/CD processes for data and ML pipelines such as GitHub Actions, Azure DevOps
 * Ability to manage enterprise-scale solutions across global environments with compliance in mind

Nice to have

 * Prompt Engineering knowledge and experience building RAG workflows
 * Familiarity with Microsoft Foundry platforms
 * Version control and CI/CD experience with GitHub
 * Understanding of ETL/ELT optimization patterns beyond Azure stack
 * Knowledge of data mesh or lakehouse architectural concepts
 * Hands-on exposure to dbt (data build tool), MKDocs and similar developer productivity tooling

## Description

* Design and implement robust data architectures using cloud-native technologies
 * Build large-scale ETL/ELT workflows to process heterogeneous datasets
 * Create data models optimized for AI/ML pipelines and advanced analytics
 * Develop streaming and batch pipelines leveraging tools like Azure Data Factory and Databricks
 * Operationalize ML solutions, integrating feature stores, model registries and inference endpoints
 * Collaborate with data scientists to deploy and monitor AI models using enterprise MLOps frameworks
 * Integrate GenAI-assisted development methods into data workflows for automation and efficiency
 * Ensure data governance, lineage, cataloging and quality frameworks across platforms
 * Optimize platform performance, manage compute cost efficiency and enable observability for critical workloads
 * Contribute to best practices, mentoring engineering teams in advanced data and AI engineering


## Related Videos

- [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next)
- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Fully Orchestrating Databricks from Airflow](https://www.wearedevelopers.com/videos/336-fully-orchestrating-databricks-from-airflow)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)
- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)