> Markdown version of [/jobs/ext/1641990-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/1641990-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - **Company:** Ust View All Jobs - **Location:** Leeds, UK (Remote available) - **Experience:** Expert - **Contract:** Temporary contract - **Skills:** Artificial Intelligence, Business Analytics Applications, Computer Vision, Cloud Computing, Cloud Database, Software Documentation, Continuous Integration, Data Architecture, Data Validation, Data Discovery, Data Governance, Data Integration, Extract Transform Load (ETL), Data Transformation, Data Profiling, Data Security, Dataspaces, Data Warehousing, Digital Assets, Scrum Methodology, Release Management, Azure Data Lake, Data Streaming, Technical Data Management Systems, Unstructured Data, File Transfer Protocol (FTP), Data Ingestion, Azure Data Factory, Apache Spark, Pyspark, Data Lineage, Api Design, Stream Processing, Software Version Control, Data Pipelines, Databricks - **Published:** July 31, 2026 - **Apply:** https://www.careerjet.co.uk/job/gbae4187eb4ddb57c33c1e8ad38355a818/eaa ## About the Role Applicants must be legally authorized to work in the United Kingdom without the need for current or future visa sponsorship, * Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions. * Strong hands-on experience with Databricks, PySpark and SparkSQL. * Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms. * Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems. * Strong understanding of data modelling, transformation techniques and data warehousing principles. * Experience working with cloud-based data lake and analytics platforms. * Strong understanding of batch and near real-time data processing patterns. * Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements. * Experience implementing data quality checks, reconciliations and monitoring processes. * Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support. * Understanding of data governance, security, data lineage and documentation standards. * Experience producing technical documentation and operational handover materials. * Strong stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams. * Experience working within Agile delivery environments. * Knowledge of source control, CI/CD practices and release management processes. * Ability to work independently while collaborating effectively within cross-functional squads. Desirable * Experience integrating data from retail technology platforms, IoT devices or third-party vendor systems. * Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (eSEL) technologies. * Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services. * Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets. * Experience working within large-scale retail or data transformation programmes., PySpark, Azure Data Factory, Agile, CI/CD ## Description * Design, build and optimise robust data ingestion pipelines to acquire, transform and load data vendor platforms into client's data ecosystem. * Develop scalable data engineering solutions using Databricks, PySpark, SparkSQL and associated cloud technologies. * Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods. * Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases. * Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions. * Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps. * Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases. * Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support. * Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks. * Produce clear technical documentation for data ingestion processes, data flows and operational support requirements. * Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines. * Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities. * Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements. * Support knowledge sharing and contribute to Engineering and Data Communities of Practice. ## Related Videos - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Cutting LLM Costs Without Cutting Quality: How to Beat Proprietary LLMs with Fine-Tuned Open Source](https://www.wearedevelopers.com/videos/100151-cutting-llm-costs-without-cutting-quality-how-to-beat-proprietary-llms-with-fine-tuned-open-source) - [API Design - Getting Started](https://www.wearedevelopers.com/videos/33-api-design-getting-started) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Parquet, Delta, Iceberg & Ducklake - An introduction for developers](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [IT Salaries in UK](https://www.wearedevelopers.com/magazine/288-it-salaries-in-uk) - [Fullstack Developer Salary UK](https://www.wearedevelopers.com/magazine/251-fullstack-developer-salary-uk) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market)