Lead Data Engineer

• Design, build and optimise robust data ingestion pipelines to acquire, transform and load data vendor platforms into client’s data ecosystem.
• Develop scalable data engineering solutions using Databricks, PySpark, SparkSQL and associated cloud technologies.
• Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods.
• Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases.
• Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions.
• Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps.
• Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases.
• Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support.
• Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks.
• Produce clear technical documentation for data ingestion processes, data flows and operational support requirements.
• Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines.
• Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities.
• Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements.
• Support knowledge sharing and contribute to Engineering and Data Communities of Practice.

Applicants must be legally authorized to work in the United Kingdom without the need for current or future visa sponsorship, * Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions.

• Strong hands-on experience with Databricks, PySpark and SparkSQL.
• Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms.
• Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems.
• Strong understanding of data modelling, transformation techniques and data warehousing principles.
• Experience working with cloud-based data lake and analytics platforms.
• Strong understanding of batch and near real-time data processing patterns.
• Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements.
• Experience implementing data quality checks, reconciliations and monitoring processes.
• Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support.
• Understanding of data governance, security, data lineage and documentation standards.
• Experience producing technical documentation and operational handover materials.
• Strong stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams.
• Experience working within Agile delivery environments.
• Knowledge of source control, CI/CD practices and release management processes.
• Ability to work independently while collaborating effectively within cross-functional squads.

Desirable

• Experience integrating data from retail technology platforms, IoT devices or third-party vendor systems.
• Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (eSEL) technologies.
• Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services.
• Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets.
• Experience working within large-scale retail or data transformation programmes., PySpark, Azure Data Factory, Agile, CI/CD

UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact-touching billions of lives in the process.