Lead Technical Engineer: Data Lakes AI LLMs Python Cloud Data Governance & Security

Lead Software Engineer needed to provide support and input to the client's management overseeing the Product Analytics and Enterprise Data Lake Teams. The role involves leading the design, development, and governance of the client's enterprise-scale data infrastructure, including Data Lakes, Pipelines, and Data Products architecture, and providing strategic and technical leadership across multiple engineering teams, ensuring scalable, secure, and high-performance data solutions.

You will need a deep understanding of modern data platforms, strong architectural vision and the ability to collaborate across business and technical domains to drive data-driven innovation.

This one-year contract sits inside IR35 and will require one day per week at the client's offices in SW London which are close to an easily accessible underground and overland station., * Data Architecture & Design: Data Lakes (eg, AWS S3, Azure Data Lake, Google Cloud Storage), Data Mesh principles, domain-oriented data ownership and federated. governance, data modelling (OLAP/OLTP, dimensional modelling, schema evolution).

• Data Engineering & Pipelines: ETL pipelines (using tools like AWS Glue, Apache Spark), Map-Reduce, streaming data platforms (eg, Kafka, SQS), Real Time and batch processing paradigms.
• Cloud & Infrastructure: cloud-native data services (AWS Glue, Azure Synapse, GCP BigQuery, Databricks), Infrastructure-as-Code (IaC) (using Terraform, CloudFormation, Lakeformation).
• Programming & Scripting: Python and SQL, C#, CI/CD pipelines, and DevOps practices for data workflows.
• Data Governance & Security: Data cataloging and lineage tools (eg, Collibra, Apache Atlas, OpenMetaData), data privacy, encryption, access control (eg, IAM, RBAC, ABAC), and compliance frameworks (GDPR).
• Observability & Reliability: Monitoring and alerting for data systems, data quality frameworks (eg, Great Expectations, Monte Carlo), designing for resilience, fault tolerance, and disaster recovery.

• Working as the Technical Lead across multiple onshore and offshore teams building data platforms, customer facing data products and/or machine learning systems.
• Working with LLMs in data engineering and using AI as an accelerator
• Large enterprise data lake projects, preferably with Python.
• Product analytics tools (Mixpanel, Power BI, Athena).
• Expertise in the full SDLC.
• Evaluating and performing competitive analysis of 3rd party software & services.