> Markdown version of [/jobs/ext/3859038-lead-data-consultant-aws-databricks](https://www.wearedevelopers.com/jobs/ext/3859038-lead-data-consultant-aws-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Consultant - AWS / Databricks - **Company:** Scale Factory - **Location:** London, UK - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Amazon Web Services, Cloud Computing, Continuous Integration, Data Architecture, Data Validation, Information Engineering, Data Governance, Data Infrastructure, Data Transformation, Data Security, Dimensional Modeling, Python (Programming Language), NoSQL, Performance Tuning, Data Mesh, SQL Databases, Feature Store, Retrieval-Augmented Generation, Apache Spark, SC Clearance, Event Driven Architecture, Integration Tests, Deployment Automation, AWS Data Analytics, Apache Kafka, Spark Streaming, Data Management, Machine Learning Operations, Code Restructuring, Data Pipelines, Databricks - **Published:** October 22, 2026 - **Apply:** https://www.totaljobs.com/job/lead-consultant/scale-factory-job108211156 ## About the Role * Modern Data Platform Delivery: Proven experience building and deploying scalable data platforms end-to-end on AWS and/or Databricks. * Production Coding: Strong, hands-on production experience with Python, SQL, Spark, and Databricks or AWS data tools. * Consulting / Client Delivery: Experience working in a consultancy, systems integrator, or client-facing setting with direct accountability to project delivery outcomes. * Data Modeling: Solid depth in dimensional modeling, NoSQL schemas, and lakehouse/medallion architecture patterns. * Data Governance & Operations: Practical experience implementing data catalogues, lineage, access controls, data quality checks, and monitoring. * Cost Optimization: Experience implementing cost controls, platform monitoring, and optimizing data compute/storage expenses. * Clearance: SC clearance, or eligibility for it. Desirable * Certifications: Databricks and/or AWS certifications at Associate or Professional level. * Streaming Architectures: Practical experience with real-time or event-driven streaming (Kafka/MSK, Kinesis, Spark Structured Streaming). * Advanced Data Architectures: Exposure to Data Mesh concepts, data products, or federated data governance models. * AI/ML Infrastructure: Exposure to AI/ML data foundations, including feature stores, MLflow, model serving, or retrieval pipelines (RAG). * Regulated Sectors: Experience working within regulated environments (Financial Services, Healthcare, Public Sector). ## Description We see data as the foundations for AI and are growing our data practice to help businesses embrace the future and make their data fit for the AI age. We are seeking an AI literate (and willing to learn / gain certifications in this important new area) Data Leader. You will act as a senior technical lead and hands-on data architect on client engagements. You'll work directly with client engineering teams, data leads, and technical stakeholders to design, build, and deploy production-grade data platforms and AI applications. Rooted in the codebase, you will lead delivery teams by example, ensuring high engineering standards, robust software practices, and architectural alignment. Off-project, you will contribute directly to shaping our technical practice by building reference implementations, accelerators, and reusable delivery assets. Technical Architecture & Delivery * Design end-to-end target data platforms (ingestion, storage, transformation, serving, and governance) in alignment with overall technical direction. * Make and document key architectural decisions and technical trade-offs (e.g., lakehouse vs. warehouse, AI context stores). * Translate business and functional requirements into scalable, resilient data pipeline architectures and detailed technical roadmaps. * Embed non-functional requirements, data security, privacy, lineage, quality controls, and FinOps guardrails directly into the engineering design. Hands-on Engineering & Technical Leadership * Serve as the Tech Lead on client engagements where required, guiding delivery teams and unblocking technical hurdles. * Write, review, and maintain production-grade code. * Establish and champion practice engineering standards, including unit/integration testing, CI/CD pipelines, automated deployments, environments, and observability. * Mentor, coach, and support mid-level and senior engineers to elevate the overall technical output of the team. Client Engagement & Technical Advisory * Facilitate technical discovery and architecture alignment sessions with client engineering teams and technical leads. * Act as a trusted technical advisor on trade-offs, performance optimization, refactoring, and risk mitigation. * Communicate architectural concepts, decision records, and technical trade-offs clearly to both technical and non-technical stakeholders. Practice Building & Knowledge Sharing * Build reusable delivery assets, reference architectures, code accelerators, and technical playbooks. * Assist with interviewing and technical assessments for prospective data engineering candidates. * Collaborate across cloud infrastructure and testing practice groups to ensure seamless technical cross-pollination., * 6+ years in data architecture and technical delivery, with hands-on experience leading data transformation projects., * Engineering-Focused: Leadswith working code, rigorous standards, and sound engineering practices. * Clear Communicator: Produces clear technical documentation, architecture diagrams, and decision records. * Collaborative Coach: Enjoys mentoring team members and sharing knowledge to raise collective engineering capability. * Adaptable: Comfortable navigating technical ambiguity within fast-paced or evolving environments., * Social Events: From annual kick-offs and Christmas parties to team socials and sporting events, we love celebrating success together. * Training & Development: Tailored learning opportunities and full support for certifications to keep your skills sharp.