> Markdown version of [/jobs/ext/3234258-data-platform-engineer](https://www.wearedevelopers.com/jobs/ext/3234258-data-platform-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Platform Engineer - **Company:** Octopus Energy Ltd - **Location:** London, UK - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Amazon Web Services, User Authentication, Microsoft Azure, Software as a Service, Cloud Computing, Databases, Continuous Integration, Data Infrastructure, Federated Identity Management, Github, Monitoring of Systems, Routing, Search Technologies, Policy as Code, Data Logging, Okta, Large Language Models, Rate Limiting, Usage Tracking, Infrastructure Automation Frameworks, Low Latency, Api Gateway, Terraform, Databricks - **Published:** September 4, 2026 - **Apply:** https://jobs.lever.co/octoenergy/1278dca1-ea17-4c2a-aa9c-72f6df67857b/apply ## About the Role * Federated Identity Mastery: Proven experience designing and implementing enterprise-grade federated identity, token exchange, permission propagation, SCIM provisioning, and access lifecycle management^. * Platform Foundations: Strong hands-on experience with IaC (Terraform) and production CI/CD pipelines (environment promotion, rollback, release controls)^. * API Gateways: Experience hardening API gateway patterns (authentication, routing, rate limiting, access policy enforcement)^. * Cloud & Data Environments: Technical comfort working across Azure, AWS, or Databricks platform ecosystems^. * Pragmatic Mindset: Highly autonomous engineer comfortable in a lean team; skilled at translating architectural specs into practical code and communicating trade-offs clearly in plain English^ And to really stand out from the crowd * Experience with AI gateways, LLM proxies, or model-routing patterns^. * Exposure to AI observability tools, vector databases (Databricks AI Search), RAG pipelines, or non-deterministic system monitoring^. * Hands-on experience implementing policy-as-code, action logging, or AI kill switches^. * Direct experience with Okta, ZScaler, Entra ID, Unity Catalog, or M365 migration environments^. * Background in regulated or high-governance industries (Financial Services, Energy, Infrastructure Investment, Fintech, or Enterprise SaaS)^. * Familiarity with FinOps practices (token budgeting, usage tracking, cost attribution)^. ## Description As part of our mission to drive smarter energy decisions, we are building a production-grade, governed platform that empowers our teams to deploy AI-enabled tools, agents, and data products safely and at scale., * Build and maintain Infrastructure-as-Code (IaC) using Terraform across development, test, and production environments. * Strengthen CI/CD practices in GitHub Actions, establishing repeatable deployment patterns, environment promotions, automated rollbacks, and release controls. * Support architecture decisions by converting designs into maintainable, well-documented engineering patterns. Identity, Access & Security * Design and implement federated identity architecture across Okta, Entra ID, and ZScaler. * Build identity, token exchange, and permission propagation models that work for both human users and non-human AI agents. * Implement SCIM provisioning/deprovisioning and access models with single-revoke capability and full auditability. AI Gateway & Governance * Harden existing AI gateway and agent orchestration layers (built on MCP patterns) into secure, production-ready capabilities. * Implement API gateway controls: authentication, routing, rate limiting, token budgets, policy-as-code, and action logging. * Build platform-, tool-, and agent-level kill switches, alongside dry-run/safe-testing modes for agent workflows. Observability & FinOps * Implement AI system observability, including prompt logging, output monitoring, quality scoring, and drift detection. * Establish operational monitoring for gateway usage, latency, error rates, and model routing logic. * Enforce token budgets, cost alerts, and usage/cost attribution by team and use case. Enablement & Integration * Deploy vector database and retrieval infrastructure (e.g., Databricks AI Search) to support Document AI and RAG use cases. * Align platform controls with Databricks and Unity Catalog governance patterns. * Provide clear documentation, reusable patterns, and self-service onboarding guidance so delivery squads can deploy tools without bespoke manual intervention. ## Related Videos - [Creating a routing app with Google Maps API from scratch](https://www.wearedevelopers.com/videos/831-creating-a-routing-app-with-google-maps-api-from-scratch) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [A Technical Introduction to Bitcoin's 2nd Layer- The Lightning Network](https://www.wearedevelopers.com/videos/15-a-technical-introduction-to-bitcoin-s-2nd-layer-the-lightning-network) - [Lights, Camera, GitHub Actions!](https://www.wearedevelopers.com/videos/734-lights-camera-github-actions) - [Reducing Cognitive Overload Through Platform Engineering](https://www.wearedevelopers.com/videos/679-reducing-cognitive-overload-through-platform-engineering) ## Related Articles - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Dev Digest 137 - AI'm not sure about this](https://www.wearedevelopers.com/magazine/485-dev-digest-137-ai-m-not-sure-about-this) - [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline)