Databricks Engineer

Engagement Overview Client is building a net-new data foundation for a major insurance client, covering Policy, Claims, and Rating data across a modern medallion lakehouse architecture. This role will play a central part in both the platform-specific build and the client-facing architecture conversations already underway. The client currently runs on a legacy Cloudera/Hadoop environment, and a core objective of this engagement is building net-new - coexisting with, but not creating new dependencies on, that legacy platform. Key Responsibilities

• Serve as the primary hands-on Azure and Databricks technical expert for the engagement
• Own Databricks-specific architecture decisions - workspace design, Unity Catalog governance strategy, and compute/cluster architecture - not just build within an existing design
• Work directly with the client’s Enterprise Data Architect and infrastructure team on both platform-agnostic and platform-specific architecture decisions
• Provide technical leadership and mentorship to the offshore delivery team
• Coordinate delivery execution between client-facing architecture discussions (onshore) and day-to-day build/engineering work (offshore), ensuring nothing gets lost in translation between the two
• Support design and implementation of a medallion lakehouse architecture using modern open table formats
• Help navigate coexistence with the client’s legacy Cloudera environment during the platform transition, without introducing new legacy dependencies
• Support evaluation and adoption of accelerator tooling (AI-assisted development tools, DBT for metric/data product development)
• Participate directly in client architecture review sessions, representing OpenKyber’s technical position with credibility
• Act as a fast escalation point for complex Azure/Databricks technical issues blocking the broader team

• 8+ years overall in data engineering/architecture, with 4-5+ years hands-on specifically with Azure and Databricks
• Prior experience specifically in a Databricks Architect role - designing enterprise-scale Databricks solutions, not solely implementation/engineering work
• Strong, hands-on production experience with Azure data services - ADLS Gen2, Azure Data Factory, Azure Synapse/Fabric
• Strong, hands-on production experience with Databricks - Unity Catalog, Delta Lake, PySpark, Databricks Workflows; familiarity with Managed Iceberg tables a plus
• Proven experience designing and implementing medallion/lakehouse architectures at enterprise scale
• Experience migrating from legacy Hadoop/Cloudera environments to modern cloud-native platforms
• Strong client-facing communication skills. Comfortable presenting and defending architecture recommendations directly to client stakeholders, not just working behind the scenes
• Experience mentoring and technically leading distributed/offshore engineering teams
• Ability to work independently and drive technical decisions with limited day-to-day oversight, * Insurance/P&C domain experience, particularly familiarity with policy administration systems
• Experience with DBT for data product/metric development
• Hands-on experience with Apache Iceberg specifically (not just Delta Lake)
• Knowledge of Claude Code and/or GitHub Copilot is an added benefit
• Prior consulting/professional services delivery experience
• Familiarity with data governance/catalog tooling (Unity Catalog)
• Current Databricks architect-level or professional-level certification (to be confirmed against Databricks’ current certification catalog)