Azure Databricks Architect

We are looking for a Senior Databricks Solution Architect to drive the architecture and technical delivery of a migration from Palantir Foundry to Databricks. You will review the current Palantir applications, map their capabilities to Databricks services, design the target-state Lakehouse, and guide the move of data engineering, AI/GenAI, analytics, chatbot, and business workflow solutions onto a scalable, secure, high-performance platform. Key Responsibilities Design the target-state Databricks architecture, including a scalable Lakehouse across Bronze, Silver, and Gold layers. Evaluate existing Palantir Foundry pipelines, datasets, applications, workflows, and AI capabilities; build feature-by-feature mappings to Databricks and flag components that need redesign. Lead the migration of data ingestion, transformation, data quality, validation, large-scale data comparison, and reconciliation capabilities. Build and tune data pipelines using Databricks Lakeflow, Spark Declarative Pipelines, Apache Spark, and Delta Lake for both batch and near-real-time processing. Architect AI/GenAI solutions covering structured and unstructured document extraction, large language models (LLMs), retrieval-augmented generation (RAG), embeddings, vector search, model serving, and the migration of Palantir AIP capabilities. Design target-state solutions for dashboards, drill-down reporting, analytical applications, chatbots, conversational analytics, and business workflows. Put enterprise data governance, security, access controls, lineage, and auditing in place using Unity Catalog and suitable role-based access control (RBAC) policies. Tune Databricks workloads, data pipelines, and compute resources for scalability, performance, reliability, and cost efficiency. Lead data engineering and AI/ML teams, run architecture reviews, produce high-level design (HLD) and low-level design (LLD) documents and migration roadmaps, and work with business stakeholders, Palantir subject matter experts (SMEs), cloud architects, and application teams. Requirements

Deep experience in Databricks solution architecture, Lakehouse design, Delta Lake, Apache Spark, and enterprise data engineering. Strong command of Bronze/Silver/Gold architecture, data ingestion, transformation, data quality, change data capture (CDC), and batch and near-real-time processing. Demonstrated experience with Palantir Foundry and/or Palantir AIP, ideally on platform migrations, capability mapping, and application or workflow redesign. Solid knowledge of AI/GenAI solutions, including LLMs, RAG, document intelligence, information extraction, embeddings, vector search, and model serving. Hands-on experience with Unity Catalog, enterprise data governance, security, access controls, and cloud platforms such as AWS, Azure, or GCP. Experience architecting analytics, dashboards, chatbots, application integrations, and business workflows in enterprise environments. Proven technical leadership, architecture documentation, performance optimization, and stakeholder management, with the ability to lead complex migration programs. Bachelor’s degree in Computer Science, Engineering, or a related field.

Founded in 2014, is a global leader in delivering innovative IT solutions and services. Headquartered in the USA with a presence in the UK, Canada, and India, we specialize in offering industry-leading expertise and cutting-edge products that help our clients maximize their technological investments. Our focus on best-in-class solutions, a highly knowledgeable team, and proactive talent mapping ensure we remain at the forefront of the IT industry. ConglomerateIT is driven by our Center for Excellence and Innovation, an initiative dedicated to keeping us ahead in a rapidly evolving technology landscape. We understand that building strong relationships is key to our success, and this commitment has enabled us to partner with Fortune 500 companies and leading system integrators worldwide. Our ability to provide local talent on a global scale ensures that we can meet the contingent project requirements of our clients efficiently and effectively.

ConglomerateIT LLC