Databricks Architect

We are looking for an experienced Senior Databricks Architect to lead the architecture and technical execution of a large-scale migration for a leading Retail/Grocery client.

The architect will be responsible for designing the target Databricks Lakehouse architecture, defining migration patterns, guiding the data engineering team, and ensuring successful modernization of existing Talend-based ETL/ELT workloads onto Databricks.

The role requires strong hands-on expertise in Databricks, Apache Spark, SQL, cloud data platforms, data architecture, ETL modernization, and migration strategy, along with a good understanding of Retail/Grocery business processes and data domains.

Key Responsibilities

Architecture & Solution Design

Lead the end-to-end architecture for migration of Talend ETL/ELT workloads to Databricks.

Define the target-state Databricks Lakehouse architecture, including data ingestion, transformation, storage, orchestration, governance, and consumption layers.

Design scalable, secure, highly available, and cost-optimized data solutions.

Define architecture standards, design patterns, coding standards, and best practices for Databricks development.

Establish migration frameworks and reusable patterns for converting Talend jobs into Databricks/Spark-based pipelines.

Evaluate existing Talend jobs and determine appropriate migration strategies, including re-platforming, re-engineering, consolidation, or retirement.

Provide technical leadership for complex data pipelines and integration scenarios.

Talend to Databricks Migration

Analyze existing Talend workflows, jobs, mappings, dependencies, schedules, and data transformations.

Develop migration strategies for batch and incremental data processing workloads.

Translate Talend transformations and business rules into PySpark/SQL/Databricks implementations.

Identify opportunities to simplify and optimize legacy ETL processes during migration.

Define data reconciliation and validation strategies to ensure parity between Talend and Databricks outputs.

Establish migration sequencing based on business criticality, dependencies, complexity, and risk.

Support migration of high-volume and business-critical data pipelines.

Databricks & Data Engineering

Design and implement solutions using Databricks, Apache Spark, Delta Lake, PySpark, and SQL.

Design Medallion Architecture (Bronze, Silver, Gold) and appropriate data processing patterns.

Implement incremental processing, CDC, SCD Type 1/Type 2, data quality, error handling, and audit frameworks.

Optimize Spark workloads, Delta tables, SQL queries, partitioning, clustering, and job execution.

Leverage Delta Live Tables / Lakeflow capabilities, Workflows, Unity Catalog, and other Databricks platform services where appropriate.

Design data pipelines integrating structured and semi-structured data from databases, files, APIs, and enterprise applications.

Establish monitoring, logging, alerting, operational support, and performance-management patterns.

Cloud & Integration

Strong experience with at least one major cloud platform, preferably Azure or AWS.

Design integration between Databricks and cloud storage, databases, messaging platforms, APIs, and enterprise applications.

Experience with technologies such as ADLS/S3, Azure Data Factory, Kafka, REST APIs, relational databases, and cloud-native services is highly desirable.

Define secure connectivity, networking, secrets management, and access-control patterns.

Retail / Grocery Domain

Work closely with business and technology stakeholders to understand Retail/Grocery data requirements.

Experience with retail data domains such as:

o Product / Item

o Store / Location

o Customer / Loyalty

o Sales / Transactions

o Pricing

o Promotions

o Inventory

o Supply Chain

o Vendors / Suppliers

o Orders

o Forecasting

o Merchandising

Understand retail-specific challenges such as high-volume transaction processing, store-level data, product hierarchies, promotions, pricing, inventory movements, and customer analytics.

Translate business requirements into scalable data architecture and technical solutions.

Technical Leadership

Act as the senior technical authority for the Databricks migration program.

Provide technical direction and mentoring to data engineers and developers.

Conduct architecture/design reviews and code reviews.

Troubleshoot complex production and performance issues.

Collaborate with offshore/onshore teams, enterprise architects, infrastructure teams, security teams, and business stakeholders.

Prepare architecture diagrams, technical design documents, migration plans, and implementation standards.

Communicate technical risks, dependencies, and recommendations effectively to client leadership.

Required Technical Skills

Databricks

Strong hands-on experience with Databricks.

Apache Spark / PySpark.

Delta Lake.

Databricks Workflows/Jobs.

Unity Catalog.

Databricks SQL.

Performance tuning and optimization.

Medallion/Lakehouse architecture.

Experience with modern Databricks data engineering capabilities.

Data Engineering

Strong SQL and data modeling skills.

ETL/ELT architecture.

Batch and incremental data processing.

CDC and SCD implementation.

Data quality and reconciliation frameworks.

10+ years overall IT experience, with 5+ years in Databricks / Cloud Data Engineering / Data Architecture