Grocery Domain -Senior Databricks Architect

Lead the end-to-end architecture for migrating Talend ETL/ELT workloads to Databricks Define target-state Databricks Lakehouse architecture, including data ingestion, transformation, storage, orchestration, governance, and consumption layers Design scalable, secure, highly available, and cost-optimized data solutions Define architecture standards, design patterns, coding standards, and best practices for Databricks development Establish reusable migration frameworks and patterns for converting Talend jobs into Databricks/Spark-based pipelines Evaluate existing Talend jobs and determine appropriate migration strategies, including re-platforming, re-engineering, consolidation, or retirement Analyze Talend workflows, jobs, mappings, dependencies, schedules, and data transformations Develop migration strategies for batch and incremental data processing workloads Translate Talend transformations and business rules into PySpark, SQL, and Databricks implementations Identify opportunities to simplify and optimize legacy ETL processes during migration Define data reconciliation and validation strategies to ensure parity between Talend and Databricks outputs Establish migration sequencing based on business criticality, dependencies, complexity, and risk Support migration of high-volume and business-critical data pipelines Design and implement solutions using Databricks, Apache Spark, Delta Lake, PySpark, and SQL Design Medallion Architecture using Bronze, Silver, and Gold layers Implement incremental processing, CDC, SCD Type 1/Type 2, data quality, error handling, and audit frameworks Optimize Spark workloads, Delta tables, SQL queries, partitioning, clustering, and job execution Leverage Databricks Workflows/Jobs, Unity Catalog, and modern Databricks data engineering capabilities Design data pipelines integrating structured and semi-structured data from databases, files, APIs, and enterprise applications Establish monitoring, logging, alerting, operational support, and performance management practices Design integrations between Databricks and cloud storage, databases, messaging platforms, APIs, and enterprise applications Define secure connectivity, networking, secrets management, and access-control patterns Work with business and technology stakeholders to understand Retail/Grocery data requirements and translate them into scalable data architecture Provide technical leadership and mentoring to data engineers and developers Conduct architecture, design, and code reviews Troubleshoot complex production and performance issues Collaborate with onshore/offshore teams, enterprise architects, infrastructure teams, security teams, and business stakeholders Prepare architecture diagrams, technical design documents, migration plans, and implementation standards Communicate technical risks, dependencies, and recommendations to client leadership

10+ years of experience Bachelor’s Degree

Skills Databricks Apache Spark PySpark SQL Delta Lake Medallion Architecture Unity Catalog Talend CDC and SCD Implementation Cloud Data Platforms Azure Data Lake Storage (ADLS) or S3 Azure Data Factory Kafka REST APIs and Cloud-Native Services Batch and Incremental Data Processing Data Security and Access Control Monitoring and Performance Management Data Migration Strategy ETL/ELT Architecture Data Modeling Data Governance Data Quality and Reconciliation Frameworks Retail/Grocery Domain Expertise Technical Leadership Mentoring and Coaching Technical Documentation Stakeholder Management Code Review Facilitation Performance Optimization Production Support and Troubleshooting Cross-functional Collaboration