Databricks Data Engineer

• We are seeking a Databricks Engineer to lead the design and implementation of a scalable Sales Data Platform as part of the One Data initiative on Databricks running on AWS.
• The role is hands-on and architecture-driven, focused on data ingestion, transformation, modelling, and optimization using modern Lakehouse patterns.
• The architect will work closely with data engineers, source system teams, and downstream consumers to deliver high-quality, governed, and performance-optimized sales datasets., Architecture & Design
• Define end-to-end Lakehouse architecture on Databricks (AWS) for Sales data domains.
• Design medallion architecture (Bronze / Silver / Gold) aligned with One Data standards.
• Establish data modelling standards for Sales facts, dimensions, hierarchies, and aggregations.
• Define scalable ingestion patterns for batch and incremental loads.
• Drive performance, scalability, and cost optimization best practices.

Data Engineering & Implementation

• Build and guide development of PySpark-based data pipelines in Databricks.
• Implement Delta Lake features:
• ACID transactions.
• Schema evolution & enforcement.
• Time travel & versioning.
• Design and optimize large-scale joins, aggregations, and window functions.
• Implement CDC and incremental processing using watermarking and change detection.
• Ensure idempotent, restart able, and fault-tolerant pipelines.

AWS & Platform Integration

• Architect solutions using AWS services:
• Amazon S3 (data lake storage).
• IAM (security & access control).
• AWS Glue / Glue CatLog.
• CloudWatch (monitoring & logging).
• Optimize Databricks cluster configurations (job vs all-purpose clusters).
• Implement secrets management and secure connectivity patterns.

Data Quality, Governance & Reliability

• Define and implement data quality checks and validations.
• Ensure data lineage and metadata capture.
• Implement error handling, auditing, and reconciliation frameworks.
• Support data governance and access control requirements.

DevOps & Operational Excellence

• Implement CI/CD pipelines for Databricks notebooks and jobs.
• Enforce code versioning, reviews, and deployment standards.
• Design monitoring, alerting, and SLA tracking for pipelines.
• Support production stabilization and performance tuning.

Collaboration & Leadership

• Act as technical lead / mentor for Databricks data engineers.
• Collaborate with:
• Source system teams (Sales, CRM, ERP).
• Data consumers (analytics, downstream apps).
• Cloud/platform teams.
• Translate business requirements into robust technical designs.

• 8+ years in Data Engineering / Data Architecture roles.
• 4+ years hands-on experience with Databricks.
• Strong expertise in PySpark & Spark SQL.
• Strong expertise in dbt.
• Deep experience with Delta Lake.
• Strong knowledge of AWS cloud services (S3, IAM, Glue, CloudWatch).

Data Engineering Expertise

• Sales data domain experience (orders, revenue, pricing, customers, products).
• Strong understanding of:
• Fact & dimension modelling.
• Slowly Changing Dimensions (SCD Type 1 / 2).
• Large-scale data processing patterns.
• Experience handling high-volume, high-velocity datasets.

Platform & Operational Skills

• Databricks job orchestration and scheduling.
• Cluster sizing and performance tuning.
• CI/CD for data platforms.
• Strong troubleshooting and debugging skills.

Nice-to-Have

• Experience with enterprise One Data / Data Mesh programs.
• Exposure to real-time or near-real-time ingestion patterns.
• Experience integrating CRM / Sales systems (e.g., Salesforce, SAP Sales data).
• AWS certifications or Databricks certifications. Equal Opportunity Employer : .