Sr. Databricks Engineer

Senior individual contributor responsible for designing, developing, enhancing, and supporting enterprise cloud data platform capabilities.

• Reports to the Sr. Director of Data Platform Transformations & Operations.
• Serves as a technical lead for major initiatives and accountable for production support and operational excellence.
• Focus: ~70% new development, ~30% operations/support/platform reliability.
• Primary technologies: Azure Databricks, PySpark, SQL, Delta Lake, cloud data engineering., Design, develop, and maintain scalable data pipelines, ingestion frameworks, and transformation processes.
• Implement Medallion (Bronze/Silver/Gold) architecture for reporting, analytics, AI, and self-service data.
• Develop and operationalize AI agent solutions on Databricks, integrating models, tools, and APIs.
• Build reusable frameworks, utilities, and platform components for productivity and quality.
• Develop and modernize batch, near-real-time, and streaming integration solutions.
• Serve as technical lead-solution design, technical reviews, tool recommendations, and implementation oversight.
• Partner with architects to ensure scalable, secure, governed, and cost-effective solutions.
• Mentor other engineers and promote coding, testing, and documentation standards.
• Implement data quality, validation, reconciliation, monitoring, and security controls.
• Build and maintain CI/CD pipelines, automated testing, and release processes (Azure DevOps/Git).
• Contribute to platform observability, alerting, monitoring, performance tuning, and cost optimization.
• Participate in production support, incident response, troubleshooting, and root cause analysis.
• Create and maintain runbooks, support documentation, and knowledge-sharing resources.
• Collaborate with cross-functional teams to translate requirements into platform solutions.
• Support technical discovery, estimation, planning, delivery, and evaluation of new cloud/data/AI capabilities.

Bachelor’s in Computer Science, Information Systems, Engineering, Data Science, or related field; Master’s preferred.

• 10+ years in data engineering, warehousing, or platform engineering.
• 5+ years designing/implementing cloud-based data platforms.
• 3+ years hands-on Azure Databricks in enterprise environments.
• Advanced SQL, Python, PySpark, and Spark optimization.
• Experience with Databricks agentic AI, Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
• Deep understanding of lakehouse concepts, Medallion architecture, data modeling, and analytics data structures.
• Experience with Azure DevOps, Git, CI/CD, automated testing/deployment; Databricks Asset Bundles preferred.
• Implementation of data quality, validation, governance, and security controls.
• Strong leadership, solution design, troubleshooting, and communication skills.
• Experience in regulated environments (financial services/banking/audit-sensitive) preferred.
• Databricks Data Engineer certification preferred.

Nice to Have:

• Financial services industry experience.
• GenAI project deployment experience.
• Data governance expertise.
• Familiarity with Power BI, MicroStrategy, MDM, streaming architectures, infrastructure automation, or AI-assisted development.