Senior Data Engineer - Databricks

Experteer Overview In this Senior Data Engineer role, you will own and optimize the Databricks-based data pipelines that power Sugar Predict, ensuring reliable, scalable data delivery across Azure and AWS. You’ll collaborate with ML, product, and Enterprise Architecture teams to maintain a fast, clean data backbone that supports global scale. You’ll reduce compute costs, modernize legacy pipelines, and enforce security and data quality across multi-tenant environments. This position offers hands-on impact on how revenue intelligence is delivered to mid-market customers and involves on-call incident response and cross-functional work to drive platform growth. Compensation / Benefits * Own Databricks production support across all data flows, including monitoring, alerting, and incident response * Maintain and report SLA metrics for data pipeline delivery and platform health * Optimize pipelines to reduce Databricks compute costs and improve throughput * Migrate legacy ETL/ELT pipelines to Databricks with automation to minimize manual intervention * Onboard and harden tenant data pipelines for new customers to ensure isolated, reliable data from day one * Design and build high-performance Databricks pipelines ingesting and transforming ERP and CRM data at scale across Azure and AWS * Own Delta Lake architecture including schema design, partitioning, quality enforcement, and incremental processing * Enforce data security best practices across Databricks (RBAC, secrets, Unity Catalog, Attestation Maps, compliance) * Implement data quality monitoring and observability to support model inputs and prediction accuracy * Apply multi-tenant isolation patterns for reliable, secure data delivery across enterprise customers * Partner with Enterprise Architecture to integrate pipelines with the SugarAI product ecosystem * Support globally distributed operations with on-call rotation and after-hours incident response * Maintain documentation, runbooks, and architectural decision records for operational readiness * Apply CI/CD practices to data pipelines with version control, testing, and deployment tooling Tasks * 4+ years of data engineering experience * 2+ years on Databricks or Apache Spark in Azure and/or AWS * Proficiency in PySpark, SQL, and Python with production-grade pipelines under SLA * Hands-on experience with Delta Lake (schema evolution, ACID, optimize/vacuum, incremental/streaming) * Pipeline performance tuning and compute optimization in Databricks * PostgreSQL knowledge for production pipelines * Experience maintaining legacy ETL tooling (SSIS, Informatica, custom pipelines) * Experience with large-scale multi-tenant architectures emphasizing isolation and data privacy * Ability to collaborate across data science, product, and infrastructure teams * Strong understanding of data governance, security, and compliance across multi-tenant environments Key requirements * Excellent healthcare package for you and your family * 401(k) match * Unlimited Paid Time Off * Paid Parental Leave * Online Legal Services (Rocket Lawyer) * Financial Planning Services (Origin)

for operational readiness * Apply CI/CD practices to data pipelines with version control, testing, and deployment tooling Tasks * 4+ years of data engineering experience * 2+ years on Databricks or Apache Spark in Azure and/or AWS * Proficiency in PySpark, SQL, and Python with production-grade pipelines under SLA * Hands-on experience with Delta Lake (schema evolution, ACID, optimize/vacuum, incremental/streaming) * Pipeline performance tuning and compute optimization in Databricks * PostgreSQL knowledge for production pipelines * Experience maintaining legacy ETL tooling (SSIS, Informatica, custom pipelines) * Experience with large-scale multi-tenant architectures emphasizing isolation and data privacy * Ability to collaborate across data science, product, and infrastructure teams * Strong understanding of data governance, security, and compliance across multi-tenant environments Key requirements * Excellent healthcare package for you and your family * 401(k) match * Unlimited Paid a and Off * Paid Parental Leave * Online Legal Services (Rocket Lawyer) * Financial Planning Services (Origin)