Senior Databricks Data Engineer

Job DescriptionJob Title: Senior Databricks Data Engineer Location-Type: Onsite, Santa Monica, CA (office relocation to Century City anticipated within approximately one year) Start Date: ASAP Duration: Permanent Compensation Range: $200,000 - $225,000 Benefits: Medical premiums 100% covered, Dental and Vision premiums 100% covered, 401K with employer match up to 6%, 14 days PTO, 9 sick days annually, 11 paid holidays, $20K lifetime fertility benefit, daily catered lunch, weekly carwash service, monthly cellphone reimbursement Visa Sponsorship: Not eligible for visa sponsorship Job Description: This role leads the technical design and hands-on development of an enterprise-grade Azure Databricks Lakehouse platform, delivering scalable data infrastructure and governance capabilities across the organization. Job Summary *Build and evolve an internal Azure Databricks Lakehouse platform using Delta Lake and standardized architectural patterns such as medallion architecture. *Develop scalable

ingestion and transformation frameworks for multiple source systems using Auto Loader, CDC incremental patterns, and Delta best practices. *Implement curated data products with clear contracts covering schemas, SLAs, data quality rules, ownership, and documentation. *Operationalize pipelines using Databricks Workflows, including scheduling, dependencies, retries, alerting, and run-time observability. *Own performance and cost optimization across workloads, including cluster policies, partitioning strategies, file sizing, and caching. *Implement and administer data governance and security using Unity Catalog, including least-privilege access patterns and audit standards. *Mentor junior and mid-level engineers, lead architecture and design reviews, and drive adoption of platform standards and best practices. Minimum Requirements: *8 years of data engineering experience, including 3 years of recent, hands-on Azure Databricks experience in production environments. *Strong proficiency in Python (PySpark) and Spark SQL for large-scale data transformation. *Strong experience with Delta Lake, including MERGE/UPSERT, schema evolution, time travel, ACID transactions, and incremental processing. *Solid Azure fundamentals relevant to a data platform, including ADLS Gen2, service principals, managed identities, and Azure Key Vault secrets management. *Demonstrated ability to work cross-functionally and communicate technical decisions clearly to non-engineering stakeholders. *Experience with financial datasets such as general ledger, subledger, and accounting line items. Preferred Qualifications: *Databricks Certified Data Engineer Professional certification (Associate certification also considered). *Experience implementing Unity Catalog in an enterprise environment, including RBAC model design and audit patterns. *Experience with CI/CD for Databricks assets using Azure DevOps or GitHub Actions, and familiarity with Terraform or asset-bundle deployment. *Experience with streaming or near-real-time ingestion using Structured Streaming, Event Hubs, or Delta Live Tables. *Private equity, credit, or alternative asset management industry experience. *Bachelor’s degree in computer science, data engineering, mathematics, or a related field (preferred but not required).