Lead Databricks Data Engineer

As a lead Databricks Data Engineer, you will be hands-on designing, building, and maintaining robust data pipelines within Databricks and contributing independently while also providing technical leadership and mentorship within the team. Your key responsibilities will include creating and optimizing end-to-end data flows, handling extract, transform, load (ETL) processes, designing data models, ensuring data security and compliance, collaborating with cross-functional teams, and continuously improving performance and scalability. Your experience in both Databricks and AWS environments and services will position you as a technical team leader. Your expertise will bridge the gap between raw data and actionable insights, enabling efficient data processing and analytics across the business.

Here, you’ll mentor a multi-disciplinary team of analysts, data engineers, developers, and data consumers in a fast-paced, agile environment. You’ll use your expertise in analytical exploration and data examination while you oversee the assessment, design, building, and maintenance of scalable platforms for your clients.

• 8+ years of experience designing, developing, deploying, and managing ETL processes and data pipelines utilizing comprehensive logging and robust error handling mechanisms to troubleshoot and resolve issues
• 3+ years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools using Databricks tools such as Delta Lake, Databricks Delta, and Unity Catalog
• 3+ years of experience writing Python and PySpark code for ETL scripts and custom transformations
• Experience in data modeling, including when to use dimensional vs relation data models within a medallion architecture, including Bronze, Silver, or Gold
• Experience using various AWS services, including AWS Glue, Amazon S3, Amazon RDS
• Knowledge of performance optimization techniques for ETL jobs such as partitioning, bucketing, caching, and troubleshooting to improve query performance
• Ability to design and implement data models from initially defining schemas to optimizing data structures that support efficient queries and analytics
• Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements
• Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology

Nice If You Have:

• Experience with AWS cloud-native data services such as AWS Lambda or CloudWatch that support serverless architectures for data processing
• Experience with tools such as AWS Glue Data Catalog or Collibra for managing metadata, data lineage, and cataloging data assets
• Knowledge of data governance practices, data lineage, and compliance requirements, including GDPR or CCPA, in support of data quality, privacy, and security
• Ability to explain complex technical concepts to a non-technical audience, including describing tradeoffs in architectural design