Lead Data Engineer -Databricks
Role details
Job location
Tech stack
Job description
Alphanumeric is hiring a DATA ENGINEER IV LEAD to work remotely (EST hours preferred) with an established leader in the financial and insurance industries. Candidates located in or near Richmond, VA are strongly preferred. This is a contract-to-hire opportunity with an approximate conversion salary of $135,000 annually. Pay Range: $93.00 - $100.00/hr. W2 No third-party agencies please. Sponsorship is not available for this position. As a member of the Data Solutions and Data Engineering team, you will play a key role in transforming enterprise data capabilities using Databricks and modern cloud-based data engineering practices. This position focuses heavily on enhancing out-of-the-box Databricks functionality, developing scalable frameworks, optimizing Delta Lake infrastructure, and supporting large-scale data transformation initiatives utilizing medallion architecture principles. You will help design and build robust data ingestion, standardization, and curation pipelines across bronze, silver, and gold layers while improving the usability and accessibility of enterprise data. This includes supporting initiatives involving AI-powered extraction of underwriting and medical data from images and PDFs to improve policy cost analysis and business insights. What You'll Be Doing:
- Partner with business users to gather and define data requirements
- Collaborate with architects and technical leads to design scalable Databricks solutions
- Build and support data engineering solutions throughout the full SDLC lifecycle
- Develop frameworks and reusable components for pipeline standardization, SLA monitoring, and data quality management
- Create and optimize Delta tables, Databricks dashboards, and orchestration workflows
- Design and maintain dimensional and ER-based data models utilizing medallion architecture
- Implement batch and streaming data pipelines within Databricks
- Create unit tests, perform SIT testing, and support UAT troubleshooting efforts
- Debug and resolve data defects and performance issues
- Implement and maintain data security policies and compliance standards
- Work closely with upstream/downstream teams, including offshore and vendor partners
- Produce technical documentation, training materials, and knowledge transfer sessions
- Research and evaluate emerging Databricks capabilities including Intelligent Document Processing, Genie AI coding, and self-service workspace features
Requirements
- Minimum 3 years of hands-on Databricks experience in Azure or AWS environments
- Strong experience utilizing medallion architecture and modern data modeling techniques
- Expertise creating reusable data engineering frameworks and standardization processes
- Experience developing dimensions and fact tables with SCD2 tracking
- Strong Databricks orchestration experience with both batch and streaming pipelines
- Experience performing testing, debugging, and production support
- Strong understanding of Databricks compute and storage optimization
- Experience with CI/CD, GitLab, and deployment automation tools
- Strong understanding of Agile methodologies, story creation, and effort estimation
Hands-On Technical Skills:
- Expert-level SQL skills
- Intermediate or higher proficiency in PySpark
- Experience with Lakeflow and data quality expectations coding
- Strong understanding of Databricks features including Unity Catalog, Spark UI, Job Scheduling, and related capabilities
- Ability to quickly learn and implement newer Databricks technologies and AI-driven capabilities, * 5+ years of Databricks experience building enterprise-scale capabilities from scratch
- Experience working within insurance or financial services environments
- Experience with data governance and observability platforms
- Understanding of enterprise data models and schema evolution strategies
Soft Skills:
- Excellent communication and presentation skills
- Ability to create training materials and technical documentation
- Strong problem-solving and analytical thinking
- Collaborative mindset with strong teamwork skills
- Design-thinking approach to solution development