Senior Data Engineer / Data Architect - Databricks

We are seeking a highly experienced Senior Data Engineer / Data Architect with strong Databricks and Insurance domain experience to design, develop, and implement scalable enterprise data solutions., The ideal candidate will have strong hands-on experience with Databricks, Apache Spark, Python, SQL, data engineering, cloud data platforms, data architecture, and insurance data. The candidate will work closely with business stakeholders, data architects, engineers, analysts, and technology teams to build modern data platforms and analytics solutions., * Design and develop scalable data architecture and data engineering solutions using Databricks.

• Build and maintain robust ETL/ELT data pipelines using Databricks, Apache Spark, Python, and SQL.
• Design data lakes, lakehouse architectures, data warehouses, and enterprise data platforms.
• Develop high-performance batch and streaming data pipelines.
• Implement data ingestion from multiple internal and external sources.
• Develop data transformation, cleansing, validation, and integration processes.
• Design scalable and reusable data models for analytics and reporting.
• Work with Delta Lake, Delta Live Tables (DLT), Unity Catalog, and Databricks workflows where applicable.
• Optimize Spark jobs, SQL queries, pipelines, and data-processing workloads.
• Implement data quality, data governance, security, lineage, and access-control processes.
• Collaborate with Data Scientists, BI teams, Business Analysts, Product Owners, and application teams.
• Translate business requirements into technical data architecture and engineering solutions.
• Participate in architecture reviews and establish data engineering best practices.
• Troubleshoot production data issues and provide root-cause analysis.
• Mentor junior and mid-level data engineers.
• Support cloud migration and modernization initiatives.

Insurance Domain Responsibilities

Strong understanding of Property & Casualty (P&C), Life, Health, or other Insurance domain data is highly preferred.

Experience with insurance data such as:

• Policy
• Policyholder
• Customer
• Claims
• Premium
• Billing
• Underwriting
• Rating
• Coverage
• Loss
• Agent / Broker
• Payments
• Risk
• Product
• Quote
• Policy Administration

• Databricks
• Apache Spark / PySpark
• Python
• SQL
• Data Engineering
• Data Architecture
• ETL / ELT
• Data Lake / Lakehouse
• Delta Lake
• Data Modeling
• REST APIs / Data Integration
• Git / CI/CD

Cloud Experience

Strong experience with at least one major cloud platform:

• Microsoft Azure
• AWS
• Google Cloud Platform

Azure Databricks experience is highly preferred.

Experience with technologies such as:

• Azure Data Factory
• Azure Data Lake Storage
• AWS S3
• AWS Glue
• Snowflake
• Kafka
• Airflow

is a plus.

Databricks Skills

Candidates should have experience with several of the following:

• Databricks Workspace
• Apache Spark
• PySpark
• Delta Lake
• Delta Live Tables
• Unity Catalog
• Databricks Workflows
• Databricks SQL
• Cluster configuration and optimization
• Performance tuning
• Data governance
• Data security
• CI/CD for Databricks, * Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
• 8+ years of experience in Data Engineering, Data Architecture, or related roles.
• Strong hands-on experience with Databricks and Spark.
• Strong Python and SQL development experience.
• Experience designing enterprise-scale data platforms.
• Strong understanding of data modeling and data integration.
• Experience working in Agile/Scrum environments.
• Strong communication and stakeholder-management skills.
• Insurance industry/domain experience is required or strongly preferred.

Preferred Experience

• Databricks certification.
• Cloud certification.
• Experience with enterprise insurance platforms.
• Experience with P&C insurance data.
• Experience with data governance and master data management.
• Experience with real-time/streaming data.
• Experience with cloud migration and legacy modernization.
• Experience leading data architecture initiatives.