Senior Data Engineer

Insight Global is seeking a Senior Data Engineer for a leading insur-tech client. This individual will join a highly technical Data Engineering team focused on modernizing a core subrogation platform and revamping customer data integration capabilities. The ideal candidate is a deeply technical engineer who thrives in large-scale data environments, understands the business impact of data solutions, and can design highly efficient data pipelines within AWS. This role will be responsible for building and optimizing ETL/ELT frameworks, maintaining data lakes and feature stores, supporting machine learning initiatives, and enabling seamless customer integrations through APIs and Guidewire-based solutions. Success in this position requires strong Spark expertise, deep AWS knowledge, excellent communication skills, and the ability to connect technical decisions to measurable business outcomes.

Day-to-Day:

Design, build, and optimize ETL/ELT pipelines that transform customer and enterprise data into actionable datasets

Develop and maintain scalable AWS-based data platforms and data lakes

Partner with Product and Market teams to create reporting tools and dashboards that support business strategy

Build and maintain feature stores used by ML Engineering teams

Develop customer data integration pipelines via APIs, Guidewire, and other ingestion methods

Implement data quality controls and monitoring frameworks

Troubleshoot data issues across distributed systems and streaming architectures

Optimize Spark workloads and cloud resources for performance and cost savings

Must-Haves:

Strong proficiency in Python and SQL

Recent, hands-on experience with Apache Spark in production environments

Deep expertise building and optimizing ETL/ELT pipelines

Extensive experience with AWS data services, Experience with distributed processing technologies beyond managed platforms

Experience with data streaming technologies:

Kafka

Spark Streaming

SQS-based event architectures

Strong data modeling experience, preferably within insurance or financial services

Experience designing and maintaining data lakes

Knowledge of medallion architecture (Bronze/Silver/Gold)

Schema evolution and incremental processing experience

Familiarity with MongoDB and document-based data stores

Infrastructure as Code experience (Terraform or CloudFormation)

Experience troubleshooting and resolving data quality issues across distributed systems

Experience implementing data quality frameworks such as Great Expectations

Strong business acumen with the ability to translate technical solutions into business outcomes

Ability to optimize pipelines for performance, scalability, and cost efficiency

Excellent communication skills and ability to explain the business problem being solved Plusses:

Experience in Insur-Tech or Technology

Experience handling insurance claims data and complex feature relationships

Experience supporting machine learning platforms or feature stores