Principal Data Engineer

Insight Global
San Diego, United States of America
yesterday

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

San Diego, United States of America

Tech stack

Unity
Amazon Web Services (AWS)
Azure
Cloud Database
Data Architecture
Information Engineering
Data Governance
ETL
Python
Laboratory Information Management Systems
Performance Tuning
SQL Databases
Data Processing
Snowflake
Spark
Data Lake
Data Management
Software Version Control
Data Pipelines
Databricks

Job description

a pharmaceutical company based in San Diego is seeking a Principal level Data Engineer to join their team in support of their Research Data Solutions. The Principal Data Engineer is a senior technical leader responsible for designing, building, and optimizing scalable data solutions that support research and scientific data initiatives. This role requires deep expertise in modern data engineering practices, with a strong emphasis on Databricks, Spark, Delta Lake, lakehouse architecture, data quality, version control, performance optimization, and engineering best practices.The Principal Data Engineer partners primarily with business analysts, platform engineers, and other data engineers to translate research data needs into reliable, governed, and reusable data assets. This individual serves as a hands-on technical expert, architectural advisor, and mentor, helping establish standards and patterns for research-focused data engineering across the organization.

Requirements

  • 7+ years of experience as Data Engineer, with proven knowledge or experience in the Biopharma/Life Science industry.
  • Expert-level experience architecting and building scalable data solutions using Databricks, including Spark, Delta Lake, Unity Catalog, and lakehouse best practices.
  • Advanced proficiency in Python and SQL, with deep experience in Spark-based data processing, performance tuning, and production analytics workloads.
  • Strong experience designing and operating production ETL/ELT data pipelines, including orchestration, incremental processing, monitoring, and reliability.
  • Hands-on experience working in cloud data environments (Azure and/or AWS) supporting enterprise-scale data platforms.
  • Proven technical leadership experience, including leading architecture decisions, mentoring engineers, and partnering with analytics, platform, and research teams. - Hands-on experience with LIMS, ELN, or scientific/discovery data sources.
  • Strong understanding of data governance, security, privacy, and compliance practices in regulated environments.
  • Experience working with Snowflake or other cloud data warehouses alongside Databricks.

Apply for this position