Data Engineer - Databricks, ADF, SQL - Technical Lead

INSYSTECH, INC.

St. Louis, MO, United States

5 days ago

Apply on www.careerjet.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

5 years minimum

Compensation

$140,000.0 - $160,000.0

Working hours

Regular working hours

Job source

www.careerjet.com

Tech stack

Amazon Web Services Business Analytics Applications Microsoft Azure Cloud Computing Continuous Integration Information Engineering Data Governance Extract Transform Load (ETL) Data Transformation Data Security Data Systems Data Warehousing

+17 more

Distributed Systems Python (Programming Language) Meta-Data Management Scrum Methodology SQL Databases Data Streaming Cloud Platform System Feature Engineering Snowflake Apache Spark Git Data Lakes Pyspark Data Lineage Machine Learning Operations Data Pipelines Databricks

Job description

Design and implement data engineering solutions using Databricks, Delta Lake, Spark, and cloud platforms (AWS/Azure)
Hands-on expertise in Snowflake integration and data warehouse optimization for the Health Insurance account, * Design and architect data solutions using Databricks and Delta Lake
Develop scalable data pipelines using Apache Spark and PySpark
Integrate Snowflake data warehouse solutions with Databricks platforms
Optimize cloud platform performance on AWS or Azure
Implement data governance and quality frameworks
Optimize Databricks cluster configurations and performance
Build and maintain data lakehouses and analytics solutions
Ensure data security, lineage tracking, and compliance
Collaborate with data scientists and analytics teams
Support cloud migrations and modernization initiatives, Role: Lead Data Engineer with Databricks & AWS or if you find any Architects do not hold them please share it Location: St. Louis, MO Work type: 3 days onsite per week Skill …

Requirements

Visa Independent: Green Card holder or US Citizen
5+ years of enterprise-scale data engineering experience
Hands-on implementation experience (not just theoretical knowledge)
Ability to work hybrid model in St. Louis, MO (or relocation to secondary locations)
Strong problem-solving and analytical skills
Experience with both batch and streaming data processing
Strong communication and collaboration skills, * 5+ years of hands-on Databricks and/or Apache Spark development/implementation experience
Strong proficiency with PySpark and Scala (or Python)
Deep understanding of Delta Lake and ACID transactions
Experience designing and implementing data lakehouses
Proficiency with SQL and data transformation logic
Cloud platform expertise: AWS or Azure (hands-on implementation)
Snowflake experience: Integration, optimization, and ETL pipeline development
Understanding of distributed computing and optimization
Git and CI/CD pipeline experience

Nice-to-Have

Healthcare or insurance industry experience
Databricks certification or advanced training
AWS or Azure cloud certifications
Data governance and metadata management experience
Experience with ML pipelines and feature engineering
Agile/Scrum experience
Experience with data quality frameworks

About the company

About Brillio: Brillio is one of the fastest growing digital technology service providers and a partner of choice for many Fortune 1000 companies seeking to turn disruption into …