Sr. Data Engineer / Lead (SQL / Databricks)

Keypixel Software Solutions
Irving, United States of America
yesterday

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Irving, United States of America

Tech stack

Azure
Continuous Integration
Data Integration
ETL
Data Transformation
Database Development
Python
Microsoft SQL Server
SQL Azure
Azure
SQL Databases
Pulumi
Data Processing
Azure
GIT
PySpark
Data Pipelines
Jenkins
Databricks

Job description

Introduction: As a Sr. Data Engineer / Lead specializing in SQL and Databricks, you will play a key role in our team, working on data transformation and integration projects. You will be responsible for designing, developing, and maintaining data pipelines to support our organization''s data needs., * Utilize SQL Server and Azure SQL to develop and optimize database solutions

  • Work with Databricks and PySpark for data processing and analysis
  • Use Python for scripting and automation tasks
  • Implement and manage Azure Data Factory (ADF) for data integration
  • Perform Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) processes
  • Work with Azure services such as ADLS Gen2 and Azure SQL
  • Implement Continuous Integration and Continuous Delivery (CI/CD) pipelines using Git and Jenkins
  • Take ownership of projects and lead a team of data engineers

Requirements

  • Experience with SQL Server and Azure SQL
  • Proficiency in Databricks and PySpark
  • Strong knowledge of Python programming language
  • Hands-on experience with Azure Data Factory (ADF)
  • Experience with ETL/ELT processes
  • Familiarity with Azure services like ADLS Gen2 and Azure SQL
  • Experience with CI/CD practices, Git, and Jenkins
  • Prior experience in a lead or ownership role

Preferred:

  • Knowledge of Pulumi for Infrastructure as Code (IaC)
  • Background in Healthcare or Clinical Domain

Apply for this position