Lead Data Engineer

Insight Global

San Francisco, CA, United States

3 months ago

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

5 years minimum

Working hours

Regular working hours

Job source

juju.com

Tech stack

Application Programming Interfaces (APIs) Amazon Web Services Business Analytics Applications Microsoft Azure Cloud Database Computer Programming Data Validation Information Engineering Extract Transform Load (ETL) Data Warehousing IBM InfoSphere DataStage Apache Hive

+14 more

IBM Cognos Business Intelligence Python (Programming Language) NoSQL Raw Data Software Engineering SQL Databases SQL Server Analysis Services Data Streaming Tableau (Software) Sql Optimization Apache Spark Apache Kafka Data Pipelines Databricks

Requirements

Must Haves:

At least 5 years of strong hands-on expertise with Databricks and Datastage (or comparable ETL tools)
Experience with migrations
Exs: Migrating SQL to cloud, SQL to Azure Databricks, (on prem to the cloud)
8-10 years of software development and data engineering experience with SQL, ADF, SSAS cubes, Cognos, Tableau, ThoughtSpot and other BI tools
Advanced SQL skills for processing raw data, Kafka ingestions, ADF pipelines, data validation and QA with experience across SQL and NoSQL databases
Programming experience with Python and/or Scala
Experience ingesting data via APIs and streaming sources
Experience building data pipelines, data warehouses, and analytics platforms
Proficiency with Spark, Hive, Kafka, and cloud data services
Experience with Azure, AWS or equivalent cloud platforms