Data Engineer

Intersources Inc.

1 month ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Languages

English

Azure

Data Governance

Data Integration

ETL

Data Security

Github

Python

MySQL

Oracle Applications

Performance Tuning

Role-Based Access Control

Power BI

Azure

Shell Script

SQL Databases

Data Streaming

Systems Integration

Azure

Snowflake

Change Tracking

Data Lake

PySpark

Deployment Automation

Data Analytics

Spark Streaming

Terraform

Azure

Data Pipelines

Jenkins

Databricks

Ability to design and orchestrate data pipelines using Databricks Workflows and DLT and strong understanding of Medallion Architecture.
Expertise in developing Databricks notebooks for scalable solutions using Python, SQL, and PySpark.
Understanding of Delta Lake architecture, CDC patterns, and Lakehouse.
Performance tuning using Liquid clustering, partitioning, Z-ordering and data skipping in delta tables.
Knowledge on data governance (unity catalog),Data Security (RBAC, Fine grain access control).
Proficient in working with Azure Data Lake Storage Gen2 (ADLS Gen2),Azure Data Factory (ADF) and terraform for provisioning and management of Azure resources.
Knowledge on dealing with Spark streaming and Auto loaders in Data bricks.
Strong experience in analyzing and understanding legacy Informatica ETL workflows, including mappings, transformations, and data flow logic, to support seamless migration to Databricks-based data pipelines.
Hands-on experience in implementing CI/CD pipelines using Jenkins to automate deployment of Databricks notebooks, jobs, and data workflows.
Integrating GitHub with Databricks Repos to enable seamless code synchronization, change tracking, and automated deployment workflows.
knowledge of Snowflake, Oracle, MySQL, and Shell scripting for diverse data integration.
Knowledge of Power BI and Azure Synapse Analytics for data analytics dashboards and reports