Senior Data Engineer (Azure and Databricks)

OMSAI TECH LLC

New Brunswick, NJ, United States

3 months ago

Apply on indeed.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Compensation

$104,000.0 - $124,800.0

Working hours

Regular working hours

Job source

indeed.com

Tech stack

Clean Code Principles Agile Methodology Microsoft Azure Bash Shell Big Data Cloud Computing Data Integration Extract Transform Load (ETL) Data Security Database Design Linux Python (Programming Language)

+19 more

Linked Data Unix Shell Microsoft SQL Server SQL Azure Oracle (Applications) Azure Data Lake Shell Script SQL Databases Systems Integration Talend Azure Data Factory System Availability Apache Spark Reliability of Systems Data Management Dynamic Data Restful APIs Data Pipelines Databricks

Job description

We are seeking a highly skilled and energetic Senior Data Engineer with expertise in Azure cloud platform and Databricks to join our dynamic data team. In this role, you will lead the design, development, and implementation of scalable data solutions that empower data-driven decision-making across the organization., * Design, develop, and maintain scalable ETL (Extract, Transform, Load) processes using Azure Data Lake, Databricks, Spark, and other big data tools to support complex analytics and reporting needs.

Collaborate with cross-functional teams to gather requirements and translate them into efficient data models and pipelines that facilitate seamless data access and analysis.
Implement cloud-based solutions utilizing Azure services such as Azure Data Factory, Azure SQL Database, and Azure Data Lake Storage to ensure high availability and security of enterprise data assets.
Write clean, efficient code in Python, Bash (Unix shell), or Shell Scripting to automate workflows and improve system reliability.
Apply Agile methodologies to manage project tasks efficiently while continuously improving processes through collaboration and feedback.
Support database design efforts by creating schemas that promote efficient storage, retrieval, and analysis of large datasets including linked data concepts for semantic interoperability.

Requirements

Do you have experience in Schema design?, * Proven experience as a Data Engineer working with Azure cloud services including Azure Data Lake, Databricks, Data Factory, and related tools.

Hands-on experience with SQL databases like Microsoft SQL Server and Oracle; knowledge of database design principles is essential.
Proficiency in programming languages such as Python
Experience with ETL tools like Talend or Informatica for complex data integration workflows.
Demonstrated ability to develop RESTful APIs for seamless system integration.
Familiarity with shell scripting (Bash/Unix shell) for automation purposes in a Linux environment.