Data Engineer

Intersources Inc.

United States

5 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

www2.jobdiva.com

Tech stack

Application Programming Interfaces (APIs) Microsoft Azure Big Data Continuous Integration Extract Transform Load (ETL) Data Warehousing DevOps Apache Hive Python (Programming Language) SQL Azure NoSQL Azure Data Lake

+9 more

Data Streaming Data Storage Technologies Data Ingestion Azure Data Factory Apache Spark Apache Kafka Cosmos DB Azure Synapse Analytics Data Pipelines

Requirements

Experience with complex data modelling, ETL design, and using large databases in a business environment
Experience with building data pipelines and applications to stream and process datasets at low latencies
Fluent with Big Data technologies like Spark, Kafka and Hive
Expert level understanding on Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, and Azure App Service is required
Designing and building of data pipelines using API ingestion and Streaming ingestion methods
Knowledge of Dev-Ops processes (including CI/CD) and Infrastructure as code is essential
Experience in developing NO SQL solutions using Azure Cosmos DB is essential
Working knowledge of Python is desirable.
Managing and optimizing data storage using Azure Data Lake Storage, Azure SQL Data Warehouse, and Azure Cosmos DB
Implementing data security measures, including encryption, access controls, and auditing, to protect sensitive information
Automating data pipelines and workflows to streamline data ingestion, processing, and distribution tasks
Utilizing Azure’s analytics services, such as Azure Synapse Analytics, to provide insights and support data-driven decision-making.