Data Engineer / Lead Data Engineer

Appiness Inc.

North Brunswick Township, NJ, United States

5 days ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

8 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Application Programming Interfaces (APIs) Artificial Intelligence Airflow Amazon Web Services Amazon S3 Microsoft Azure Cloud Database Cloud Storage Computer Programming Continuous Integration Information Engineering Data Integration

+41 more

Extract Transform Load (ETL) Data Migration Data Warehousing Relational Databases Github Apache Hive Python (Programming Language) Machine Learning Microsoft SQL Server MySQL Operational Databases Oracle (Applications) Power BI Azure Data Lake Reverse Engineering Shell Script SQL Databases Data Streaming Tableau (Software) Unstructured Data Data Logging Data Processing Azure Data Factory Large Language Models Snowflake Apache Spark Git Data Lakes Pyspark Kubernetes AWS Glue Apache Kafka Bitbucket Machine Learning Operations Amazon Simple Queue Service (SQS) Azure Synapse Analytics Data Pipelines Docker Jenkins Databricks Control M

Job description

Design, develop, and maintain scalable ETL/ELT data pipelines for large-volume structured and unstructured data.
Develop data processing solutions using Python, PySpark, Apache Spark, and SQL.
Build and optimize data pipelines using Databricks, Azure Data Factory, AWS Glue, and Snowflake.
Work with both Azure and AWS cloud platforms to implement modern data engineering solutions.
Work with Delta Lake, Azure Data Lake, Amazon S3, Azure Blob Storage, Azure Synapse, and Snowflake.
Develop pipeline orchestration and scheduling using Airflow, Control-M, Databricks Workflows, Azure Data Factory, and AWS services.
Implement monitoring, logging, alerting, and troubleshooting processes for production data pipelines.
Work with CI/CD processes using GitHub, Bitbucket, Jenkins, Docker, and Azure DevOps.

Requirements

8+ years of experience in Data Engineering or related roles.
Strong hands-on experience with Python, PySpark, Apache Spark, and SQL.
Strong experience with Azure and/or AWS cloud environments.
Experience with Databricks and Delta Lake.
Strong knowledge of Azure Data Factory, Azure Synapse, Azure Data Lake/Blob Storage.
Experience with AWS services such as S3, Glue, Athena, DMS, Lambda, SNS, SQS, and EventBridge.
Strong experience with Snowflake and cloud data warehousing.
Strong understanding of ETL/ELT, data warehousing, data modeling, and data integration.
Experience working with Oracle, SQL Server, MySQL, Hive, and other relational databases.
Strong programming and scripting experience using Python, SQL, and Shell scripting.
Experience with Git, Bitbucket/GitHub, Jenkins, Docker, and Azure DevOps.
.