Data Engineer / Lead Data Engineer
Appiness Inc.
New York, NY, United States
about 1 month ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
8 years minimum
Working hours
Regular working hours
Job source
Tech stack
Application Programming Interfaces (APIs)
Artificial Intelligence
Airflow
Amazon Web Services
Amazon S3
Microsoft Azure
Cloud Database
Cloud Storage
Computer Programming
Continuous Integration
Information Engineering
Data Integration
+41 more
Extract Transform Load (ETL)
Data Migration
Data Warehousing
Relational Databases
Github
Apache Hive
Python (Programming Language)
Machine Learning
Microsoft SQL Server
MySQL
Operational Databases
Oracle (Applications)
Power BI
Azure Data Lake
Reverse Engineering
Shell Script
SQL Databases
Data Streaming
Tableau (Software)
Unstructured Data
Data Logging
Data Processing
Azure Data Factory
Large Language Models
Snowflake
Apache Spark
Git
Data Lakes
Pyspark
Kubernetes
AWS Glue
Apache Kafka
Bitbucket
Machine Learning Operations
Amazon Simple Queue Service (SQS)
Azure Synapse Analytics
Data Pipelines
Docker
Jenkins
Databricks
Control M
Job description
- Design, develop, and maintain scalable ETL/ELT data pipelines for large-volume structured and unstructured data.
- Develop data processing solutions using Python, PySpark, Apache Spark, and SQL.
- Build and optimize data pipelines using Databricks, Azure Data Factory, AWS Glue, and Snowflake.
- Work with both Azure and AWS cloud platforms to implement modern data engineering solutions.
- Work with Delta Lake, Azure Data Lake, Amazon S3, Azure Blob Storage, Azure Synapse, and Snowflake.
- Develop pipeline orchestration and scheduling using Airflow, Control-M, Databricks Workflows, Azure Data Factory, and AWS services.
- Implement monitoring, logging, alerting, and troubleshooting processes for production data pipelines.
- Work with CI/CD processes using GitHub, Bitbucket, Jenkins, Docker, and Azure DevOps.
Requirements
- 8+ years of experience in Data Engineering or related roles.
- Strong hands-on experience with Python, PySpark, Apache Spark, and SQL.
- Strong experience with Azure and/or AWS cloud environments.
- Experience with Databricks and Delta Lake.
- Strong knowledge of Azure Data Factory, Azure Synapse, Azure Data Lake/Blob Storage.
- Experience with AWS services such as S3, Glue, Athena, DMS, Lambda, SNS, SQS, and EventBridge.
- Strong experience with Snowflake and cloud data warehousing.
- Strong understanding of ETL/ELT, data warehousing, data modeling, and data integration.
- Experience working with Oracle, SQL Server, MySQL, Hive, and other relational databases.
- Strong programming and scripting experience using Python, SQL, and Shell scripting.
- Experience with Git, Bitbucket/GitHub, Jenkins, Docker, and Azure DevOps.
- .
Preferred Skills
- Experience with Kafka and real-time streaming.
- Experience with Airflow and Control-M.
- Knowledge of MLOps and machine-learning data pipelines.
- Experience with Kubernetes and containerized data workloads.
- Experience with API data integration and modernization.
- Knowledge of LLM/AI-assisted data engineering solutions.
- Experience with Power BI or Tableau for data reporting and analytics.
- Experience with data migration and reverse engineering of legacy data models.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦