> Markdown version of [/jobs/ext/3558747-data-engineer-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/3558747-data-engineer-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Engineer / Lead Data Engineer - **Company:** Appiness Inc. - **Location:** New York, NY, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Airflow, Amazon Web Services, Amazon S3, Microsoft Azure, Cloud Database, Cloud Storage, Computer Programming, Continuous Integration, Information Engineering, Data Integration, Extract Transform Load (ETL), Data Migration, Data Warehousing, Relational Databases, Github, Apache Hive, Python (Programming Language), Machine Learning, Microsoft SQL Server, MySQL, Operational Databases, Oracle (Applications), Power BI, Azure Data Lake, Reverse Engineering, Shell Script, SQL Databases, Data Streaming, Tableau (Software), Unstructured Data, Data Logging, Data Processing, Azure Data Factory, Large Language Models, Snowflake, Apache Spark, Git, Data Lakes, Pyspark, Kubernetes, AWS Glue, Apache Kafka, Bitbucket, Machine Learning Operations, Amazon Simple Queue Service (SQS), Azure Synapse Analytics, Data Pipelines, Docker, Jenkins, Databricks, Control M - **Published:** October 2, 2026 - **Apply:** https://www.dice.com/job-detail/861b175e-1ecb-4e4d-819f-5972009d0cd1 ## About the Role * 8+ years of experience in Data Engineering or related roles. * Strong hands-on experience with Python, PySpark, Apache Spark, and SQL. * Strong experience with Azure and/or AWS cloud environments. * Experience with Databricks and Delta Lake. * Strong knowledge of Azure Data Factory, Azure Synapse, Azure Data Lake/Blob Storage. * Experience with AWS services such as S3, Glue, Athena, DMS, Lambda, SNS, SQS, and EventBridge. * Strong experience with Snowflake and cloud data warehousing. * Strong understanding of ETL/ELT, data warehousing, data modeling, and data integration. * Experience working with Oracle, SQL Server, MySQL, Hive, and other relational databases. * Strong programming and scripting experience using Python, SQL, and Shell scripting. * Experience with Git, Bitbucket/GitHub, Jenkins, Docker, and Azure DevOps. * . Preferred Skills * Experience with Kafka and real-time streaming. * Experience with Airflow and Control-M. * Knowledge of MLOps and machine-learning data pipelines. * Experience with Kubernetes and containerized data workloads. * Experience with API data integration and modernization. * Knowledge of LLM/AI-assisted data engineering solutions. * Experience with Power BI or Tableau for data reporting and analytics. * Experience with data migration and reverse engineering of legacy data models. ## Description * Design, develop, and maintain scalable ETL/ELT data pipelines for large-volume structured and unstructured data. * Develop data processing solutions using Python, PySpark, Apache Spark, and SQL. * Build and optimize data pipelines using Databricks, Azure Data Factory, AWS Glue, and Snowflake. * Work with both Azure and AWS cloud platforms to implement modern data engineering solutions. * Work with Delta Lake, Azure Data Lake, Amazon S3, Azure Blob Storage, Azure Synapse, and Snowflake. * Develop pipeline orchestration and scheduling using Airflow, Control-M, Databricks Workflows, Azure Data Factory, and AWS services. * Implement monitoring, logging, alerting, and troubleshooting processes for production data pipelines. * Work with CI/CD processes using GitHub, Bitbucket, Jenkins, Docker, and Azure DevOps. ## Related Videos - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [MySQL Protocol Features You Should Be Aware Of](https://www.wearedevelopers.com/videos/100267-mysql-protocol-features-you-should-be-aware-of) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Branch your database like your code: How schema changes and pull requests go hand in hand](https://www.wearedevelopers.com/videos/350-branch-your-database-like-your-code-how-schema-changes-and-pull-requests-go-hand-in-hand) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [The Most Popular IT Jobs on the Market](https://www.wearedevelopers.com/magazine/376-the-most-popular-it-jobs-on-the-market)