> Markdown version of [/jobs/ext/3793264-data-engineer](https://www.wearedevelopers.com/jobs/ext/3793264-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Data Engineer

- **Company:** Insight Global
- **Location:** Harrison, NY, United States
- **Contract:** Permanent contract
- **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Airflow, Amazon Web Services, Automation of Tests, Microsoft Azure, Cloud Engineering, Continuous Integration, Data Governance, Distributed Computing Environment, Python (Programming Language), Machine Learning, Metadata, Meta-Data Management, Azure Machine Learning, SQL Databases, Data Streaming, Systems Integration, Data Ingestion, Apache Spark, Generative AI, Event Driven Architecture, Build Management, Microsoft Fabric, Data Lakes, AI Platforms, Pyspark, Kubernetes, Infrastructure Automation Frameworks, Deployment Automation, Apache Kafka, Data Management, Machine Learning Operations, Software Version Control, Data Pipelines, Databricks
- **Published:** October 17, 2026
- **Apply:** https://dejobs.org/x/x/032DA5EF148941B0A9CC04EE8B3157BE/job/

## About the Role

* Extensive experience designing and building enterprise-scale data engineering solutions. * Strong hands-on experience with Python, PySpark, SQL, Apache Spark and distributed data processing. * Experience designing and operating large-scale batch and real-time data pipelines and modern lakehouse platforms like databricks. * Experience building cloud-native data solutions using AWS, Azure, or comparable enterprise cloud platforms. * Experience with modern data architecture patterns including data lakes, lakehouses, data products, and event-driven architectures. * Experience with orchestration technologies such as Apache Airflow, Databricks Workflows equivalent. * Strong understanding of data governance, data quality, lineage, metadata management, security, privacy, and access controls. * Ability to diagnose and resolve complex performance, scalability, reliability, and operational issues. * Strong problem-solving, communication, and cross-functional collaboration skills. *
Experience integrating structured, semi-structured, unstructured, and streaming data. * Ability to work effectively in an R&D environment with evolving requirements and rapid experimentation. * Experience supporting machine learning, generative AI, or agentic AI applications. * Experience with CI/CD, automated testing, version control, Infrastructure as Code, observability, and platform automation. * Experience with MLflow, Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric. * Experience with Kafka or other event-streaming technologies. * Experience in payments, financial services, or another regulated industry.

## Description

This is a highly hands-on engineering role focused on translating evolving R&D requirements into secure, reliable, scalable, and production-ready data platforms and pipelines. You will collaborate with AI Engineers, Data Scientists, Researchers, Software Engineers, and Product Managers to accelerate experimentation and production delivery.

 * Design and build scalable batch, streaming, and event-driven data pipelines and platforms. * Develop data ingestion, transformation, enrichment, processing, and consumption capabilities using modern lakehouse and cloud-native architectures. * Build reusable data products, APIs, services, and pipelines supporting analytics, machine learning, generative AI, and agentic AI workloads. * Optimize solutions for performance, scalability, reliability, resiliency, observability, and cost efficiency. * Implement engineering practices including data modeling, automated testing, CI/CD, Infrastructure as Code, and deployment automation. * Implement data quality, lineage, metadata, cataloging, security, privacy, governance, and access controls. * Evaluate emerging data and AI technologies through prototypes and proof-of-concepts and transition successful solutions toward production. * Troubleshoot complex data, performance, scalability, and production issues. * Translate evolving product and
R&D requirements into practical technical designs and incremental delivery plans. * Collaborate with cross-functional teams and communicate technical decisions, dependencies, risks, and trade-offs.