> Markdown version of [/jobs/ext/3976105-lead-data-engneer](https://www.wearedevelopers.com/jobs/ext/3976105-lead-data-engneer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Lead Data Engneer

- **Company:** New York, Inc.
- **Location:** Cary, NC, United States
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** LangGraph Framework, Artificial Intelligence, Microsoft Azure, Code Review, Continuous Integration, Information Engineering, Data Governance, Data Transformation, Graph Database, Python (Programming Language), Performance Tuning, OpenAI, Azure Data Lake, Scala (Programming Language), SQL Databases, Azure Data Factory, Retrieval-Augmented Generation, Large Language Models, Prompt Engineering, Apache Spark, Model Validation, Llamaindex, Data Layers, Data Lakes, Pyspark, Production Code, Machine Learning Operations, Terraform, GPT, Data Pipelines, Databricks
- **Published:** October 27, 2026
- **Apply:** https://www.dice.com/job-detail/e59e4f01-b5e4-4ed7-bea8-7f274e05a904

## About the Role

Mandatory Skills - Must-Have

 * Python, PySpark, Scala and SQL - strong production-level data engineering skills.
 * Azure Databricks - Delta Lake, Spark performance tuning, Jobs/Workflows and Unity Catalog.
 * Azure data services - ADLS Gen2 and Azure Data Factory (ADF).
 * Lakehouse / Medallion Architecture - proven experience implementing Bronze, Silver and Gold layers at enterprise scale.
 * Data Pipeline Engineering - metadata-driven ingestion frameworks, data transformation, data quality and large-scale pipeline optimization.
 * Production AI/LLM Experience - hands-on experience designing and deploying LLM-based solutions, including RAG, embeddings, retrieval and prompt engineering.
 * LLM Frameworks - experience with LangChain, LlamaIndex or LangGraph, plus Azure OpenAI, OpenAI or Databricks Model Serving.
 * Azure DevOps / CI-CD - experience automating data pipeline deployments; Terraform and Databricks Asset Bundles are highly relevant.
 * Technical Leadership - ability to design solutions, write production code, review code and guide data engineering teams.
 * Financial services or insurance data, especially financial reconciliation, general ledger or accounting data.
 * AI-driven data quality, anomaly detection and automated reconciliation.
 * Knowledge graphs, ontologies, semantic layers or enterprise Text-to-SQL solutions.
 * Azure security, data governance, PII protection and fine-grained access controls.
 * MLOps/LLMOps, model evaluation and production AI monitoring.