Lead Data Engneer
New York, Inc.
Cary, NC, United States
4 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours
Job source
Tech stack
LangGraph Framework
Artificial Intelligence
Microsoft Azure
Code Review
Continuous Integration
Information Engineering
Data Governance
Data Transformation
Graph Database
Python (Programming Language)
Performance Tuning
OpenAI
+19 more
Azure Data Lake
Scala (Programming Language)
SQL Databases
Azure Data Factory
Retrieval-Augmented Generation
Large Language Models
Prompt Engineering
Apache Spark
Model Validation
Llamaindex
Data Layers
Data Lakes
Pyspark
Production Code
Machine Learning Operations
Terraform
GPT
Data Pipelines
Databricks
Requirements
Mandatory Skills - Must-Have
- Python, PySpark, Scala and SQL - strong production-level data engineering skills.
- Azure Databricks - Delta Lake, Spark performance tuning, Jobs/Workflows and Unity Catalog.
- Azure data services - ADLS Gen2 and Azure Data Factory (ADF).
- Lakehouse / Medallion Architecture - proven experience implementing Bronze, Silver and Gold layers at enterprise scale.
- Data Pipeline Engineering - metadata-driven ingestion frameworks, data transformation, data quality and large-scale pipeline optimization.
- Production AI/LLM Experience - hands-on experience designing and deploying LLM-based solutions, including RAG, embeddings, retrieval and prompt engineering.
- LLM Frameworks - experience with LangChain, LlamaIndex or LangGraph, plus Azure OpenAI, OpenAI or Databricks Model Serving.
- Azure DevOps / CI-CD - experience automating data pipeline deployments; Terraform and Databricks Asset Bundles are highly relevant.
- Technical Leadership - ability to design solutions, write production code, review code and guide data engineering teams.
- Financial services or insurance data, especially financial reconciliation, general ledger or accounting data.
- AI-driven data quality, anomaly detection and automated reconciliation.
- Knowledge graphs, ontologies, semantic layers or enterprise Text-to-SQL solutions.
- Azure security, data governance, PII protection and fine-grained access controls.
- MLOps/LLMOps, model evaluation and production AI monitoring.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦