> Markdown version of [/jobs/ext/4456523-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/4456523-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - **Company:** RAPS PROFESSIONAL SERVICES PRIVATE LIMITED - **Location:** Cary, NC, United States - **Experience:** Expert - **Salary:** $135,000.0 - $138,000.0 - **Contract:** Permanent contract - **Skills:** LangGraph Framework, Artificial Intelligence, Automation of Tests, Microsoft Azure, Profiling, Code Review, Databases, Continuous Integration, Information Engineering, Data Governance, Data Systems, Graph Database, Information Retrieval, Python (Programming Language), Neo4j, Performance Tuning, Prism (Software), Role-Based Access Control, OpenAI, Azure Data Lake, Scala (Programming Language), SPARQL, SQL Databases, Data Streaming, Management of Software Versions, Azure Service Bus, Data Ingestion, Azure Data Factory, Great Expectations (Foster Youth College-readiness and Support Program), Retrieval-Augmented Generation, Large Language Models, Prompt Engineering, Apache Spark, Model Validation, Llamaindex, Build Management, Data Lakes, AI Platforms, Pyspark, Data Lineage, Integration Frameworks, Apache Kafka, Spark Streaming, Data Management, Machine Learning Operations, Terraform, Data Pipelines, Workday, Key Vault, Databricks - **Published:** October 5, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=39fd22a1636ee81a ## About the Role This is a hands-on technical leadership role. The ideal candidate will have deep experience building production-grade data platforms using Azure, Databricks, Spark, Delta Lake, Python, PySpark, and SQL, along with recent hands-on experience delivering LLM/GenAI solutions., * 12+ years of experience in data engineering, data platforms, architecture, or a related discipline. * Strong hands-on expertise with Azure and Databricks. * Expert-level experience with Python, PySpark, Spark, and SQL; Scala experience is highly desirable. * Proven experience designing and implementing enterprise-scale lakehouse/Medallion architectures. * Strong experience with Delta Lake, Databricks Workflows/Jobs, Unity Catalog, and Spark performance tuning. * Hands-on experience with Azure services such as ADLS Gen2, Azure Data Factory, Event Hubs, Entra ID, RBAC, Key Vault, managed identities, private endpoints, and ACLs. * 3+ years of production experience with LLM/GenAI solutions, including RAG, embeddings, vector/hybrid retrieval, prompt engineering, structured outputs, agent/tool-calling workflows, and evaluation. * Experience with at least one of LangChain, LlamaIndex, or LangGraph. * Experience with Azure OpenAI, OpenAI, or Databricks Model Serving. * Experience building metadata-driven data frameworks, schema inference/profiling, lineage, and configuration-driven engineering. * Experience with Azure DevOps, Terraform, CI/CD, and automated testing. * Strong understanding of enterprise security, governance, PII protection, and responsible AI. * Recent production coding and code-review experience. Preferred Qualifications * Experience in financial services, banking, insurance, accounting, or actuarial data. * Knowledge of RDF/SPARQL, Neo4j, knowledge graphs, ontologies, or property graphs. * Experience with enterprise text-to-SQL and semantic-layer solutions. * Experience with ML-based anomaly detection for transactional or time-series data. * LLMOps/MLOps experience, including model/prompt versioning, evaluation, cost governance, and observability. * Experience with dbt, Great Expectations, or similar data-quality tools. * Exposure to Workday, Workday Prism, or Accounting Center. * Databricks Data Engineer Professional, Azure DP-203/DP-700, or AZ-305 certification. What We're Looking For We are looking for a senior engineer/architect who can design it, build it, review it, and lead others through it. This is not a purely architectural or management position. Candidates should be able to demonstrate recent hands-on delivery of complex Azure + Databricks data platforms and production AI/LLM solutions. Applicants who do not meet the core Azure, Databricks, hands-on data engineering, and production GenAI requirements may not be considered., * data engineering, data platforms, architecture: 10 years (Preferred) * hands-on production experience with Azure and Databricks: 5 years (Preferred) * Expert Python, PySpark/Spark & SQL: 10 years (Preferred) * Hands-on LLM/GenAI & RAG production: 4 years (Preferred) Work Location: Hybrid remote in Cary, NC 27513 ## Description * Lead enterprise lakehouse architecture using Bronze/Silver/Gold (Medallion) patterns, ADLS Gen2, Delta Lake, data contracts, schema evolution, partitioning, and retention strategies. * Design and build metadata-driven and configuration-driven data ingestion frameworks for batch, database extracts, CDC, and streaming workloads. * Develop production-grade solutions using Python, PySpark, Scala, Spark, and SQL. * Optimize Spark and Databricks workloads for performance, scalability, reliability, and cost. * Build streaming pipelines using Spark Structured Streaming, Azure Event Hubs, and/or Kafka. * Implement CI/CD and Infrastructure as Code using Azure DevOps, Terraform, and Databricks Asset Bundles. * Establish data quality, validation, reconciliation, anomaly detection, and observability frameworks. * Build AI-assisted data engineering capabilities including metadata generation, source-to-canonical mapping, data quality rule generation, and automated testing. * Design and implement production LLM/GenAI solutions, including RAG, embeddings, vector/hybrid retrieval, structured outputs, tool calling, and evaluation frameworks. * Work with Azure OpenAI, OpenAI, and/or Databricks Model Serving. * Design semantic-layer and conversational data solutions, including text-to-SQL and knowledge-graph/ontology-based retrieval where applicable. * Implement Unity Catalog, data lineage, fine-grained access controls, PII protection, and enterprise data governance. * Establish AI governance covering model selection, evaluation, guardrails, security, cost, latency, observability, and human-in-the-loop controls. * Mentor engineers and conduct architecture, design, and code reviews. * Troubleshoot complex production issues and remain hands-on with implementation. * Communicate technical architecture and trade-offs to both technical and business stakeholders.