Lead Data Engineer

• Lead enterprise lakehouse architecture using Bronze/Silver/Gold (Medallion) patterns, ADLS Gen2, Delta Lake, data contracts, schema evolution, partitioning, and retention strategies.
• Design and build metadata-driven and configuration-driven data ingestion frameworks for batch, database extracts, CDC, and streaming workloads.
• Develop production-grade solutions using Python, PySpark, Scala, Spark, and SQL.
• Optimize Spark and Databricks workloads for performance, scalability, reliability, and cost.
• Build streaming pipelines using Spark Structured Streaming, Azure Event Hubs, and/or Kafka.
• Implement CI/CD and Infrastructure as Code using Azure DevOps, Terraform, and Databricks Asset Bundles.
• Establish data quality, validation, reconciliation, anomaly detection, and observability frameworks.
• Build AI-assisted data engineering capabilities including metadata generation, source-to-canonical mapping, data quality rule generation, and automated testing.
• Design and implement production LLM/GenAI solutions, including RAG, embeddings, vector/hybrid retrieval, structured outputs, tool calling, and evaluation frameworks.
• Work with Azure OpenAI, OpenAI, and/or Databricks Model Serving.
• Design semantic-layer and conversational data solutions, including text-to-SQL and knowledge-graph/ontology-based retrieval where applicable.
• Implement Unity Catalog, data lineage, fine-grained access controls, PII protection, and enterprise data governance.
• Establish AI governance covering model selection, evaluation, guardrails, security, cost, latency, observability, and human-in-the-loop controls.
• Mentor engineers and conduct architecture, design, and code reviews.
• Troubleshoot complex production issues and remain hands-on with implementation.
• Communicate technical architecture and trade-offs to both technical and business stakeholders.

This is a hands-on technical leadership role. The ideal candidate will have deep experience building production-grade data platforms using Azure, Databricks, Spark, Delta Lake, Python, PySpark, and SQL, along with recent hands-on experience delivering LLM/GenAI solutions., * 12+ years of experience in data engineering, data platforms, architecture, or a related discipline.

• Strong hands-on expertise with Azure and Databricks.
• Expert-level experience with Python, PySpark, Spark, and SQL; Scala experience is highly desirable.
• Proven experience designing and implementing enterprise-scale lakehouse/Medallion architectures.
• Strong experience with Delta Lake, Databricks Workflows/Jobs, Unity Catalog, and Spark performance tuning.
• Hands-on experience with Azure services such as ADLS Gen2, Azure Data Factory, Event Hubs, Entra ID, RBAC, Key Vault, managed identities, private endpoints, and ACLs.
• 3+ years of production experience with LLM/GenAI solutions, including RAG, embeddings, vector/hybrid retrieval, prompt engineering, structured outputs, agent/tool-calling workflows, and evaluation.
• Experience with at least one of LangChain, LlamaIndex, or LangGraph.
• Experience with Azure OpenAI, OpenAI, or Databricks Model Serving.
• Experience building metadata-driven data frameworks, schema inference/profiling, lineage, and configuration-driven engineering.
• Experience with Azure DevOps, Terraform, CI/CD, and automated testing.
• Strong understanding of enterprise security, governance, PII protection, and responsible AI.
• Recent production coding and code-review experience.

Preferred Qualifications

• Experience in financial services, banking, insurance, accounting, or actuarial data.
• Knowledge of RDF/SPARQL, Neo4j, knowledge graphs, ontologies, or property graphs.
• Experience with enterprise text-to-SQL and semantic-layer solutions.
• Experience with ML-based anomaly detection for transactional or time-series data.
• LLMOps/MLOps experience, including model/prompt versioning, evaluation, cost governance, and observability.
• Experience with dbt, Great Expectations, or similar data-quality tools.
• Exposure to Workday, Workday Prism, or Accounting Center.
• Databricks Data Engineer Professional, Azure DP-203/DP-700, or AZ-305 certification.

What We’re Looking For

We are looking for a senior engineer/architect who can design it, build it, review it, and lead others through it. This is not a purely architectural or management position.

Candidates should be able to demonstrate recent hands-on delivery of complex Azure + Databricks data platforms and production AI/LLM solutions.

Applicants who do not meet the core Azure, Databricks, hands-on data engineering, and production GenAI requirements may not be considered., * data engineering, data platforms, architecture: 10 years (Preferred)

• hands-on production experience with Azure and Databricks: 5 years (Preferred)
• Expert Python, PySpark/Spark & SQL: 10 years (Preferred)
• Hands-on LLM/GenAI & RAG production: 4 years (Preferred)

Work Location: Hybrid remote in Cary, NC 27513