Senior Data Engineer / ML Platform Engineer (Azure Databricks & GenAI)

We are augmenting an internal team with contract contractors who have strong Azure Databricks and ML data platform skills. Our staff already owns platform standards, Unity Catalog governance, and architecture; we need additional capacity to deliver lakehouse pipelines, catalogued engineering and enterprise datasets, and GenAI-ready data products (embeddings, retrieval, batch enrichment) in parallel with other priorities. Assignments may span multiple programs-not a single fixed product-so candidates should be comfortable ramping on new domains and codebases quickly. Responsibilities

• Design and build production data pipelines on Databricks (PySpark/SQL, Delta Lake, Unity Catalog, Workflows/Jobs) with idempotency, testing, monitoring, and clear lineage.
• Model bronze / silver / gold layers and data contracts (schemas, SLAs, quality checks) for analytics and downstream applications.

• Develop ML-oriented data products: feature stores or curated tables, embedding generation, chunking/indexing for RAG, batch classification/enrichment, and evaluation of retrieval/data quality.

• Integrate with Azure services (ADLS Gen2, Key Vault, managed identities, private connectivity) following enterprise security and compliance patterns.

• Partner with application and ML teams to expose stable APIs, tables, and volumes that power agentic workflows (tool-backed assistants, multi-step orchestration) without owning full UI or real-time serving unless scoped.

• Document data dictionaries, runbooks, and handoff materials; participate in code review and agile ceremonies with full-time staff.

• 5+ years in data engineering, including 2+ years production Databricks on Azure.
• Strong Python (PySpark, pandas); SQL; experience operationalizing jobs at scale.
• Hands-on Delta Lake and Unity Catalog (schemas, grants, environments, promotion patterns).
• Experience building pipelines that feed ML/GenAI use cases: embeddings, vector or hybrid search inputs, labeled datasets, batch inference outputs.
• Git, CI-friendly repo layout (notebooks + packages), unit/integration testing for data pipelines.

• Strong communication; effective in staff augmentation models (embedded with internal leads, shared standards). Preferred qualifications

• GenAI / agentic systems: RAG design, retrieval evaluation, metadata for grounding, guardrails-aware data prep.
• Multi-source integration: documents, PLM/BOM-style structured data, files in object storage, APIs, spreadsheets-normalized into a catalog.

• Exposure to containerized or microservice backends (for collaboration with app teams, not primary ownership). Nice to have (role-dependent)

• MLOps on Databricks (Model Serving, Feature Serving, experiment tracking).
• Stream or near-real-time ingestion (Structured Streaming, event-driven triggers).
• Infrastructure-as-code literacy (Terraform/Bicep)-often owned internally; ability to consume outputs is enough.
• Graph or highly relational domains (requirements traceability, product structure, hierarchies).
• Automotive or complex manufacturing data (helpful, not required).

• Platform expertise, Databricks/Azure standards, architecture review, and access to tooling and repos per assignment.
• Clear priorities from product/engineering owners; expectations to align with existing team patterns rather than invent a separate stack.