Databricks AI Engineer

Note: The ideal candidate is not simply a traditional Databricks Data Engineer. We are looking for someone who can combine: Data Engineering + Azure Databricks + Python/PySpark + Generative AI + LLMs + AI Agents + Automation, * Design, develop, and deploy intelligent AI agents that automate data engineering workflows within Azure Databricks.

• Build LLM-powered agents for code generation, code review, refactoring, optimization, and troubleshooting of Python, PySpark, and SQL workloads.
• Develop AI-driven solutions to automate data quality validation, testing, reconciliation, and QA processes.
• Build agents to support metadata management, data lineage, governance, and compliance automation within Databricks.
• Develop intelligent automation for CI/CD pipelines, including test-case generation, deployment validation, release checks, and rollback processes.
• Implement AI-powered monitoring and automation for data pipeline failures, anomalies, and root-cause analysis.
• Develop reusable prompt engineering frameworks, agent patterns, tools, and orchestration workflows for data engineering use cases.
• Integrate LLMs and AI agents with Databricks, Azure services, APIs, data platforms, and enterprise workflows.
• Work with Data Engineers, Data Architects, QA teams, and Platform teams to identify high-value automation opportunities.
• Establish standards and best practices for AI agent development, testing, deployment, observability, security, and governance.
• Build production-grade automation solutions with appropriate logging, error handling, monitoring, testing, and documentation.
• Evaluate emerging Generative AI, agentic AI, and automation frameworks and recommend technologies that provide measurable business or engineering value.
• Document agent architecture, workflows, prompts, decision logic, integration patterns, and operational runbooks.

• Bachelor’s degree in Computer Science, Software Engineering, Data Science, or a related technical discipline. Master’s degree is a plus.
• 10+ years of experience in data engineering, software engineering, cloud data platforms, or related technical roles.
• Strong hands-on experience with Azure Databricks and modern cloud data platforms.
• Proven experience developing AI/LLM-powered automation, intelligent agents, or Generative AI solutions.
• Strong programming skills in Python and SQL with experience developing production-quality applications.
• Strong hands-on experience with PySpark and distributed data processing.
• Experience with Databricks notebooks, Jobs/Workflows, Delta Lake, and Databricks Asset Bundles.
• Experience with LLM APIs, prompt engineering, agent orchestration, and AI frameworks.
• Experience with technologies/frameworks such as Azure OpenAI, OpenAI, LangChain, AutoGen, Databricks Agent Framework/Agent Bricks, or comparable agentic AI technologies.
• Hands-on experience with CI/CD and automation, including GitHub Actions and/or Azure DevOps Pipelines.
• Experience building data quality, testing, QA automation, or data validation frameworks.
• Strong understanding of software engineering principles including Git, testing, version control, documentation, code quality, and observability.