> Markdown version of [/jobs/ext/2978129-databricks-ai-engineer](https://www.wearedevelopers.com/jobs/ext/2978129-databricks-ai-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Databricks AI Engineer - **Company:** IT America - **Location:** Austin, TX, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Automation of Tests, Microsoft Azure, Code Generation, Software Quality, Code Review, Computer Programming, Continuous Integration, Data Validation, Information Engineering, Distributed Computing Environment, Github, Python (Programming Language), Meta-Data Management, Cloud Services, Azure DevOps Pipelines, Software Engineering, SQL Databases, Data Logging, Large Language Models, Multi-Agent Systems, Prompt Engineering, Generative AI, Git, Data Lakes, Pyspark, Information Technology, Data Lineage, Data Management, Virtual Agents, Code Restructuring, Software Version Control, Data Pipelines, Databricks - **Published:** September 18, 2026 - **Apply:** https://www.dice.com/job-detail/58f9bbbe-4c83-4e2e-9304-501d323d96e2 ## About the Role * Bachelor's degree in Computer Science, Software Engineering, Data Science, or a related technical discipline. Master's degree is a plus. * 10+ years of experience in data engineering, software engineering, cloud data platforms, or related technical roles. * Strong hands-on experience with Azure Databricks and modern cloud data platforms. * Proven experience developing AI/LLM-powered automation, intelligent agents, or Generative AI solutions. * Strong programming skills in Python and SQL with experience developing production-quality applications. * Strong hands-on experience with PySpark and distributed data processing. * Experience with Databricks notebooks, Jobs/Workflows, Delta Lake, and Databricks Asset Bundles. * Experience with LLM APIs, prompt engineering, agent orchestration, and AI frameworks. * Experience with technologies/frameworks such as Azure OpenAI, OpenAI, LangChain, AutoGen, Databricks Agent Framework/Agent Bricks, or comparable agentic AI technologies. * Hands-on experience with CI/CD and automation, including GitHub Actions and/or Azure DevOps Pipelines. * Experience building data quality, testing, QA automation, or data validation frameworks. * Strong understanding of software engineering principles including Git, testing, version control, documentation, code quality, and observability. ## Description Note: The ideal candidate is not simply a traditional Databricks Data Engineer. We are looking for someone who can combine: Data Engineering + Azure Databricks + Python/PySpark + Generative AI + LLMs + AI Agents + Automation, * Design, develop, and deploy intelligent AI agents that automate data engineering workflows within Azure Databricks. * Build LLM-powered agents for code generation, code review, refactoring, optimization, and troubleshooting of Python, PySpark, and SQL workloads. * Develop AI-driven solutions to automate data quality validation, testing, reconciliation, and QA processes. * Build agents to support metadata management, data lineage, governance, and compliance automation within Databricks. * Develop intelligent automation for CI/CD pipelines, including test-case generation, deployment validation, release checks, and rollback processes. * Implement AI-powered monitoring and automation for data pipeline failures, anomalies, and root-cause analysis. * Develop reusable prompt engineering frameworks, agent patterns, tools, and orchestration workflows for data engineering use cases. * Integrate LLMs and AI agents with Databricks, Azure services, APIs, data platforms, and enterprise workflows. * Work with Data Engineers, Data Architects, QA teams, and Platform teams to identify high-value automation opportunities. * Establish standards and best practices for AI agent development, testing, deployment, observability, security, and governance. * Build production-grade automation solutions with appropriate logging, error handling, monitoring, testing, and documentation. * Evaluate emerging Generative AI, agentic AI, and automation frameworks and recommend technologies that provide measurable business or engineering value. * Document agent architecture, workflows, prompts, decision logic, integration patterns, and operational runbooks. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Agentic AI - From Theory to Practice: Developing Multi-Agent AI Systems on Azure](https://www.wearedevelopers.com/videos/1532-agentic-ai-from-theory-to-practice-developing-multi-agent-ai-systems-on-azure) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)