> Markdown version of [/jobs/ext/3691091-sr-ai-data-engineer-databricks](https://www.wearedevelopers.com/jobs/ext/3691091-sr-ai-data-engineer-databricks). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Sr. AI & Data Engineer - Databricks

- **Company:** Accenture
- **Location:** Columbus, OH, United States
- **Experience:** Expert
- **Salary:** $70,350.0 - $205,800.0
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Microsoft Azure, Cloud Computing, Continuous Integration, Data as a Services, Data Architecture, Information Engineering, Data Integration, Data Transformation, Github, Python (Programming Language), Performance Tuning, Search Technologies, SQL Databases, Data Storage Technologies, Azure Data Factory, Retrieval-Augmented Generation, Large Language Models, Multi-Agent Systems, Apache Spark, Agentic-AI, Git, Data Lakes, Pyspark, Data Analytics, AutoGen, Machine Learning Operations, Virtual Agents, Data Delivery, Software Version Control, Data Pipelines, Databricks
- **Published:** October 11, 2026
- **Apply:** https://www.columbusjobsite.com/job.asp?id=3435011506&tx=JT989TYI&pt=1&aff=0B19D771-A501-4A5E-8338-2A822B784D54&utm_source=Job%20Feed&utm_medium=textkernel&utm_campaign=DE&utm_term=0B19D771-A501-4A5E-8338-2A822B784D54

## About the Role

* A minimum of three years of professional experience as a Data & AI Engineer or in a related role.- A minimum of three years of experience working with Microsoft Azure cloud data services.- A minimum of two years experience with Databricks (PySpark, SQL, Delta Lake, workflows).- Bachelor's degree or equivalent (minimum 12 years) work experience. (If Associate Degree, must have minimum 6 years work experience), * Advanced Python (PySpark) and SQL for data transformation and pipeline development.- Experience with Azure Databricks: Spark, Delta Lake, Auto Loader, Lakehouse Medallion Architecture (Bronze, Silver, Gold).- Experience with Git-based version control and CI/CD pipelines (Azure DevOps or GitHub Actions).- Experience in agent development using Mosaic AI Agent Framework.- Strong problem-solving and performance optimization skills.
 * Experience in MLOps / LLMOps (CI/CD for ML and LLM lifecycle).- Genie product family.- Databricks Certified Data Engineer Professional (2025).- Microsoft Certified: Azure Data Engineer Associate.

## Description

An experienced, hands-on Data & AI Engineer with deep hands-on expertise in Databricks and Azure data services to design, build, and optimize scalable data pipelines. You design and implement autonomous agents using Databricks Mosaic AI Agent Framework and frameworks like LangChain or AutoGen. You'll work closely with analytics, data science, and cloud teams to ensure reliable, high-performance data delivery for the clients we serve., There will never be a typical day and that's why people love it here. The opportunities to make a difference within exciting client initiatives are unlimited in the ever-changing technology landscape. You will be part of a highly collaborative and growing network of technology and data experts, who are taking on today's biggest, most complex business challenges using the latest data and analytics technologies. We will nurture your talent in an inclusive culture that values diversity. You will have an opportunity to work in a variety of roles covering all aspects of Data Engineering including Data Modelling, Data Integration, Data Analytics, Pipeline Development, Agentic AI, RAG, Cloud and Infrastructure integration, Data Quality, Observability & Governance., * Architect, develop, and maintain scalable, low-latency data pipelines to ingest and transform diverse datasets leveraging Azure Databricks workflows and Spark technologies.- Design and optimize Lakehouse architecture using Delta Lake tables following Bronze, Silver, and Gold layers to ensure efficient, reliable, and governed data storage.- Partner closely with data scientists and business analysts to deliver clean, validated, and well-modeled datasets that support advanced analytics and AI initiatives.- Design and implement autonomous agents using Databricks Mosaic AI Agent Framework and frameworks like LangChain or AutoGen.- Build and optimize vector search indexes using Databricks Vector Search for accurate context retrieval.- Process large-scale distributed datasets on the Databricks Intelligence Platform with Apache Spark.- Design and implement data models that meet business requirements and enhance data usability.- Continuously evaluate and improve data architecture processes to
enhance efficiency and effectiveness.- Design AI agents with retrieval, orchestration, tool invocation, evaluation harnesses, and lifecycle observability.- Continuously monitor pipeline health and performance; implement automated alerting and observability frameworks to facilitate rapid troubleshooting and proactive optimization.

This is a hybrid role and will require three days per week in the office. Hiring locations: Dallas, TX; Houston, TX; Columbus, OH; or Tampa, FL.