AI Engineer - Databricks / LLM Systems / AgentOps

• Design and deliver end-to-end AI pipelines, covering ingestion, transformation, retrieval, model inference, and output generation using Databricks-native capabilities (Lakeflow, Delta Tables, Workflows)
• Develop LLM-based applications, focusing on prompt engineering, response optimization, and integration into enterprise workflows
• Implement retrieval-augmented generation (RAG) architectures to enhance reliability, grounding, and enterprise search use cases
• Build AI solutions directly on Databricks using PySpark, Python, and Databricks AI/ML & AI capabilities
• Integrate structured and unstructured data within a Databricks Lakehouse architecture, leveraging Delta Lake for scalable and governed data processing
• Apply hallucination prevention techniques, including grounding strategies, validation layers, and structured retrieval approaches
• Establish AI guardrails and monitoring frameworks, ensuring governance, data security, and responsible AI practices
• Work with Databricks Asset Bundles and Workflows to operationalize and manage AI/data pipelines
• Deploy AI solutions into production with robust monitoring, logging, and performance optimization within Databricks environments
• Integrate AI outputs into downstream applications, dashboards, and enterprise systems
• Optimize cost and performance of Databricks compute workloads through efficient data access, caching, and pipeline design
• Build and maintain CI/CD pipelines using Azure DevOps, enabling automated deployment, testing, and version control of data and AI assets

• Strong experience with Python and PySpark for AI pipeline development and large-scale data processing
• Advanced proficiency in SQL and Delta Lake for data transformation and modeling within Databricks
• Hands-on experience with Databricks platform (Lakehouse, Delta Tables, Workflows, Asset Bundles)
• Experience developing LLM/GenAI applications and RAG architectures, including grounding and hallucination mitigation
• Familiarity with frameworks such as LangChain, LangGraph, or similar agentic AI frameworks
• Experience implementing CI/CD pipelines using Azure DevOps
• Proven experience deploying AI solutions to production, including monitoring, logging, and performance optimization

Good to Have

• Exposure to Databricks AI / MLflow / Feature Store
• Experience with agent-based (AgentOps) architectures and LLM orchestration
• Knowledge of Databricks Apps and end-to-end data + AI pipeline integration
• Exposure to DataOps / MLOps practices in Databricks ecosystem