Senior Databricks & AI Engineer [W2 ROLE]-(Onsite - 4 Days/Week)

We are seeking a Senior Databricks & AI Engineer to optimize and enhance an enterprise AI platform powered by Azure Databricks and Databricks Genie. The primary focus will be improving chatbot response times, resolving performance bottlenecks, optimizing data pipelines, and supporting GenAI capabilities. The ideal candidate will have strong hands-on experience with Databricks performance tuning, Spark optimization, Data Engineering, DataOps, Azure cloud services, API development, and GenAI solutions. Responsibilities

• Optimize Databricks architecture, Spark jobs, SQL queries, and cluster performance.
• Identify and resolve latency and performance issues impacting AI chatbot response times.
• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Implement DataOps, CI/CD, automation, monitoring, and deployment best practices.
• Support Databricks Genie, GenAI applications, LLM integrations, RAG, and AI agents.
• Integrate Azure services and RESTful APIs with Databricks and enterprise applications.
• Support onboarding of new datasets and data sources.
• Collaborate with application development and architecture teams.
• Contribute to enterprise AI platform scalability and long-term technology initiatives.

• 5+ years of experience in Data Engineering, Analytics Engineering, or related roles.
• Advanced hands-on experience with Azure Databricks performance optimization.
• Strong knowledge of Apache Spark, SQL tuning, cluster management, and troubleshooting.
• Experience developing scalable data pipelines and ETL/ELT workflows.
• Experience with DataOps, CI/CD, automation, and monitoring.
• Hands-on experience with Databricks Genie and AI-enabled analytics.
• Strong Microsoft Azure cloud services and architecture experience.
• Experience developing RESTful APIs and data integration services.
• Experience building or supporting AI/GenAI solutions, LLMs, or RAG applications.
• Strong proficiency in Python and SQL.
• Excellent problem-solving and performance tuning skills.

Preferred Qualifications

• Experience developing AI agents and autonomous workflows.
• Angular or full-stack development experience.
• Knowledge of Google Cloud Platform (GCP) and AI services.
• Experience with conversational AI, semantic search, and vector databases.
• Experience integrating AI solutions into enterprise applications.

Ideal Candidate Candidates with strong Databricks performance optimization experience, combined with GenAI/Genie expertise, are highly preferred. ML Engineers with extensive Databricks experience and Data Engineers with production AI platform optimization experience are encouraged to apply. Work Arrangement: Onsite 4 days per week in Atlanta, GA.

• $105,000-160,000 per year Cargill’s size and scale allows us to make a positive impact in the world. Our purpose is to nourish the world in a safe, responsible and sustainable way. We are a family company p…

• 17 days ago +

Cargill is committed to providing food and agricultural solutions to nourish the world in a safe, responsible, and sustainable way. Sitting at the heart of the supply chain, we par…