> Markdown version of [/jobs/ext/3112910-data-engineer-databricks-spark-ai](https://www.wearedevelopers.com/jobs/ext/3112910-data-engineer-databricks-spark-ai). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Engineer Databricks / Spark / AI - **Company:** Innova Software Services Inc. - **Location:** San Francisco, CA, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Agile Methodology, Artificial Intelligence, Business Analytics Applications, Data Analysis, JIRA, Big Data, Code Review, Databases, Information Engineering, Data Files, Extract Transform Load (ETL), Data Transformation, Distributed Data Store, Python (Programming Language), Performance Tuning, Scrum Methodology, Systems Development Life Cycle, Requirements Management, Software Engineering, SQL Databases, Enterprise Data Management, Data Processing, Scripting, Freeform SQL, Apache Spark, Generative AI, Pyspark, Atlassian Tools, Data Analytics, Data Management, Data Pipelines, Databricks - **Published:** September 27, 2026 - **Apply:** https://www.careerbuilder.com/job-details/data-engineer-databricks-spark-ai-san-francisco-ca--f959a226-26d7-4d5a-a5f8-4e7b4f9f86fb ## About the Role We are seeking a highly skilled Senior Data Engineer with strong, recent hands-on experience in Databricks, Apache Spark, SQL, and Python. This is primarily a Data Engineering role with additional exposure to AI and Generative AI capabilities within the Databricks ecosystem. The ideal candidate will have extensive experience designing and developing scalable data pipelines and data processing solutions using Databricks and Spark. Candidates should also have hands-on experience with Databricks Genie and an understanding of how AI capabilities can be applied to enterprise data and analytics solutions. Strong, recent Databricks experience is essential for this position., Agile Programming Methodologies, Apache Spark, Artificial Intelligence (AI), Atlassian JIRA, Code Reviews, Data Analysis, Data Management, Data Processing, Data Sets, Database Extract Transform and Load (ETL), Ecosystems, Identify Issues, Performance Tuning/Optimization, Production Support, Python Programming/Scripting Language, Requirements Management, SQL (Structured Query Language), Scalable System Development, Scrum Project Management and Software Development, Technical/Engineering Design, Testing, Validation Testing, Workflow Analysis ## Description * Design, develop, and maintain scalable data engineering solutions using Databricks, Apache Spark, SQL, and Python. * Build and optimize high-volume ETL/ELT data pipelines and data-processing workflows. * Develop complex SQL queries and transformations for large-scale datasets. * Build distributed data-processing solutions using PySpark/Spark. * Design reliable and scalable data architectures within the Databricks ecosystem. * Work hands-on with Databricks Genie to enable AI-powered conversational data and analytics capabilities. * Integrate AI/GenAI capabilities with enterprise data platforms and analytics workflows. * Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency. * Perform data transformation, cleansing, validation, and quality checks. * Troubleshoot performance issues across Spark jobs, SQL workloads, pipelines, and Databricks environments. * Work with structured and semi-structured datasets from multiple enterprise sources. * Collaborate with Data Engineering, Analytics, AI/ML, Product, and business teams. * Translate business and analytical requirements into scalable data solutions. * Participate in technical design, code reviews, testing, deployment, and production support. * Maintain engineering standards, documentation, and data-development best practices. * Participate in Agile/Scrum development activities using Jira and Confluence. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Improving quality with Agentic AI with Rovo Dev and Xray](https://www.wearedevelopers.com/videos/2005-improving-quality-with-agentic-ai-with-rovo-dev-and-xray) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [What Industries Outside of AI Are Hiring The Most AI Experts?](https://www.wearedevelopers.com/magazine/98-what-industries-outside-of-ai-are-hiring-the-most-ai-experts) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)