> Markdown version of [/jobs/ext/2266631-databricks-sme](https://www.wearedevelopers.com/jobs/ext/2266631-databricks-sme). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Databricks SME - **Company:** August Schell - **Location:** United States - **Experience:** Experienced - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Artificial Intelligence, Airflow, Apache HTTP Server, Data Compression, Data Infrastructure, Data Integration, Extract Transform Load (ETL), Data Transformation, Data Security, Data Stores, Python (Programming Language), NoSQL, Performance Tuning, SQL Databases, Data Streaming, Data Processing, Data Storage Management, Snowflake, Apache Spark, Indexer, Apache Flink, Data Analytics, Integration Frameworks, Apache Kafka, Data Management, Software Coding, Stream Processing, Data Pipelines, Databricks - **Published:** August 27, 2026 - **Apply:** https://www.clearancejobs.com/jobs/9120115/databricks-sme ## About the Role 5+ years' experience as a Data Engineer with a deep understanding of data management, ETL pipelines and streaming data. 5+ years' experience writing code using Python, Scala and/or Java. Consultant-level, expert experience in Databricks. Apache Spark experience (mandatory). 2+ years' expertise in Databricks, Apache Spark, Snowflake and/or similar data processing frameworks. BA / BS degree with 4+ years of experience (or) MS degree with 2+ years of experience as a data engineer. Stand Out With... Experience with data streaming technologies (e.g., Apache Kafka, Apache Flink) and real-time data processing. Familiarity with AI and machine learning concepts and the ability to work with data scientists and AI engineers. Experience working on SQL or NoSQL data stores. Proficiency in tools like Cribl, Airflow or similar for data pipeline orchestration and transformation. Knowledge of data security and compliance principles. ## Description We are looking for a highly skilled Data Engineer with expert-level knowledge to assume a key role in managing data, both at rest and in streaming, and to lead efforts in data management and optimization. Your proficiency in ETL pipeline development, data tools and your deep understanding of data management principles will be instrumental in ensuring our data infrastructure operates efficiently. What you will do... Data Management Leadership: Provide technical leadership in data management, overseeing data at rest and data streaming, and guiding the team. ETL Pipeline Development: Design, develop, and maintain efficient ETL pipelines to process and transform data, ensuring data quality, integrity, and consistency. Databricks Expertise: Optimize data processing by using your expertise in Databricks, develop Spark-based solutions, and enhance data integration and analytics capabilities. Streaming Data: Manage streaming data sources and implement real-Ime data processing solutions using tools like Apache Kana or similar technologies. Data Transformation: Implement and manage data routing, transformation, and enrichment using Cribl or similar data pipeline orchestration tools. Data Optimization: Work on data optimization initiatives, including performance tuning, indexing, and data compression, to ensure efficient data storage and retrieval. Data Security and Compliance: Collaborate with security teams to ensure data security and compliance with data privacy regulations. Documentation: Build comprehensive documentation for data management processes, ETL pipelines, and facilitating knowledge transfer within the team. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Leveraging Real time data in FSIs](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis) - [Optimizing Discovery: PostgreSQL's Role in Transforming GetYourGuide's Search](https://www.wearedevelopers.com/videos/1647-optimizing-discovery-postgresql-s-role-in-transforming-getyourguide-s-search) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Dynamic Entities in .NET: Building Low-Code Systems on Top of Entity Framework Core](https://www.wearedevelopers.com/videos/100218-dynamic-entities-in-net-building-low-code-systems-on-top-of-entity-framework-core) - [AI Model Management Life Circles: ML Ops For Generative AI Models From Research to Deployment](https://www.wearedevelopers.com/videos/1152-ai-model-management-life-circles-ml-ops-for-generative-ai-models-from-research-to-deployment) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift)