> Markdown version of [/jobs/ext/457231-databricks-expert-data-lakehouse-consultant](https://www.wearedevelopers.com/jobs/ext/457231-databricks-expert-data-lakehouse-consultant). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Databricks Expert / Data Lakehouse Consultant - **Company:** BILINK CORP. - **Location:** Chicago, IL, United States (Remote available) - **Experience:** Expert - **Salary:** $110,000.0 - $160,000.0 - **Contract:** Permanent contract - **Skills:** Unity 3d, SAP Cloud, Application Programming Interfaces (APIs), Agile Methodology, Artificial Intelligence, Airflow, Amazon Web Services, Amazon S3, Data Analysis, Business Logic, Computing Platforms, Microsoft Azure, Cloud Computing, Cloud Database, Cloud Storage, Continuous Integration, Data Validation, Data Discovery, Information Engineering, Data Infrastructure, Extract Transform Load (ETL), Data Vault Modeling, Data Warehousing, Software Design Patterns, Github, Apache Hive, Python (Programming Language), Query Optimization, Power BI, Standard Sql, Azure Data Lake, SAP (Applications), SAP HANA, SAP NetWeaver Data Management, Data Streaming, Tableau (Software), Azure Data Factory, Apache Spark, SAP Business Technology Platform, Data Layers, Build Management, Data Lakes, Pyspark, Git Flow, Star Schema, SAP S/4HANA, AWS Data Analytics, Machine Learning Operations, Data Lakehouse, Azure Synapse Analytics, Data Pipelines, Databricks - **Published:** June 6, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=6ef7670a3dacf51f ## About the Role Do you have experience in Spark?, * 5+ years of data engineering experience * Strong hands-on experience with Databricks (notebooks, jobs, workspace administration) * Proficiency in PySpark / Spark SQL and Python (data engineering patterns, testing, packaging) * Delta Lake in production: ACID transactions, time travel, schema management, MERGE/CDC patterns * Experience with Medallion Architecture and lakehouse design patterns * Working knowledge of Unity Catalog (governance, permissions, lineage) * Experience with cloud storage and integration (Azure ADLS / AWS S3) * Solid SQL skills for data modeling and query optimization * Excellent communication skills (client-facing role) Nice-to-Have * Delta Live Tables (DLT) pipelines - declarative, streaming, and batch * Databricks Asset Bundles for CI/CD across multi-environment deployments (dev / QA / prod) * Databricks Account Administration (SCIM / IdP integration, Unity Catalog metastore management, workspace provisioning) * Databricks GenAI features: Genie AI/BI and Agent Bricks * Background in financial data, supply chain, or CPG data domains * Apache Airflow or Azure Data Factory orchestration; dbt for transformation * Great Expectations or similar DQ frameworks; MLflow and Databricks Feature Store * Databricks certifications (Data Engineer Associate/Professional, Spark Developer) * Familiarity with SAP data extraction patterns (SAP HANA, CDS views, ODP) - a strong plus Profiles We Are Looking For * Hands-on data engineer (not just functional or purely architectural) * Comfortable working across Data Engineering, Analytics, and IT * Curious, structured, and quality-driven * Able to challenge requirements and propose better solutions * Interested in growing toward Solution Architect or Lead Data Engineer roles ## Description We help clients move from legacy reporting to future-proof analytics and planning architectures, leveraging SAP S/4HANA, SAP Analytics Cloud, SAP Datasphere, and SAP BTP as well as modernize their data infrastructure by migrating from legacy data warehouses and ETL pipelines to scalable, cloud-native Lakehouse architectures powered by Databricks, Delta Lake, and the broader Azure/AWS data ecosystem., We are seeking an experienced Databricks Expert to lead and support enterprise data lakehouse initiatives. This role spans data engineering, data modeling, and platform architecture across Databricks, Delta Lake, Unity Catalog, and cloud data ecosystems (Azure / AWS). You will work closely with data engineering, analytics, and business stakeholders to design scalable ingestion pipelines, curated data layers, and consumption-ready datasets - delivering solutions aligned with enterprise architecture and governance standards. This role requires a strong technical foundation in Spark and Python-based data engineering, combined with a solid understanding of data modeling and lakehouse design patterns., Data Engineering & Pipeline Development (Primary) * Design and build scalable data ingestion pipelines using Databricks (Spark, PySpark, Spark SQL) * Implement Medallion Architecture (Bronze / Silver / Gold) patterns for raw, curated, and consumption layers * Develop and optimize Delta Lake tables: * * Schema evolution and enforcement* MERGE / UPSERT patterns for CDC and incremental loads* Z-Ordering, compaction, and vacuuming for performance * Build and orchestrate workflows using Databricks Workflows / Apache Airflow / Azure Data Factory * Implement data quality checks and validation frameworks (e.g., Great Expectations, custom DQ layers)Data Modeling & Lakehouse Design * Design semantic and analytical data models (Star Schema, Data Vault, OBT) for BI and ML consumption * Implement and govern Unity Catalog for data discovery, lineage, and access control * Model business logic: KPI frameworks (actuals vs. plan vs. forecast), Slowly Changing Dimensions (SCD Type 1/2), aggregated fact tables and pre-computed summary layers * Ensure semantic consistency and reusability across data consumers (BI, ML, APIs)Platform & Infrastructure * Configure and manage Databricks workspaces, clusters, and job compute * Implement cost optimization strategies (auto-scaling, spot instances, cluster policies) * Support CI/CD for data pipelines using Git-based workflows (GitHub / Azure DevOps) and Databricks Asset Bundles or dbx * Work with cloud infrastructure across Azure (ADLS Gen2, Azure Data Factory, Synapse) and AWS (S3, Glue, Redshift) Analytics & ML Enablement * Prepare and expose datasets for BI tools (Power BI, Tableau, SAP Analytics Cloud) * Support MLflow for experiment tracking and model registry * Collaborate with data scientists and analysts to ensure data readiness for advanced analytics * Design Feature Store tables and serve them for ML pipelines where applicable Client & Project Delivery * Gather business and data requirements and translate them into Databricks platform solutions * Work in Agile or hybrid delivery models * Support UAT, training, go-live, and post-go-live enhancements * Collaborate closely with Bilink architects and client stakeholders ## Related Videos - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering) - [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)