Databricks Expert / Data Lakehouse Consultant
Role details
Job location
Tech stack
Job description
We help clients move from legacy reporting to future-proof analytics and planning architectures, leveraging SAP S/4HANA, SAP Analytics Cloud, SAP Datasphere, and SAP BTP as well as modernize their data infrastructure by migrating from legacy data warehouses and ETL pipelines to scalable, cloud-native Lakehouse architectures powered by Databricks, Delta Lake, and the broader Azure/AWS data ecosystem., We are seeking an experienced Databricks Expert to lead and support enterprise data lakehouse initiatives. This role spans data engineering, data modeling, and platform architecture across Databricks, Delta Lake, Unity Catalog, and cloud data ecosystems (Azure / AWS).
You will work closely with data engineering, analytics, and business stakeholders to design scalable ingestion pipelines, curated data layers, and consumption-ready datasets - delivering solutions aligned with enterprise architecture and governance standards.
This role requires a strong technical foundation in Spark and Python-based data engineering, combined with a solid understanding of data modeling and lakehouse design patterns., Data Engineering & Pipeline Development (Primary)
- Design and build scalable data ingestion pipelines using Databricks (Spark, PySpark, Spark SQL)
- Implement Medallion Architecture (Bronze / Silver / Gold) patterns for raw, curated, and consumption layers
- Develop and optimize Delta Lake tables:
-
- Schema evolution and enforcement* MERGE / UPSERT patterns for CDC and incremental loads* Z-Ordering, compaction, and vacuuming for performance
- Build and orchestrate workflows using Databricks Workflows / Apache Airflow / Azure Data Factory
- Implement data quality checks and validation frameworks (e.g., Great Expectations, custom DQ layers)Data Modeling & Lakehouse Design
- Design semantic and analytical data models (Star Schema, Data Vault, OBT) for BI and ML consumption
- Implement and govern Unity Catalog for data discovery, lineage, and access control
- Model business logic: KPI frameworks (actuals vs. plan vs. forecast), Slowly Changing Dimensions (SCD Type 1/2), aggregated fact tables and pre-computed summary layers
- Ensure semantic consistency and reusability across data consumers (BI, ML, APIs)Platform & Infrastructure
- Configure and manage Databricks workspaces, clusters, and job compute
- Implement cost optimization strategies (auto-scaling, spot instances, cluster policies)
- Support CI/CD for data pipelines using Git-based workflows (GitHub / Azure DevOps) and Databricks Asset Bundles or dbx
- Work with cloud infrastructure across Azure (ADLS Gen2, Azure Data Factory, Synapse) and AWS (S3, Glue, Redshift)
Analytics & ML Enablement
- Prepare and expose datasets for BI tools (Power BI, Tableau, SAP Analytics Cloud)
- Support MLflow for experiment tracking and model registry
- Collaborate with data scientists and analysts to ensure data readiness for advanced analytics
- Design Feature Store tables and serve them for ML pipelines where applicable
Client & Project Delivery
- Gather business and data requirements and translate them into Databricks platform solutions
- Work in Agile or hybrid delivery models
- Support UAT, training, go-live, and post-go-live enhancements
- Collaborate closely with Bilink architects and client stakeholders
Requirements
Do you have experience in Spark?, * 5+ years of data engineering experience
- Strong hands-on experience with Databricks (notebooks, jobs, workspace administration)
- Proficiency in PySpark / Spark SQL and Python (data engineering patterns, testing, packaging)
- Delta Lake in production: ACID transactions, time travel, schema management, MERGE/CDC patterns
- Experience with Medallion Architecture and lakehouse design patterns
- Working knowledge of Unity Catalog (governance, permissions, lineage)
- Experience with cloud storage and integration (Azure ADLS / AWS S3)
- Solid SQL skills for data modeling and query optimization
- Excellent communication skills (client-facing role)
Nice-to-Have
- Delta Live Tables (DLT) pipelines - declarative, streaming, and batch
- Databricks Asset Bundles for CI/CD across multi-environment deployments (dev / QA / prod)
- Databricks Account Administration (SCIM / IdP integration, Unity Catalog metastore management, workspace provisioning)
- Databricks GenAI features: Genie AI/BI and Agent Bricks
- Background in financial data, supply chain, or CPG data domains
- Apache Airflow or Azure Data Factory orchestration; dbt for transformation
- Great Expectations or similar DQ frameworks; MLflow and Databricks Feature Store
- Databricks certifications (Data Engineer Associate/Professional, Spark Developer)
- Familiarity with SAP data extraction patterns (SAP HANA, CDS views, ODP) - a strong plus
Profiles We Are Looking For
- Hands-on data engineer (not just functional or purely architectural)
- Comfortable working across Data Engineering, Analytics, and IT
- Curious, structured, and quality-driven
- Able to challenge requirements and propose better solutions
- Interested in growing toward Solution Architect or Lead Data Engineer roles
Benefits & conditions
515 North State Street, Chicago, IL 60654 Hybrid work $110,000 - $160,000 a year - Full-time, Contract, Pulled from the full job description
- 401(k)
- Health insurance
- 401(k) matching
- Paid time off
- Dental insurance, * 401(k) with matching
- Health and dental insurance
- Paid time off
Bilink Corp. is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status.
Note: We prefer to hire directly - applications via staffing agencies or C2C arrangements may be considered with a lower priority.
Job Types: Full-time, Contract
Pay: $110,000.00 - $160,000.00 per year, * 401(k)
- 401(k) matching
- Dental insurance
- Health insurance
- Paid time off
Application Question(s):
- How many years of hands-on Databricks (Spark/PySpark) experience do you have?
- Have you built and maintained Delta Lake tables in a production environment (MERGE/CDC, schema evolution)?
- Have you worked hands-on with Unity Catalog (permissions, lineage, governance)?
- This role is hybrid in Chicago, IL (3 days/week onsite). Can you reliably work onsite 3 days per week?
- Are you applying as an individual candidate (not through a staffing agency or C2C arrangement)?
- Will you now or in the future require visa sponsorship? If yes, what is your current status?
- What is your expected annual salary (or expected W2 hourly rate, if contract)?
- Briefly describe the largest Databricks/lakehouse project you delivered: your role, scope, and approximate data volume. (3-5 sentences)