> Markdown version of [/jobs/ext/2049570-databricks-engineer](https://www.wearedevelopers.com/jobs/ext/2049570-databricks-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Databricks Engineer

- **Company:** SMBC Group
- **Location:** New York, United States (Remote available)
- **Salary:** $73,000.0
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Amazon Web Services, Amazon S3, Computer Programming, Continuous Integration, Data Architecture, Information Engineering, Data Governance, Extract Transform Load (ETL), Github, Identity and Access Management, Python (Programming Language), Meta-Data Management, Performance Tuning, SQL Databases, Data Streaming, Unstructured Data, Data Processing, DevOps Tools - Open-source, Delivery Pipeline, Apache Spark, Git, Data Lakes, Pyspark, Gitlab-ci, Data Lineage, Deployment Automation, AWS Glue, Apache Kafka, Terraform, Data Pipelines, Databricks
- **Published:** August 14, 2026
- **Apply:** https://www.dice.com/job-detail/3eb43122-ce72-41a7-a2fd-d43a58e7cc36

## About the Role

* Professional experience in architectural design and development within the Databricks platform, working in an AWS cloud environment.
 * CI/CD & DevOps Tooling: Proven proficiency in automated deployments using Databricks Asset Bundles (DABs), Terraform (specifically the Databricks and AWS providers), and standard Git pipelines (e.g., GitHub Actions, GitLab CI/CD, or AWS CodePipeline).
 * Technical Proficiency: Programming skills in Python (PySpark) and SQL for complex data manipulation and transformation.
 * Core Concepts: Strong understanding of Apache Spark internals, Delta Lake mechanics, and streaming data concepts (e.g., interacting with Amazon MSK or Kafka data streams).
 * Data Engineering Stack: Proven experience building production-grade ETL/ELT pipelines, handling data schema validation, and cleansing raw capture feeds.
 * Certifications: Databricks Certified Data Engineer Professional or AWS Certified Data Engineer - Professional is highly advantageous.

## Description

We are seeking a Databricks Engineer with AWS expertise to build, optimize, and maintain our enterprise data Lakehouse infrastructure. In this role, you will be responsible for designing high-performance data pipelines, implementing regulatory and real time reporting along with advanced analytics environments, and ensuring seamless integration between Databricks and core AWS services to support real-time financial trading data consumption.

Role Objectives

 * Lakehouse Architecture: Design and implement robust data pipelines using the Databricks Medallion Architecture (Bronze, Silver, Gold layers) to process structured and unstructured data.
 * Pipeline Automation: Develop, scale, and orchestrate complex data workflows utilizing Databricks Jobs and Delta Live Tables (DLT).
 * Data Ops & CI/CD Deployment: Standardize and automate the deployment of Databricks assets, workspace configurations, and code pipelines across Dev, QA, and Production environments.
 * AWS Integration: Ensure seamless data cataloging, storage, and movement across the AWS ecosystem, specifically integrating Databricks with Amazon S3, AWS Glue, and AWS IAM for secure access control.
 * Performance Optimization: Tune Spark clusters, optimize Delta Lake storage (e.g., Z-Ordering, partitioning), and manage compute costs within the AWS environment.
 * Data Governance & Security: Implement fine-grained data access controls, data lineage, and auditing using Unity Catalog or native cloud security controls.
 * Collaboration: Partner with Data Scientists, Risk Managers, and downstream analytics teams to deliver clean, business-ready data views for reporting and AI modeling.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt)
- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)
- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again)