Databricks Data Engineer Architect

Cyma Systems Inc
United States
18 days ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours
Job source

Tech stack

Application Programming Interfaces (APIs) Amazon Web Services Microsoft Azure Cloud Computing Continuous Integration Data Architecture Information Engineering Data Governance Data Integration Extract Transform Load (ETL) Data Migration Software Design Documents
+20 more
Digital Architecture Python (Programming Language) Netsuite Salesforce.Com SQL Databases Data Streaming Workflow Management Systems Data Processing Google Cloud Data Ingestion Change Data Capture Data Lakes Pyspark Deployment Automation Data Management Data Lakehouse Api Design Restful APIs Data Pipelines Databricks

Job description

We are seeking an experienced Databricks Data Engineer Architect to lead the architecture, design, and implementation of enterprise data engineering solutions on the Databricks platform. The ideal candidate will have extensive hands-on experience with Databricks, a proven track record of architecting scalable data solutions, and demonstrated experience migrating and integrating data from Salesforce and NetSuite into Databricks.

The candidate will be responsible for defining the data architecture, establishing data integration patterns, and ensuring the development of reliable, scalable, secure, and high-performing data pipelines. This role will work closely with business stakeholders, technical teams, and offshore data engineers to deliver the project’s data engineering objectives., * Architect, design, and implement end-to-end data engineering solutions using Databricks.

  • Lead the design and development of scalable data ingestion and transformation pipelines.
  • Design and implement data migration solutions to ingest data from Salesforce and NetSuite into Databricks.
  • Define data architecture, data models, integration patterns, and best practices for the Databricks platform.
  • Develop and optimize ETL/ELT pipelines using Python, SQL, PySpark, and other relevant technologies.
  • Establish data quality, validation, reconciliation, monitoring, and error-handling frameworks.
  • Design secure and efficient data processing solutions, including appropriate access controls and governance practices.
  • Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
  • Collaborate with business stakeholders to understand data requirements and translate them into technical architecture and implementation plans.
  • Provide technical leadership and guidance to offshore data engineering resources.
  • Review technical designs, code, and implementation approaches to ensure adherence to architectural standards.
  • Support testing, deployment, troubleshooting, and production readiness activities.
  • Document architecture decisions, data flows, technical specifications, and operational procedures.

Requirements

  • Extensive hands-on experience with Databricks and its data engineering capabilities.
  • Proven experience architecting and delivering end-to-end data engineering solutions.
  • Demonstrated experience migrating and integrating data from both Salesforce and NetSuite into Databricks.
  • Strong proficiency in Python, SQL, and PySpark.
  • Extensive experience building and managing ETL/ELT pipelines and complex data integration workflows.
  • Strong understanding of data lakehouse architecture, data modeling, and modern cloud data platforms.
  • Experience with Delta Lake and related Databricks capabilities.
  • Knowledge of data governance, security, data quality, and pipeline monitoring.
  • Experience with API-based integrations, incremental data loads, and change data capture where applicable.
  • Strong analytical, problem-solving, communication, and technical leadership skills.
  • Ability to work collaboratively with onshore stakeholders and offshore delivery teams., * Databricks certifications in data engineering or solution architecture.
  • Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud.
  • Experience with orchestration tools, CI/CD pipelines, and automated deployment practices.
  • Familiarity with Salesforce APIs, NetSuite APIs, and enterprise application integration patterns.
  • Experience delivering enterprise-scale data migration and modernization projects.

Key Deliverables

  • Databricks solution architecture and technical design documentation.
  • Data ingestion and migration architecture for Salesforce and NetSuite.
  • Scalable and production-ready data pipelines.
  • Data quality, validation, and reconciliation mechanisms.
  • Technical standards and implementation guidelines for the data engineering team.
  • Technical leadership, design reviews, and support for project delivery.

Benefits & conditions

Depends on Experience

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…