Databricks Data Engineer Architect
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+20 more
Job description
We are seeking an experienced Databricks Data Engineer Architect to lead the architecture, design, and implementation of enterprise data engineering solutions on the Databricks platform. The ideal candidate will have extensive hands-on experience with Databricks, a proven track record of architecting scalable data solutions, and demonstrated experience migrating and integrating data from Salesforce and NetSuite into Databricks.
The candidate will be responsible for defining the data architecture, establishing data integration patterns, and ensuring the development of reliable, scalable, secure, and high-performing data pipelines. This role will work closely with business stakeholders, technical teams, and offshore data engineers to deliver the projectβs data engineering objectives., * Architect, design, and implement end-to-end data engineering solutions using Databricks.
- Lead the design and development of scalable data ingestion and transformation pipelines.
- Design and implement data migration solutions to ingest data from Salesforce and NetSuite into Databricks.
- Define data architecture, data models, integration patterns, and best practices for the Databricks platform.
- Develop and optimize ETL/ELT pipelines using Python, SQL, PySpark, and other relevant technologies.
- Establish data quality, validation, reconciliation, monitoring, and error-handling frameworks.
- Design secure and efficient data processing solutions, including appropriate access controls and governance practices.
- Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
- Collaborate with business stakeholders to understand data requirements and translate them into technical architecture and implementation plans.
- Provide technical leadership and guidance to offshore data engineering resources.
- Review technical designs, code, and implementation approaches to ensure adherence to architectural standards.
- Support testing, deployment, troubleshooting, and production readiness activities.
- Document architecture decisions, data flows, technical specifications, and operational procedures.
Requirements
- Extensive hands-on experience with Databricks and its data engineering capabilities.
- Proven experience architecting and delivering end-to-end data engineering solutions.
- Demonstrated experience migrating and integrating data from both Salesforce and NetSuite into Databricks.
- Strong proficiency in Python, SQL, and PySpark.
- Extensive experience building and managing ETL/ELT pipelines and complex data integration workflows.
- Strong understanding of data lakehouse architecture, data modeling, and modern cloud data platforms.
- Experience with Delta Lake and related Databricks capabilities.
- Knowledge of data governance, security, data quality, and pipeline monitoring.
- Experience with API-based integrations, incremental data loads, and change data capture where applicable.
- Strong analytical, problem-solving, communication, and technical leadership skills.
- Ability to work collaboratively with onshore stakeholders and offshore delivery teams., * Databricks certifications in data engineering or solution architecture.
- Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud.
- Experience with orchestration tools, CI/CD pipelines, and automated deployment practices.
- Familiarity with Salesforce APIs, NetSuite APIs, and enterprise application integration patterns.
- Experience delivering enterprise-scale data migration and modernization projects.
Key Deliverables
- Databricks solution architecture and technical design documentation.
- Data ingestion and migration architecture for Salesforce and NetSuite.
- Scalable and production-ready data pipelines.
- Data quality, validation, and reconciliation mechanisms.
- Technical standards and implementation guidelines for the data engineering team.
- Technical leadership, design reviews, and support for project delivery.
Benefits & conditions
Depends on Experience
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦