AWS/Azure Databricks Architect

Smart IT Frame
Raritan, United States of America
2 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Raritan, United States of America

Tech stack

Amazon Web Services (AWS)
Azure
Cloud Computing
Continuous Integration
Data Architecture
Information Engineering
Data Governance
Data Infrastructure
Data Security
Data Systems
Software Debugging
DevOps
Performance Tuning
Software Deployment
Workflow Management Systems
Azure
Spark
Amazon Web Services (AWS)
GIT
Data Lake
PySpark
Cloudwatch
Terraform
Databricks

Job description

We are looking for an experienced AWS Databricks Architect to design implement and optimize scalable Lakehouse solutions across AWS as well as Azure cloud platforms The ideal candidate will provide technical leadership in data architecture platform engineering data governance performance optimization and enterprisegrade Databricks implementations, Design and architect endtoend Databricks Lakehouse solutions across AWS and Azure environments

Define scalable data architecture patterns for ingestion transformation storage analytics and reporting workloads

Lead implementation of Delta Lake medallion architecture Databricks SQL Unity Catalog and workflow orchestration

Optimize Apache Spark jobs clusters notebooks pipelines and workloads for performance reliability and cost efficiency

Implement security access control secrets management data governance and compliance best practices across cloud platforms

Collaborate with data engineering BI data science DevOps and business teams to deliver highquality data solutions

Guide CICD infrastructureascode monitoring alerting and production deployment practices for Databricks workloads

Provide technical leadership architecture reviews troubleshooting support and mentoring to project teams

Requirements

6 years of experience in data engineering data architecture or big data platform implementation in AWS cloud platform and knowledge in Azure platform as well

Strong handson experience with Databricks platform architecture workspace setup cluster configuration jobs notebooks and workflow orchestration

Expertise in AWS services such as S3 IAM VPC Glue Lambda CloudWatch and related data platform services

Expertise in Azure services such as ADLS Azure Data Factory Azure Synapse Key Vault Entra ID and Azure Monitor

Strong knowledge of Apache Spark PySpark SQL Delta Lake and Lakehouse architecture concepts

Experience with data security governance access controls Unity Catalog encryption and cloud networking concepts

Handson experience with performance tuning cost optimization debugging and production support of Databricks workloads

Good understanding of CICD Git Terraform or equivalent infrastructureascode tools and DevOps practices"

Apply for this position