Databricks Lead/Architecture
Tech Tammina LLC
San Francisco, CA, United States
15 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.careerbuilder.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours
Job source
Tech stack
Artificial Intelligence
Amazon Web Services
Cloud Computing
Continuous Integration
Information Engineering
Data Masking
DevOps
Metadata
Meta-Data Management
Performance Tuning
Role-Based Access Control
SQL Databases
+13 more
Data Streaming
Azure Service Bus
Data Ingestion
Data Lakes
Pyspark
Data Lineage
Apache Kafka
Data Management
Cloud Optimization
Stream Processing
Data Pipelines
Serverless Computing
Databricks
Job description
Architecture & Design:
- Define and design end-to-end data Lakehouse architecture leveraging Databricks, Delta Lake, and cloud-native services.
- Create reference architectures for batch, real-time, and streaming data pipelines.
- Architect data ingestion, curation, storage, and governance frameworks.
- Ensure the platform is scalable, secure, and optimized for performance and cost.
- Establish standards for data lineage, metadata management, and compliance.
- Work with enterprise architects to align Databricks solutions with the overall cloud strategy (AWS).
Leadership & Delivery:
- Lead and mentor a team of data engineers in building robust data pipelines.
- Collaborate with data scientists, BI teams, and business stakeholders to enable advanced analytics and AI/ML use cases.
- Drive adoption of DevOps and CI/CD practices for data engineering.
- Review designs and solutions to ensure adherence to architectural principles.
Implementation & Optimization:
- Build, optimize, and manage large-scale PySpark/SQL pipelines in Databricks.
- Enable real-time data processing through Kafka, Kinesis, or Event Hubs.
- Implement security best practices, including RBAC, data masking, and encryption.
Requirements
Cloud Computing, Data Management, Performance Tuning/Optimization, Metadata, Enterprise Architecture, Amazon Web Services (AWS), Mentoring, Use Cases, Business Intelligence, Data Science
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.careerbuilder.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦