Databricks Engineer
Daman Consulting
United States
2 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours
Job source
Tech stack
Microsoft Azure
Big Data
Continuous Integration
Data Validation
Information Engineering
Data Governance
Extract Transform Load (ETL)
Data Manipulation Languages
Data Warehousing
Database Storage Structures
Software Debugging
DevOps
+22 more
R (Programming Language)
Apache Hive
Python (Programming Language)
Machine Learning
Meta-Data Management
Cloud Services
Tensorflow
Azure Data Lake
Systems Integration
Unstructured Data
Data Storage Technologies
Azure Data Factory
Apache Spark
Data Lakes
Scikit Learn
Data Lineage
Deployment Automation
Data Management
Machine Learning Operations
Software Version Control
Data Pipelines
Databricks
Job description
The Worker is responsible for developing, maintaining, and optimizing big data solutions using the Databricks Unified Analytics Platform. This role supports data engineering, machine learning, and analytics initiatives within this organization that relies on large-scale data processing. Duties include:
- Designing and developing scalable data pipelines
- Implementing ETL/ELT workflows
- Optimizing Spark jobs
- Integrating with Azure Data Factory
- Automating deployments
- Collaborating with cross-functional teams
- Ensuring data quality, governance, and security.
Requirements
- Implement ETL/ELT workflows for both structured and unstructured data
- Automate deployments using CI/CD tools
- Collaborate with cross-functional teams including data scientists, analysts, and stakeholders
- Design and maintain data models, schemas, and database structures to support analytical and operational use cases
- Evaluate and implement appropriate data storage solutions, including data lakes (Azure Data Lake Storage) and data warehouses
- Implement data validation and quality checks to ensure accuracy and consistency
- Contribute to data governance initiatives, including metadata management, data lineage, and data cataloging
- Implement data security measures, including encryption, access controls, and auditing; ensure compliance with regulations and best practices
- Proficiency in Python and R programming languages
- Strong SQL querying and data manipulation skills
- Experience with Azure cloud platform
- Experience with DevOps, CI/CD pipelines, and version control systems
- Working in agile, multicultural environments
- Strong troubleshooting and debugging capabilities
- Design and develop scalable data pipelines using Apache Spark on Databricks
- Optimize Spark jobs for performance and cost-efficiency
- Integrate Databricks solutions with cloud services (Azure Data Factory)
- Ensure data quality, governance, and security using Unity Catalog or Delta Lake
- Deep understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL
- Hands-on experience with Databricks notebooks, clusters, jobs, and Delta Lake
- Knowledge of ML libraries (MLflow, Scikit-learn, TensorFlow)
- Databricks Certified Associate Developer for Apache Spark
- Azure Data Engineer Associate
Benefits & conditions
Depends on Experience
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦