Data Engineer
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+1 more
Job description
Design, develop, and maintain scalable batch and real-time data pipelines using Databricks.
Requirements
n
Bachelor’s degree in Computer Science, Information Technology, or a related field.
\n
Strong experience with Databricks, Apache Spark, PySpark, Spark SQL, and Delta Lake.
\n
Experience designing and developing large-scale data pipelines and ETL processes.
\n
Strong understanding of Data Lakehouse architecture and data modeling concepts.
\n
Experience with cloud platforms such as Azure, AWS, or GCP.
\n
Knowledge of data governance, security, and enterprise data management practices.
\n
Excellent problem-solving, analytical, and communication skills.
Benefits & conditions
Build robust data solutions using PySpark, Spark SQL, and Delta Lake.
\n
Implement data validation, quality checks, and performance tuning processes.
\n
Develop ETL/ELT workflows to support enterprise data integration needs.
\n
Data Lakehouse Management
\n
Design and manage data architecture within the Databricks Lakehouse Platform.
\n
Implement Delta Lake tables, partitioning strategies, optimization techniques, and schema evolution.
\n
Develop data models to support analytics, reporting, and business intelligence requirements.
\n
Ensure data reliability, scalability, governance, and availability.
\n
Performance Optimization
\n
Analyze and optimize Spark jobs for performance and cost efficiency.
\n
Monitor Databricks cluster utilization and implement best practices for resource management.
\n
Troubleshoot bottlenecks in data processing workflows.
\n
Improve processing times through caching, partitioning, and query optimization techniques.
\n
Data Governance & Security
\n
Implement role-based access controls (RBAC) and data security standards.
\n
Support data governance initiatives, data lineage tracking, and compliance requirements.
\n
Ensure adherence to enterprise data management policies and security frameworks.
\n
Collaboration & Delivery
\n
Work closely with Data Architects, Data Scientists, Business Analysts, and Application Teams.
\n
Participate in Agile development processes, sprint planning, and scrum ceremonies.
\n
Prepare technical design documentation and support knowledge transfer activities.
\n
Collaborate with stakeholders to gather requirements and translate them into scalable technical solutions.
\n
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this role…