Senior Data Engineer
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+10 more
Job description
We are seeking a highly skilled Senior Databricks Data Engineer with 8+ years of experience in data engineering and strong hands-on expertise in Databricks. The ideal candidate must have proven production experience designing, developing, and maintaining scalable data pipelines using Databricks, Apache Spark, PySpark, and Delta Lake. The candidate should have extensive experience building end-to-end ETL/ELT pipelines, working with large-scale datasets, optimizing data processing workflows, and managing production-grade Databricks Jobs and Workflows. Note: Hands-on production experience with Databricks is mandatory. General Apache Spark experience without actual Databricks implementation experience will not be considered., * Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Build and optimize ETL/ELT workflows using Python, PySpark, and Spark SQL.
- Develop, manage, and optimize Delta Lake tables and Delta architectures.
- Create, schedule, monitor, and troubleshoot Databricks Jobs and Workflows.
- Implement data ingestion, transformation, and processing solutions for large-scale datasets.
- Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
- Work with data lakes, data warehouses, and cloud-based data platforms.
- Implement data quality checks, validation, and error-handling mechanisms.
- Collaborate with data architects, analysts, and cross-functional teams to understand business requirements.
- Troubleshoot production issues and ensure reliability and performance of data pipelines.
- Follow best practices for data security, governance, and pipeline optimization.
Requirements
Apache Spark, Data Management, Scalable System Development, Data Sets, Data Processing, Database Extract Transform and Load (ETL), Production Management, Python Programming/Scripting Language, SQL (Structured Query Language), Identify Issues
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦