Senior Data Engineer

Tech3pillars Technologies
Los Angeles, CA, United States
1 day ago
Apply on www.careerbuilder.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
8 years minimum
Working hours
Regular working hours

Tech stack

Big Data Databases Data Validation Information Engineering Data Files Extract Transform Load (ETL) Data Security Data Warehousing Apache Hive Python (Programming Language) Systems Development Life Cycle SQL Databases
+10 more
Data Processing Scripting Cloud Platform System Data Ingestion Apache Spark Data Lakes Pyspark Data Management Data Pipelines Databricks

Job description

We are seeking a highly skilled Senior Databricks Data Engineer with 8+ years of experience in data engineering and strong hands-on expertise in Databricks. The ideal candidate must have proven production experience designing, developing, and maintaining scalable data pipelines using Databricks, Apache Spark, PySpark, and Delta Lake. The candidate should have extensive experience building end-to-end ETL/ELT pipelines, working with large-scale datasets, optimizing data processing workflows, and managing production-grade Databricks Jobs and Workflows. Note: Hands-on production experience with Databricks is mandatory. General Apache Spark experience without actual Databricks implementation experience will not be considered., * Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.

  • Build and optimize ETL/ELT workflows using Python, PySpark, and Spark SQL.
  • Develop, manage, and optimize Delta Lake tables and Delta architectures.
  • Create, schedule, monitor, and troubleshoot Databricks Jobs and Workflows.
  • Implement data ingestion, transformation, and processing solutions for large-scale datasets.
  • Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
  • Work with data lakes, data warehouses, and cloud-based data platforms.
  • Implement data quality checks, validation, and error-handling mechanisms.
  • Collaborate with data architects, analysts, and cross-functional teams to understand business requirements.
  • Troubleshoot production issues and ensure reliability and performance of data pipelines.
  • Follow best practices for data security, governance, and pipeline optimization.

Requirements

Apache Spark, Data Management, Scalable System Development, Data Sets, Data Processing, Database Extract Transform and Load (ETL), Production Management, Python Programming/Scripting Language, SQL (Structured Query Language), Identify Issues

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerbuilder.com
Prepare application

Good distractions

Loading talks and stories from around this role…