Senior Data Engineer in Oakland

Energy Jobline

Oakland, CA, United States

5 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Job source

www.energyjobline.com

Tech stack

Airflow Amazon Web Services Microsoft Azure Big Data Continuous Integration Information Engineering Extract Transform Load (ETL) Data Transformation Data Warehousing Software Debugging DevOps Distributed Data Store

+13 more

Python (Programming Language) Cloud Services Standard Sql Workflow Management Systems Data Ingestion Azure Data Factory Snowflake Data Lakes Pyspark AWS Glue Software Version Control Data Pipelines Databricks

Job description

Build and maintain ETL/ELT data pipelines using PySpark and Databricks.
Work with distributed datasets, ensuring efficiency, scalability, and accuracy.
Develop and optimize workflows for data ingestion, transformation, and warehousing.
Write high-quality Python code for data engineering use cases.
Collaborate with data science and analytics teams to translate requirements into solutions.
Troubleshoot issues, optimize queries, and manage performance within AWS cloud environments.
Support CI/CD and automation workflows for reliability and repeatability.

Requirements

Strong hands-on experience in Python and PySpark for large-scale data engineering.
Proven expertise in using Databricks for data transformation and analytics.
Solid working knowledge of SQL and cloud data platforms (e.g., Snowflake).
Experience with major cloud providers (AWS ; Azure/GCP considered).
Good understanding of data modelling, data warehouses, and data lakes.
Exposure to orchestration tools like Airflow, AWS Glue, or Azure Data Factory.
Familiarity with DevOps practices, CI/CD, and version control systems.
Strong problem-solving and debugging skills in distributed data environments.