Data Engineer

Fourth

Santa Clara, CA, United States

7 days ago

Apply on www.dice.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Big Data Databases Information Engineering Extract Transform Load (ETL) Data Structures Database Queries Distributed Computing Environment Distributed Systems SQL Databases Data Processing Freeform SQL Apache Spark

+2 more

Pyspark Data Pipelines

Job description

Develop and maintain scalable data pipelines using PySpark and SQL.
Build data processing and transformation workflows.
Process large datasets using distributed computing technologies.
Write and optimize complex SQL queries.
Ensure data quality, validation, and reliability.
Troubleshoot and optimize data pipeline and Spark performance.
Collaborate with data and engineering teams to deliver scalable solutions.

Requirements

Strong PySpark experience.
Strong SQL skills.
Hands-on Data Engineering experience.
Experience building data processing and transformation pipelines.
Strong understanding of databases, data structures, and data quality.
Experience with distributed data processing.
Experience with ETL/ELT and data pipeline development.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com

Prepare application

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

Moments

Explore playlists

3:28 min

Defining big data and machine learning fundamentals

Ayon Roy · LIVE

3:09 min

Balancing data science skillings alongside systems engineering rigor

Nico Schmidt · LIVE

3:04 min

Database evolution and the funding behind vector databases

Erik Bamberg · LIVE

4:32 min

Harnessing Spark with Python using PySpark and Py4J

Ayon Roy · LIVE

2:10 min

Why organizations combine big data and machine learning

Ayon Roy · LIVE

3:37 min

Scaling machine learning pipelines from prototypes to petabytes

Julian Joseph · LIVE

Videos

See all

PySpark - Combining Machine Learning & Big Data thumbnail