Data Engineer

Horizontal Talent

San Francisco, CA, United States

21 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

8 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Java (Programming Language) Agile Methodology Artificial Intelligence Amazon Web Services Business Analytics Applications Microsoft Azure Big Data Continuous Integration Information Engineering Data Governance Data Infrastructure Extract Transform Load (ETL)

+22 more

Dataspaces Data Systems DevOps Distributed Data Store Distributed Systems Apache Hadoop Python (Programming Language) NoSQL Cloud Services SQL Databases Data Streaming Data Processing Google Cloud Azure Data Factory Apache Spark Apache Flink Integration Frameworks Apache Kafka Dynamic Data Machine Learning Operations Data Pipelines Databricks

Job description

Join a dynamic data engineering team supporting large-scale analytics and cloud data solutions for a hybrid onsite role in San Francisco, CA. This position focuses on building reliable, high-performance pipelines and modern data platform capabilities using Databricks, Azure Data Factory, and DevOps practices. Responsibilities

Design, build, and maintain scalable data pipelines and analytics solutions for big data environments.
Develop and optimize workflows that support efficient, reliable data processing and delivery.
Work closely with cross-functional partners to translate data needs into actionable technical solutions.
Support data platform operations with a focus on quality, availability, performance, and stability.
Apply cloud and DevOps practices to automate data workflows and improve delivery efficiency.
Troubleshoot issues across data pipelines and environments to help ensure smooth day-to-day operations.
Contribute to data governance, security, and compliance practices within the data ecosystem.
Share knowledge and help mentor others while promoting collaborative ways of working.

Requirements

8+ years of experience in data engineering with a strong big data background.
Hands-on experience with Databricks and Azure Data Factory.
Experience with DevOps tools and practices related to data engineering and CI/CD.
Strong working knowledge of Hadoop, Spark, Kafka, or similar distributed data technologies.
Proficiency in Python, Scala, or Java.
Solid understanding of SQL and NoSQL databases.
Experience building ETL processes, data pipelines, and real-time or batch data solutions.
Strong problem-solving skills and the ability to communicate effectively with technical and business stakeholders.

Preferred Skills

Experience with cloud data platforms such as Azure, AWS, or Google Cloud Platform.
Knowledge of Flink or other streaming/data processing frameworks.
Familiarity with data modeling and distributed computing concepts.
Background supporting machine learning pipelines or AI-enabled data solutions.
Experience in Agile environments and with team leadership or mentoring.
Knowledge of California emissions-related data or compliance work is a plus.

Horizontal Talent is committed to fostering an inclusive workplace where diverse perspectives are valued and everyone has the opportunity to thrive. We welcome candidates of all backgrounds and experiences who are excited to contribute to a collaborative, respectful, and equitable environment.