Data Engineer

Openmind Technologies

Boston, MA, United States

5 days ago

Role details

Contract type

Temporary to permanent

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Microsoft Azure Big Data Cloud Computing Information Engineering Data Governance Data Integration Data Warehousing DevOps Distributed Systems Python (Programming Language) Machine Learning Performance Tuning

+11 more

SQL Databases Data Streaming Unstructured Data Data Ingestion Azure Data Factory Snowflake Apache Spark Data Lakes Apache Kafka Spark Streaming Data Pipelines

Job description

Design, build, and optimize big data ingestion pipelines to handle large volumes of structured and unstructured data.
Develop and maintain real-time streaming solutions using Kafka and Spark Streaming.
Collaborate with data architects and business stakeholders to define data integration strategies.
Implement scalable solutions on Azure cloud services and integrate with Snowflake for analytics and reporting.
Ensure data quality, reliability, and performance across distributed systems.
Troubleshoot and optimize existing pipelines to improve efficiency and reduce latency.
Document technical designs, workflows, and best practices for long-term maintainability.

Requirements

We are seeking a highly skilled Data Engineer to join our consulting team for a remote engagement with a client headquartered in Boston, MA. The ideal candidate will have extensive experience working with large-scale data pipelines, real-time streaming technologies, and cloud-based big data platforms. This role requires strong expertise in Apache Spark, Apache Kafka, Python, Machine learning and data ingestion frameworks, along with hands-on experience in Azure, SQL, and Snowflake., * 12+ years of professional experience as abig Data Engineer or similar role.

Strong expertise in Apache Spark (batch and streaming) and Apache Kafka.
Proven experience with large-scale data ingestion and processing.
Hands-on experience with Azure Data Services (e.g., Data Lake, Data Factory, Synapse).
Proficiency in SQL and experience with Snowflake for data warehousing.
Solid understanding of distributed systems and scalable architectures.
Strong problem-solving skills and ability to work independently in a remote environment., * Experience with performance tuning of big data pipelines.
Familiarity with CI/CD pipelines and DevOps practices for data engineering.
Exposure to data governance and security best practices.