Senior Data Engineer

VeeRteq Solutions Inc

United States

2 months ago

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

5 years minimum

Working hours

Regular working hours

Job source

indeed.com

Tech stack

Amazon Web Services Amazon S3 Apache HTTP Server Information Engineering Data Infrastructure Extract Transform Load (ETL) Data Warehousing Amazon DynamoDB Python (Programming Language) Metadata SQL Databases Data Streaming

+14 more

Parquet AWS Cdk Data Processing Apache Spark AWS Lambda Cloudformation Build Management Pyspark Information Technology Avro AWS Glue Data Analytics Integration Frameworks Data Pipelines

Job description

We are seeking a Senior Data Engineer to design and build the data pipelines, data products, and integration flows. This role involves hands-on pipeline architecture, data quality validation, and building the foundational data products that enable the Industrial Data Mesh., Technical Leadership

Design and architect batch and streaming data pipelines for industrial data

Define data product schemas, contracts, and quality validation rules

Implement data integration patterns (CDC, event-driven, pub/sub) across OT and IT systems

Design schema evolution strategies using Avro, Parquet, and Apache Iceberg

Optimize pipeline performance and cost efficiency

Customer Engagement

Collaborate with data teams to understand existing data flows

Support data domain workshops with technical pipeline feasibility input

Present pipeline design recommendations to customer engineering teams

Solution Development

Build production-ready data pipelines on AWS infrastructure

Implement data quality validation and enrichment at ingestion

Requirements

Do you have experience in Spark implementation?, Must-have: Strong Glue, ETL, and Spark skills; experience with Iceberg; ability to work with both batch and real-time or streaming ingestion; familiarity with Kinesis and/or MSK; and understanding of Redshift and data warehouse patterns. Nice-to-have: Working knowledge of DynamoDB for metadata-related use cases, broader awareness of AWS streaming options, and prototype-building capability to support architecture validation., 5-7 years in data engineering or ETL/ELT development

Experience with large-scale streaming and batch data processing

Experience in manufacturing or industrial data environments preferred

Technical Skills (AWS Services, would consider competitive alternatives)

AWS Glue (ETL, Spark, Iceberg), AWS Step Functions, AWS Lambda

Amazon S3 (partitioning, lifecycle management)

Amazon DynamoDB (metadata/state)

AWS CDK / CloudFormation

Programming: Python, PySpark, SQL

Soft Skills

Strong problem-solving and analytical skills

Ability to work collaboratively with architects and customer teams

Experience in agile environments