WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
Back

Lead Data Engineer with Java, Python, Spark & AWS - Onsite - W2 only

MSYS Inc.
Columbus, OH, United States
1 day ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
10 years minimum
Working hours
Regular working hours
Job source
www.dice.com

Tech stack

Java (Programming Language) Artificial Intelligence Airflow Amazon Web Services Apache HTTP Server Automation of Tests Continuous Integration Data Architecture Data Governance Extract Transform Load (ETL) Distributed Computing Environment Python (Programming Language)
+15 more
Performance Tuning Software Engineering Parquet Data Ingestion Snowflake Apache Spark Generative AI Agentic-AI Data Lakes Avro Apache Kafka Spark Streaming Restful APIs Data Pipelines Control M

Job description

Seeking an experienced Lead Data Engineer with strong expertise in Java, Python, AWS, ETL/ELT pipelines, and Apache Spark., * Lead architecture and development of large-scale ETL/ELT data pipelines.

  • Design data ingestion, transformation, validation, and reconciliation workflows.
  • Develop distributed data processing solutions using Apache Spark.
  • Design and implement data lake and lakehouse architectures.
  • Optimize Spark workloads, including joins, caching, partitioning, and performance tuning.
  • Develop Java-based services and REST APIs.
  • Implement automated testing, CI/CD, monitoring, and production support.
  • Drive engineering best practices and mentor development teams.
  • Promote enterprise-approved AI-assisted engineering practices.

Requirements

  • 10+ years of software engineering experience.
  • Strong hands-on Java development.
  • Python proficiency.
  • ETL/ELT pipeline development.
  • Apache Spark and distributed processing.
  • AWS cloud experience.
  • Parquet, Avro, and data lake/lakehouse architecture.
  • Snowflake or equivalent cloud data warehouse experience.
  • AI-assisted development tools and responsible AI practices.

Preferred Skills

  • Airflow, Dagster, or Control-M.
  • Kafka and Spark Structured Streaming.
  • Delta Lake, Apache Iceberg, or Hudi.
  • Data governance and data quality.
  • Financial services or regulated industry experience.
  • Agentic AI and Generative AI exposure.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…