Lead Data Engineer

TechDigital Corporation
Warsaw, IN, United States
1 day ago
Apply on www.careerbuilder.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Working hours
Regular working hours

Tech stack

Java (Programming Language) Amazon Web Services Automation of Tests Microsoft Azure Big Data BigQuery Computer Programming Databases Data Architecture Information Engineering Extract Transform Load (ETL) Data Warehousing
+16 more
DevOps Dimensional Modeling Python (Programming Language) Performance Tuning DataOps Data Streaming Data Processing Scripting Real Time Systems Snowflake Change Data Capture Data Lakes Azure Synapse Analytics Data Pipelines Amazon Redshift Programming Languages

Job description

Data Architecture & Solution Design

  • Design end to end data engineering architectures
  • Define scalable solutions for: o Data lakes / lakehouse o Data warehouses o Streaming and real time systems

  • Ensure alignment with enterprise architecture, security, and compliance standards
  • Review and approve technical designs

Data Pipeline Development & Management

  • Lead development of ETL / ELT pipelines
  • Handle: o Batch and real time ingestion o Structured and semi structured data

  • Optimize pipelines for performance, reliability, and cost
  • Manage schema evolution and data dependencies

Data Quality, Reliability & Operations

  • Establish data quality standards and validation rules
  • Implement monitoring, alerting, and observability
  • Perform root cause analysis for data incidents
  • Drive operational excellence and stability

DevOps / DataOps Enablement

  • Build CI/CD pipelines for data workloads
  • Automate testing, deployment, and rollback
  • Improve reliability through automation

Requirements

Database Extract Transform and Load (ETL), Data Processing, Structured Data, Python Programming/Scripting Language, Scala Programming Language, Computer Programming, Java, Performance Tuning/Optimization, Microsoft Windows Azure, Amazon Web Services (AWS), * Advanced expertise in ETL / ELT pipeline design

  • Handling: o Batch data processing o Near real time / streaming data

  • Experience with structured and semi structured data
  • Strong knowledge of: o Incremental loading o CDC (Change Data Capture)

  • Pipeline orchestration and dependency management
  • Strong programming skills in Python or Scala or Java (nice to have)
  • Performance optimization for large scale data processing
  • Solid understanding of: o Dimensional modeling (Star / Snowflake) o Normalized and denormalized models

  • Strong experience on: Azure, AWS, or GCP
  • Hands on with Data Warehouses (Snowflake, Synapse, BigQuery, Redshift)

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.careerbuilder.com
Prepare application

Good distractions

Loading talks and stories from around this role…