Senior Data Engineer

Insight Global is seeking a Senior Data Engineer to design and build the cloud data foundation supporting Off-Prem. The platform depends on large volumes of operational, engineering, analytical and contextual information. This role will develop the pipelines, storage architecture, data models, APIs and quality controls required to make that information securely and reliably available applications and AI capabilities. The ideal candidate combines modern AWS data engineering expertise with a strong understanding of data quality, time-series information and scalable data architectures.

Design and implement the a cloud data architecture.

Build scalable ingestion, transformation, storage and serving pipelines on AWS.

Develop pipelines for operational, historical, engineering, application and contextual data.

Design data models optimized for analytical applications and AI consumption.

Establish appropriate storage patterns across relational, object, time-series and analytical data stores.

Develop APIs and services enabling tools and agents to retrieve data consistently.

Establish automated data-quality validation, reconciliation and monitoring.

Implement metadata management, lineage, cataloging and data-governance practices.

Design appropriate mechanisms for data segregation and access control.

Optimize data pipelines for scalability, performance, reliability and cost.

Partner with operational subject-matter experts to validate the meaning and quality of source data.

Support migration of relevant information from document-based repositories into structured cloud-hosted data services.

Collaborate with AI/ML engineers to create reliable datasets, feature pipelines, retrieval mechanisms and knowledge sources.

Establish reusable data integration patterns that accelerate onboarding of additional platform use cases and sites.

Bachelor’s degree in Computer Science, Data Engineering, Engineering, or a related field.

5+ years of data engineering experience.

Strong Python and SQL skills.

Hands-on experience building production data pipelines in AWS.

Experience with AWS data technologies such as S3, Glue, Lambda, RDS/Aurora, Redshift, Athena, Kinesis or equivalent technologies.

Experience with ETL/ELT architecture and orchestration frameworks.

Strong knowledge of relational and non-relational database design.

Experience implementing automated data-quality processes.

Experience building APIs or data services.

Understanding of data governance, lineage, security and access-control principles.