Data Engineer

IBA InfoTech Inc.

Charlotte, NC, United States

about 2 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

10 years minimum

Working hours

Regular working hours

Job source

ibainfotech.com

Tech stack

Amazon Web Services Amazon Elastic Compute Cloud Amazon S3 Batch Processing Cloud Computing Continuous Integration Data Architecture Extract Transform Load (ETL) Data Mining IBM DB2 Database Theory Software Debugging

+21 more

Distributed Systems Fault Tolerance Apache Hadoop Apache Hive Identity and Access Management Machine Learning Oracle (Applications) Performance Tuning Software Architecture Data Processing Load Balancing Data Ingestion Autoscaling Apache Spark Amazon Virtual Private Cloud (VPC) AWS ECS Amazon Relational Database Service Information Technology Cloudwatch Amazon Simple Queue Service (SQS) Data Pipelines

Job description

Create a Low-Level implementation plan for the ingestion pipeline.
Develop the ingestion pipeline based on HLD/LLD specifications.
Define the storage directories in S3 and script the compaction and portioning based on consumption patterns.
Work on the issues identified in the data pipeline.
Integrate the pipeline jobs to enterprise scheduler and CI/CD Process.
Work with support teams to promote it to prod

Requirements

10 plus years of Experience as Data Engineer
Expertise in Amazon EC2, Amazon S3, Amazon RDS, VPC, IAM, Amazon Elastic Load Balancing, Auto Scaling, Cloud Front, CloudWatch, SNS, SES, SQS and other services of the AWS family.
Experience in working with Oracle and DB2. And making the data to be batch processing using distributed computing.
Good knowledge of High-Availability, Fault Tolerance, Scalability, Database Concepts, System and Software Architecture, Security and IT Infrastructure.
Expertise in Spark SQL, Tuning and Debugging the Spark Cluster
Familiar with data architecture including data ingestion pipeline design, Hadoop information architecture, data modeling and data mining, machine learning and advanced data processing. Experience optimizing ETL workflows.
Knowledge of tools like Datameer