Data Engineer

IBA InfoTech Inc.
Charlotte, United States of America
27 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Charlotte, United States of America

Tech stack

Amazon Web Services (AWS)
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Batch Processing
Cloud Computing
Continuous Integration
Data Architecture
ETL
Data Mining
IBM DB2
Database Theory
Software Debugging
Distributed Systems
Fault Tolerance
Hadoop
Hive
Identity and Access Management
Machine Learning
Oracle Applications
Performance Tuning
Software Architecture
Data Processing
Load Balancing
Data Ingestion
Autoscaling
Spark
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Information Technology
Cloudwatch
Amazon Web Services (AWS)
Data Pipelines

Job description

  • Create a Low-Level implementation plan for the ingestion pipeline.
  • Develop the ingestion pipeline based on HLD/LLD specifications.
  • Define the storage directories in S3 and script the compaction and portioning based on consumption patterns.
  • Work on the issues identified in the data pipeline.
  • Integrate the pipeline jobs to enterprise scheduler and CI/CD Process.
  • Work with support teams to promote it to prod

Requirements

  • 10 plus years of Experience as Data Engineer
  • Expertise in Amazon EC2, Amazon S3, Amazon RDS, VPC, IAM, Amazon Elastic Load Balancing, Auto Scaling, Cloud Front, CloudWatch, SNS, SES, SQS and other services of the AWS family.
  • Experience in working with Oracle and DB2. And making the data to be batch processing using distributed computing.
  • Good knowledge of High-Availability, Fault Tolerance, Scalability, Database Concepts, System and Software Architecture, Security and IT Infrastructure.
  • Expertise in Spark SQL, Tuning and Debugging the Spark Cluster
  • Familiar with data architecture including data ingestion pipeline design, Hadoop information architecture, data modeling and data mining, machine learning and advanced data processing. Experience optimizing ETL workflows.
  • Knowledge of tools like Datameer

Apply for this position