Data Engineer

Spectraforce
Newark, United States of America
yesterday

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Newark, United States of America

Tech stack

API
Agile Methodologies
Airflow
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Data analysis
Confluence
JIRA
Automation of Tests
Unit Testing
Software as a Service
Cloud Computing
Software Quality
Code Review
Computer Programming
Continuous Integration
ETL
Data Systems
Data Warehousing
Relational Databases
DevOps
File Systems
Amazon DynamoDB
Gradle
Groovy
Identity and Access Management
Python
Maven
Network Attached Storage (Server Appliance)
Role-Based Access Control
Standard Sql
Shell Script
Amazon Web Services (AWS)
Software Engineering
SONAR (Symantec)
SonarQube
SQL Stored Procedures
Enterprise Data Management
Cloudbees
Software Repository
Scripting (Bash/Python/Go/Ruby)
Data Ingestion
Spark
Cloudformation
Servicebus
Amazon Web Services (AWS)
Information Technology
Atlassian Tools
Production Code
Amazon Web Services (AWS)
Kafka
Build Tools
SAP Ariba
Bitbucket
Cloudwatch
REST
Amazon Web Services (AWS)
Software Version Control
Data Pipelines
Docker
Jenkins
Vulnerability Analysis
Artifactory
Programming Languages

Job description

  • Designing, building, and maintaining efficient, reusable, and reliable code
  • Ensure the best possible performance and quality of high scale data applications and services
  • Participate in system design discussions
  • Independently perform hands on development and unit testing of the applications
  • Collaborate with the development team and build individual components into enterprise data platform
  • Work in a team environment with product, QE/QA, and cross functional teams to deliver a project throughout the whole software development cycle.
  • Responsible to identify and resolve any performance issues
  • Keep up to date with new technology development and implementation
  • Participate in code review to make sure standards and best practices are met
  • Project management: Agile developers take responsibility for estimating, planning, and managing all tasks and report on progress
  • Software quality: The Agile developer is also responsible for the quality of the software he/she produces. The team takes responsibility for the quality of the work they produce instead of turning over some code to a separate and independent group for testing
  • Teamwork: This includes collaboration with all other team members with the aim to take shared responsibility for the overall efforts
  • Understanding user needs: This is about interacting with users as necessary to clarify requirements

Requirements

  • Bachelor's degree in Computer Science, Software Engineering, MIS or equivalent combination of education and experience
  • 8+ years of experience as Data Engineer on AWS Stack with experience on DevOps tool
  • AWS Solutions Architect or AWS Developer Certification required
  • Solid experience of AWS services such as CloudFormation, S3, Athena, Glue, Glue DataBrew, EMR/Spark, RDS, Redshift, DataSync, DMS, DynamoDB, Lambda, Step Functions, IAM, KMS, SM, EventBridge, EC2, SQS, SNS, LakeFormation, CloudWatch, Cloud Trail
  • Implement high velocity streaming solutions and orchestration using Amazon Kinesis, AWS Managed Airflow and AWS Managed Kafka (preferred)
  • Solid experience building solutions on AWS data lake/data warehouse
  • Analyze, design, development, and implementation of data ingestion pipeline in AWS
  • Knowledge implementing ETL/ELT for data solutions
  • End-to-end data solutions (ingest, storage, integration, processing, access) on AWS
  • Knowledge implementing RBAC strategy/solutions using AWS IAM and Redshift RBAC model
  • Build & implement CI/CD pipelines for EDP Platform using CloudFormation and Jenkins
  • Programming experience with Python, Shell scripting and SQL
  • Knowledge of analyzing data using SQL Stored procedures
  • Build automated data pipelines to ingest data from relational database systems, file systems, NAS shares to AWS relational databases such as Amazon RDS, Aurora, and Redshift
  • Build Automated data pipelines to ingest data from Rest APIs to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and Redshift.
  • Good Experience of DevOps practice
  • Experience with modern source code management and software repository systems (Bitbucket)
  • Experience with programming languages (Python)
  • Experience with scripting languages (Shell, Groovy)
  • Experience with API deployment for tooling integration
  • Experience using Jenkins, CloudBees (Pipeline as Code, Shared Libraries)
  • The ability to document exceptions/issues/action plans/meeting minutes/lessons learned accurately and in a timely fashion
  • Experience with administering DevOps tools in SaaS
  • Experience using DevOps Tools (SonarQube, Artifactory etc.)
  • Experience using build tools (Maven, MS Build and Gradle)
  • Experience using containers (Docker)
  • Experience using Atlassian suite (Jira, Confluence)
  • Experience of Infrastructure as Code using CloudFormation
  • Creating Jenkins CI pipelines to integrate Sonar/Security scans and test automation scripts
  • Part of DevOps QA and AWS team focusing on building CI/CD pipeline
  • Responsible for writing and maintaining Jenkins Pipelines

Apply for this position