Data Engineer

TechDigital Corporation

Princeton, NJ, United States

3 days ago

Apply on www.careerbuilder.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Working hours

Regular working hours

Job source

www.careerbuilder.com

Tech stack

Airflow Amazon Web Services Amazon Elastic Compute Cloud Amazon S3 Data Analysis Apache HTTP Server Cloud Computing Cloud Database Configuration Management Computer Programming Databases Continuous Delivery

+35 more

Continuous Integration Data Integration Extract Transform Load (ETL) Data Systems DevOps Amazon DynamoDB Python (Programming Language) PostgreSQL MySQL NoSQL Performance Tuning Systems Development Life Cycle Standard Sql Requirements Management Software Engineering SQL Databases Unstructured Data Workflow Management Systems Data Processing Scripting Apache Spark Reliability of Systems Electronic Medical Records AWS Lambda Git Cloudformation Pandas Pyspark AWS Glue Data Management Terraform Multiplatform Software Version Control Data Pipelines Programming Languages

Job description

*Data Pipeline Development - Design, build, and maintain scalable and reliable data pipelines to ingest, process, and transform data from various sources.

Data Integration & Management - Integrate structured and unstructured data from internal and external systems. *Ensure data quality, consistency, and availability across platforms.
Cloud-Based Data Engineering- Leverage AWS services (e.g., S3, Lambda, Glue, Redshift, EMR) to build cloud-native data solutions. *Optimize cloud resources for performance and cost-efficiency.
Programming & Automation - Use Python for data manipulation, ETL workflows, and automation of data tasks. *Develop reusable scripts and modules for data processing.
Collaboration & Stakeholder Engagement *Work closely with data scientists, analysts, and business teams to understand data needs. *Translate business requirements into technical solutions.
Monitoring & Optimization - Monitor data pipelines and troubleshoot issues proactively. *Continuously improve performance, scalability, and reliability of data systems.

Requirements

Programming Languages: Python (primary), SQL *Cloud Platforms: AWS (S3, Glue, Lambda, Redshift, EC2, EMR) *Data Tools: Apache Spark, Pandas, PySpark, Airflow *Databases: PostgreSQL, MySQL, NoSQL (e.g., DynamoDB) *ETL & Workflow Orchestration: AWS Glue, Apache Airflow *Version Control: Git *DevOps & CI/CD: Basic understanding of CI/CD pipelines and infrastructure as code (e.g., Terraform, CloudFormation), AWS Lambda, Amazon Simple Storage Service (S3), Amazon Web Services (AWS), Apache, Apache Spark, Automation, Cloud Computing, Continuous Deployment/Delivery, Continuous Improvement, Continuous Integration, Data Analysis, Data Management, Data Processing, Data Quality, Data Science, Database Extract Transform and Load (ETL), DevOps, Electronic Medical Records, Git, Identify Issues, Multiplatform/Cross-Platform, MySQL, Needs Assessment, NoSQL, Performance Management, Performance Tuning/Optimization, PostgreSQL, Programming Languages, Python Programming/Scripting Language, Requirements Management, SQL (Structured Query Language), Scalable System Development, Scripting (Scripting Languages), Software Engineering, Source Code/Configuration Management (SCM), Structured Data, Systems Reliability, Unstructured Data