Senior Data Engineer

Anywhere, Inc

Madison, NJ, United States

3 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

5 years minimum

Working hours

Regular working hours

Job source

indeed.com

Tech stack

Java (Programming Language) Application Programming Interfaces (APIs) Artificial Intelligence Amazon Web Services Amazon Elastic Compute Cloud Amazon S3 Code Review Continuous Delivery Continuous Integration Data Cleansing Information Engineering Extract Transform Load (ETL)

+22 more

Distributed Computing Environment Amazon DynamoDB PostgreSQL MongoDB MySQL NoSQL Online Transaction Processing Oracle (Applications) SQL Databases Data Streaming Enterprise Data Management Data Processing Data Ingestion Delivery Pipeline Apache Spark Gitlab Data Lakes Information Technology Apache Flink Apache Kafka Data Pipelines Jenkins

Job description

Design and build high performance, scalable data engineering solutions that meet needs of millions of Anywhere’s independent agents, brokers, as well as their home buyers and sellers;
Design and develop data ingestion pipelines for batch and real-time streaming of data from in-house OLTP systems and third-party data;
Collaborate with other Data Engineers at Anywhere for the build out of Next Generation Data Ingestion and Data Enrichment Pipelines;
Work with Data Engineering Team to design and develop a Data Lake to store and process terabytes of industry data;
Design and develop ETL pipelines to process data in Data Lake for use with APIs, reporting, analytics and related AI and data science;
Develop ETL data pipelines to build Enterprise Data Models for Property, Agent, Broker, office and other master partners and related clients and entities;
Design and develop CI/CD process for continuous delivery in AWS Cloud;
Design, develop, and test robust, scalable data platform components; and
Partner with a variety of data and delivery teams within the Anywhere organization, including product engineers to understand their data pipeline needs to assist them with coming up with innovative data engineering solutions.

Requirements

Must have a Bachelor’s degree in Computer Science, Information Technology or related field, plus five (5) years of experience in the job offered or in any occupation that includes the required experience and skills. Experience must include:
5 years of experience working with Scala or Java;
5 years of experience designing Real-Time Streaming Data Pipelines utilizing Apache Spark or Apache Flink and Apache Kafka;
5 years of development experience in Apache Spark for distributed data processing;
5 years of experience with AWS services including EC2, ECS, S3, EMR and Lambda;
5 years of experience in SQL databases, including PostgreSQL, MySQL, or Oracle, and NoSQL databases, including MongoDB or DynamoDB;
5 years of experience with Continuous Integration & Deployment (CI/CD) processes utilizing tools including AWS CodePipeline, Gitlab, or Jenkins; and
5 years of experience in serving in a lead role for onshore and offshore development teams, providing code reviews, architectural guidance, and technical leadership.