Big Data Lead

SumasEdge Corporation

Philadelphia, PA, United States

13 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

6 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Java (Programming Language) Application Programming Interfaces (APIs) Amazon Web Services Microsoft Azure Big Data Cloud Computing Data as a Services Data Architecture Data Integration Extract Transform Load (ETL) Data Mining Apache HBase

+27 more

Python (Programming Language) Enterprise Messaging Systems MongoDB NoSQL Open Source Technology Performance Tuning RabbitMQ Scala (Programming Language) SQL Databases Web Services Extensible Markup Language (XML) Data Processing Google Cloud Cloud Platform System Data Ingestion Apache Spark Data Lakes Pyspark Apache Flume Cassandra Data Analytics Apache Kafka Spark Streaming Stream Processing Data Pipelines Databricks Programming Languages

Job description

Design & Implement Data ingestion and Data lakes-based solutions using Big Data Technologies.
The Tech Lead should be highly proficient in the use of Big Data / Open-Source Technologies and standard techniques of Data Integration, Data Manipulation.
Should be able to design and develop cost efficient and performant data pipelines in the cloud platform
Create data environment to support our data analytics, reporting and data science teams
Experience with integration of data from multiple data sources
Knowledge of various Data Pipeline techniques and frameworks
Performance optimization - need to monitor the complete process and apply necessary infrastructure changes to speed up the query execution.
Efficient data ingestion - Discovering patterns in data sets with data mining techniques and using different data ingestion APIs and inject data into the data lake as per need.

The Role offers:

Great opportunities to learn various tools and technologies used in a sophisticated data architecture within the Business Intelligence and Analytics Data Services
Gives an opportunity to showcase candidates strong analytical skills and problem-solving ability
An outstanding opportunity to re-imagine, redesign, and apply technology to add value to the business and operations
Grow into a Technical architect role over a period

Requirements

6+ Years hands on knowledge on SQL as well as SQL/NoSQL databases
Proficient in programming languages such as Python, PySpark, Scala and Java
Experience with Spark , Databricks
Working knowledge of XML, ETL, API and Web Services
Experience with integration of data from multiple data sources
Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
Knowledge of various ETL techniques and frameworks, such as Flume
Experience with various messaging systems, such as Kafka or RabbitMQ
Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
Working knowledge and experience in Big data services in one of the Cloud Provider will be good (AWS or Azure or Google Cloud Platform)
Experience in leading offshore teams