Big Data Lead

Here Technologies
Newtown Square, United States of America
yesterday

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Newtown Square, United States of America

Tech stack

Java
API
Amazon Web Services (AWS)
Azure
Big Data
Cloud Computing
Data as a Services
Data Architecture
Data Integration
ETL
Data Mining
HBase
Python
Enterprise Messaging Systems
MongoDB
NoSQL
Open Source Technology
Performance Tuning
RabbitMQ
Scala
SQL Databases
Web Services
XML
Data Processing
Google Cloud Platform
Cloud Platform System
Data Ingestion
Spark
Data Lake
PySpark
Apache Flume
Information Technology
Cassandra
Data Analytics
Kafka
Spark Streaming
Stream Processing
Data Pipelines
Databricks
Programming Languages

Job description

  • Design & Implement Data ingestion and Data lakes-based solutions using Big Data Technologies.
  • The Tech Lead should be highly proficient in the use of Big Data / Open-Source Technologies and standard techniques of Data Integration, Data Manipulation.
  • Should be able to design and develop cost efficient and performant data pipelines in the cloud platform.
  • Create data environment to support our data analytics, reporting and data science teams.
  • Experience with integration of data from multiple data sources.
  • Knowledge of various Data Pipeline techniques and frameworks.
  • Performance optimization - need to monitor the complete process and apply necessary infrastructure changes to speed up the query execution.
  • Efficient data ingestion - Discovering patterns in data sets with data mining techniques and using different data ingestion APIs and inject data into the data lake as per need.

The Role offers

  • Great opportunities to learn various tools and technologies used in sophisticated data architecture within the Business Intelligence and Analytics Data Services.
  • Gives an opportunity to showcase candidates strong analytical skills and problem-solving ability.
  • An outstanding opportunity to re-imagine, redesign, and apply technology to add value to the business and operations.
  • Grow into a Technical architect role over a period.

Requirements

  • 6+ Years hands on knowledge on SQL as well as SQL/NoSQL databases.
  • Proficient in programming languages such as Python, PySpark, Scala and Java
  • Experience with Spark , Databricks.
  • Working knowledge of XML, ETL, API and Web Services.
  • Experience with integration of data from multiple data sources.
  • Experience with NoSQL databases, such as HBase, Cassandra, MongoDB.
  • Knowledge of various ETL techniques and frameworks, such as Flume.
  • Experience with various messaging systems, such as Kafka or RabbitMQ.
  • Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming.
  • Working knowledge and experience in Big data services in one of the Cloud Provider will be good (AWS or Azure or Google Cloud Platform).
  • Experience in leading offshore teams.

Essential Qualification

  • BE/Btech in Computer Science, Engineering or relevant field.

Apply for this position