Big Data Pyspark Developer

Futran Solutions

Tampa, FL, United States

3 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Experience required

6 years minimum

Working hours

Regular working hours

Job source

dice.com

Tech stack

Application Programming Interfaces (APIs) Airflow Data Analysis Bash Shell Big Data Unix Code Review Continuous Integration Data Validation Extract Transform Load (ETL) Database Queries Database Testing

+11 more

File Systems Perl (Programming Language) Korn Shell Shell Script SQL Databases Apache Spark Pyspark Build Process Data Delivery Software Coding Data Pipelines

Requirements

Exp 6 Years Must have good technical experience and should be able to provide technical solutions for multiple modules in parallel on need basis and bring the task to closure on time
Unix SQL and Shell Scripting experience is a must have
Expertise in Designing and developing scalable Apache spark ETL based Data processing pipelines
Strong commandline knowledge in UnixLinux with Shell scripting using Bash Kornshell or Perl and File processing using awk scripts
Expertise in SQL querying and complex joins
Implementing comprehensive Spark based Data validation frameworks transforming large volumes of Financial data within the Project lifecycle
Expertise with complex Data workflows with Apache AirFlow managing task dependencies SLAs etc to ensure timely data delivery and corresponding automated validation controls
Strong Analytical skills and expertise on SparkSQL for Data analysis and validation ensuring the delivery of clean queryready datasets for business consumption
Expertise in Data quality checks and monitoring
Quality Engineering team where 70percent of effort will be for developing automation frameworks for testing Remaining 30percent effort will be on manual testing until its fully automated
Handson with Automation Framework Design for ETL and API
SME in Data Analysis Database testing Messaging queues
Experience with coding standards code reviews source management build processes CICD pipeline