Big Data Pyspark Developer
Futran Solutions
Tampa, United States of America
yesterday
Role details
Contract type
Permanent contract Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
English Experience level
SeniorJob location
Tampa, United States of America
Tech stack
API
Airflow
Data analysis
Bash
Big Data
Unix
Code Review
Continuous Integration
Data Validation
ETL
Database Queries
Database Testing
File Systems
Perl
Korn Shell
Shell Script
SQL Databases
Spark
PySpark
Build Process
Data Delivery
Software Coding
Data Pipelines
Requirements
- Exp 6 Years Must have good technical experience and should be able to provide technical solutions for multiple modules in parallel on need basis and bring the task to closure on time
- Unix SQL and Shell Scripting experience is a must have
- Expertise in Designing and developing scalable Apache spark ETL based Data processing pipelines
- Strong commandline knowledge in UnixLinux with Shell scripting using Bash Kornshell or Perl and File processing using awk scripts
- Expertise in SQL querying and complex joins
- Implementing comprehensive Spark based Data validation frameworks transforming large volumes of Financial data within the Project lifecycle
- Expertise with complex Data workflows with Apache AirFlow managing task dependencies SLAs etc to ensure timely data delivery and corresponding automated validation controls
- Strong Analytical skills and expertise on SparkSQL for Data analysis and validation ensuring the delivery of clean queryready datasets for business consumption
- Expertise in Data quality checks and monitoring
- Quality Engineering team where 70percent of effort will be for developing automation frameworks for testing Remaining 30percent effort will be on manual testing until its fully automated
- Handson with Automation Framework Design for ETL and API
- SME in Data Analysis Database testing Messaging queues
- Experience with coding standards code reviews source management build processes CICD pipeline