BigData ,Python and Py Spark

Stanley David and Associates
Phoenix, United States of America
5 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Phoenix, United States of America

Tech stack

Airflow
Business Analytics Applications
Data analysis
Application Frameworks
Software Applications
Big Data
Google BigQuery
Computer Programming
Information Engineering
Data Integrity
ETL
Data Transformation
Data Systems
Data Warehousing
Software Debugging
Hadoop
Hive
Python
Shell
Performance Tuning
SQL Databases
Freeform SQL
Google Cloud Platform
Spark
PySpark
Stream Processing
Data Pipelines

Requirements

  • Design, develop, and maintain scalable ETL/ELT pipelines using PySpark, Airflow, and Google Cloud Platform-native tools.
  • Build and optimize data warehouses and analytics solutions in BigQuery.
  • Implement and manage workflow orchestration with Airflow/Cloud Composer.
  • Write complex SQL queries for data transformations, analytics, and performance optimization.
  • Ensure data reliability, security, and governance across pipelines.
  • Conduct performance tuning and cost optimization of BigQuery and PySpark workloads.
  • Collaborate with analysts and product teams to deliver reliable data solutions.
  • Troubleshoot, debug, and resolve production issues in large-scale data pipelines.
  • Contribute to best practices, reusable frameworks, and automation for data engineering.
  • 5+ years of experience within Data Engineering/ Data Warehousing using Big Data technologies will be a addon
  • Expert on Distributed ecosystem
  • Hands-on experience with programming using Python
  • Expert on Hadoop and Spark Architecture and its working principle
  • Hands-on experience on writing and understanding complex SQL(Hive/PySpark-dataframes),
  • optimizing joins while processing huge amount of data
  • Experience in UNIX shell scripting Ability to design and develop optimized Data pipelines for batch and real time data processing
  • Should have experience in analysis, design, development, testing, and implementation of system applications
  • Demonstrated ability to develop and document technical and functional specifications and analyze software and system processing flows.

Apply for this position