> Markdown version of [/jobs/ext/553718-python-big-data-developer](https://www.wearedevelopers.com/jobs/ext/553718-python-big-data-developer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Python/Big Data Developer

- **Company:** IBA InfoTech Inc.
- **Location:** Charlotte, NC, United States
- **Experience:** Experienced
- **Contract:** Permanent contract
- **Skills:** Application Programming Interfaces (APIs), Data Analysis, Big Data, Information Engineering, Apache Hadoop, Apache Hive, Python (Programming Language), Simple Data Format, SQL Databases, Parquet, Data Processing, Data Ingestion, Apache Spark, Data Lakes, Pyspark, Information Technology
- **Published:** June 14, 2026
- **Apply:** https://ibainfotech.com/python-big-data-developer

## About the Role

* Proven experience in developing solutions using Spark architecture and PySpark for data engineering pipelines, transformation, and aggregation of data from a variety of sources into the data lake.
 * At least 3 or more years of relevant experience in developing PySpark programs using APIs. Expertise in different file formats like parquet, ORC.
 * Experience with troubleshooting, fine-tuning Spark and python based applications for scalability and performance.
 * Experience in designing hive tables to handle velocity, variety and to handle huge volumes.
 * Experience in data ingestion, processing and analyzing data using Spark/SQL from disparate sources.
 * Knowledge in using Spark-Submit and Spark UI. Experience in creating and then performing operations on Spark RDD.
 * Experience in creating Spark Data Frames from RDD, HIVE and Parquet files and then performing Joins and Aggregations on Dataframes.
 * Experience in processing data from Python and other API modules.

## Description

* In-depth understanding and knowledge of Hadoop and Spark architecture and RDD transformation


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology)
- [Parquet, Delta, Iceberg & Ducklake - An introduction for developers](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers)
- [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)
- [The 13 Best Python Libraries for Developers in 2025](https://www.wearedevelopers.com/magazine/371-the-13-best-python-libraries-for-developers-in-2025)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)