> Markdown version of [/jobs/ext/202597-python-pyspark-engineer](https://www.wearedevelopers.com/jobs/ext/202597-python-pyspark-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Python/PySpark Engineer

- **Company:** Tata Consultancy Services Limited
- **Location:** Jersey City, NJ, United States
- **Salary:** $100,000.0 - $125,000.0
- **Contract:** Permanent contract
- **Skills:** Agile Methodology, Unit Testing, Microsoft Azure, Cloud Computing, Cloudera Impala, Program Optimization, Code Review, Software Debugging, Distributed Systems, Apache Hadoop, Hadoop Distributed File System, Apache Hive, Iterative and Incremental Development, Python (Programming Language), Key Management, Performance Tuning, Scrum Methodology, Cloudera, Secure Coding, Management of Software Versions, Data Logging, Data Ingestion, Apache Spark, Virtual Environment, Caching, Git, Data Lakes, Pyspark, Git Flow, Information Technology, Jenkins, Databricks
- **Published:** May 19, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=dc6f909cb7ec3483

## About the Role

* Strong proficiency in Python, packaging, dependency management, and virtual environments.
 * Hands-on experience with PySpark, including Spark performance tuning (partitioning, caching, broadcast joins, memory optimization).
 * Expertise in data ingestion (batch/stream), schema management, and robust error-handling/retry logic.
 * Solid unit and integration testing practices, including data quality validations.
 * Experience with CI/CD pipelines (Azure DevOps/Jenkins), Git branching strategies, and artifact versioning.
 * Working experience with Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (Delta Lake, clusters, jobs, notebooks).
 * Knowledge of observability techniques: structured logging, metrics, tracing, and debugging in distributed systems.
 * Secure coding practices including secrets management, PII protection, and compliance-aware development.

Functional Skills

 * Strong documentation discipline for frameworks, reusable components, and best-practice patterns.
 * Effective collaboration with Cloud Architects and Data Ops to ensure stable and supportable pipelines.
 * Clear communication of technical ideas and solution approaches.
 * Comfort working in Agile environments with iterative development and frequent releases., Qualifications : BACHELOR OF COMPUTER SCIENCE
You must create an Indeed account before continuing to the company website to apply

## Description

* Implement schema evolution logic, validation frameworks, and resilient error-handling mechanisms.
 * Optimize Spark jobs for performance, cost efficiency, and production readiness.
 * Integrate all jobs into automated CI/CD pipelines, ensuring versioning and release governance.
 * Work closely with Ops teams to ensure proper monitoring, logging, and operational supportability.
 * Participate in Agile ceremonies, sprint planning, code reviews, and demo sessions


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [HTTP headers that make your website go faster](https://www.wearedevelopers.com/videos/1676-http-headers-that-make-your-website-go-faster)
- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)
- [Event based cache invalidation in GraphQL](https://www.wearedevelopers.com/videos/433-event-based-cache-invalidation-in-graphql)

## Related Articles

- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Résumé-Driven Development: How IT trends affect the job market for software developers](https://www.wearedevelopers.com/magazine/59-resume-driven-development-how-it-trends-affect-the-job-market-for-software-developers)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)
- [The 13 Best Python Libraries for Developers in 2025](https://www.wearedevelopers.com/magazine/371-the-13-best-python-libraries-for-developers-in-2025)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)