> Markdown version of [/jobs/ext/151428-senior-data-engineer-_-python-with-spark](https://www.wearedevelopers.com/jobs/ext/151428-senior-data-engineer-_-python-with-spark). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer _ Python with Spark - **Company:** Tata Consultancy Services Limited - **Location:** Jersey City, NJ, United States - **Experience:** Expert - **Salary:** $100,000.0 - $115,000.0 - **Contract:** Permanent contract - **Skills:** Adobe InDesign, Airflow, Amazon Web Services, Microsoft Azure, Big Data, Cloud Computing, Data as a Services, Data Architecture, Data Validation, Information Engineering, Extract Transform Load (ETL), Data Security, Data Systems, Data Warehousing, DevOps, Distributed Systems, Apache Hadoop, Apache Hive, JSON, Python (Programming Language), Performance Tuning, Standard Sql, SQL Databases, Unstructured Data, Parquet, Data Processing, Cloud Platform System, Data Ingestion, Apache Spark, Pyspark, Information Technology, Avro, Integration Frameworks, Non-relational Database, Data Pipelines - **Published:** May 14, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=6c8b463a5152c979 ## About the Role Do you have experience in Spark implementation?, Do you have a Bachelor's degree?, Must Have Technical/Functional Skills * Strong hands-on experience in Python programming for data engineering and data processing. * Extensive experience with Apache Spark (PySpark) for large-scale data processing and distributed computing. * Strong knowledge of SQL and experience working with relational and non-relational databases. * Experience in building and maintaining ETL/ELT pipelines for data ingestion and transformation. * Good understanding of data warehousing concepts, data modeling, and data architecture. * Experience working with big data technologies such as Hadoop ecosystem, Hive, or similar platforms. * Familiarity with cloud platforms (AWS, Azure, or GCP) and related data services. * Hands-on experience with data pipeline orchestration tools such as Airflow or similar. * Knowledge of data formats such as Parquet, Avro, JSON, and CSV. * Experience with performance tuning and optimization of Spark jobs and data pipelines. * Strong problem-solving skills and ability to work with cross-functional teams., Qualifications : BACHELOR OF COMPUTER SCIENCE ## Description * Design, develop, and maintain scalable data pipelines using Python and Spark. * Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data. * Work closely with data analysts, data scientists, and business stakeholders to understand data requirements. * Develop efficient and reusable data processing frameworks and components. * Perform data validation, cleansing, and transformation to ensure quality and consistency. * Optimize and tune Spark jobs and SQL queries for performance and scalability. * Collaborate with DevOps and platform teams to deploy and manage data solutions in cloud environments. * Ensure data security, governance, and compliance standards are met. * Troubleshoot production issues, perform root cause analysis, and implement fixes. * Participate in design discussions, contribute to data architecture decisions, and promote best practices. * Work in an Agile environment, supporting sprint activities and continuous improvement initiatives. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [Tips and Tricks for Working with JSON](https://www.wearedevelopers.com/videos/1229-tips-and-tricks-for-working-with-json) - [From event streaming to event sourcing 101](https://www.wearedevelopers.com/videos/91-from-event-streaming-to-event-sourcing-101) - [Introducing JSON Structure](https://www.wearedevelopers.com/videos/100219-introducing-json-structure) - [DevOps Maturity Check – a way to balance autonomy and alignment](https://www.wearedevelopers.com/videos/58-devops-maturity-check-a-way-to-balance-autonomy-and-alignment) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [The Best Job Search Websites of 2025](https://www.wearedevelopers.com/magazine/368-the-best-job-search-websites-of-2025)