Python/PySpark Engineer

Tata Consultancy Services Limited

Jersey City, NJ, United States

3 months ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Compensation

$100,000.0 - $125,000.0

Working hours

Regular working hours

Job source

indeed.com

Tech stack

Agile Methodology Unit Testing Microsoft Azure Cloud Computing Cloudera Impala Program Optimization Code Review Software Debugging Distributed Systems Apache Hadoop Hadoop Distributed File System Apache Hive

+20 more

Iterative and Incremental Development Python (Programming Language) Key Management Performance Tuning Scrum Methodology Cloudera Secure Coding Management of Software Versions Data Logging Data Ingestion Apache Spark Virtual Environment Caching Git Data Lakes Pyspark Git Flow Information Technology Jenkins Databricks

Job description

Implement schema evolution logic, validation frameworks, and resilient error-handling mechanisms.
Optimize Spark jobs for performance, cost efficiency, and production readiness.
Integrate all jobs into automated CI/CD pipelines, ensuring versioning and release governance.
Work closely with Ops teams to ensure proper monitoring, logging, and operational supportability.
Participate in Agile ceremonies, sprint planning, code reviews, and demo sessions

Requirements

Strong proficiency in Python, packaging, dependency management, and virtual environments.
Hands-on experience with PySpark, including Spark performance tuning (partitioning, caching, broadcast joins, memory optimization).
Expertise in data ingestion (batch/stream), schema management, and robust error-handling/retry logic.
Solid unit and integration testing practices, including data quality validations.
Experience with CI/CD pipelines (Azure DevOps/Jenkins), Git branching strategies, and artifact versioning.
Working experience with Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (Delta Lake, clusters, jobs, notebooks).
Knowledge of observability techniques: structured logging, metrics, tracing, and debugging in distributed systems.
Secure coding practices including secrets management, PII protection, and compliance-aware development.

Functional Skills

Strong documentation discipline for frameworks, reusable components, and best-practice patterns.
Effective collaboration with Cloud Architects and Data Ops to ensure stable and supportable pipelines.
Clear communication of technical ideas and solution approaches.
Comfort working in Agile environments with iterative development and frequent releases., Qualifications : BACHELOR OF COMPUTER SCIENCE You must create an Indeed account before continuing to the company website to apply

Benefits & conditions

Discretionary Annual Incentive. Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans. Family Support: Maternal & Parental Leaves. Insurance Options: Auto & Home Insurance, Identity Theft Protection. Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement. Time Off: Vacation, Time Off, Sick Leave & Holidays. Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing. Salary Range: $100,000 - $125,000 a year