Python/PySpark Engineer

• Implement schema evolution logic, validation frameworks, and resilient error-handling mechanisms.
• Optimize Spark jobs for performance, cost efficiency, and production readiness.
• Integrate all jobs into automated CI/CD pipelines, ensuring versioning and release governance.
• Work closely with Ops teams to ensure proper monitoring, logging, and operational supportability.
• Participate in Agile ceremonies, sprint planning, code reviews, and demo sessions

• Strong proficiency in Python, packaging, dependency management, and virtual environments.
• Hands-on experience with PySpark, including Spark performance tuning (partitioning, caching, broadcast joins, memory optimization).
• Expertise in data ingestion (batch/stream), schema management, and robust error-handling/retry logic.
• Solid unit and integration testing practices, including data quality validations.
• Experience with CI/CD pipelines (Azure DevOps/Jenkins), Git branching strategies, and artifact versioning.
• Working experience with Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (Delta Lake, clusters, jobs, notebooks).
• Knowledge of observability techniques: structured logging, metrics, tracing, and debugging in distributed systems.
• Secure coding practices including secrets management, PII protection, and compliance-aware development.

Functional Skills

• Strong documentation discipline for frameworks, reusable components, and best-practice patterns.
• Effective collaboration with Cloud Architects and Data Ops to ensure stable and supportable pipelines.
• Clear communication of technical ideas and solution approaches.
• Comfort working in Agile environments with iterative development and frequent releases., Qualifications : BACHELOR OF COMPUTER SCIENCE You must create an Indeed account before continuing to the company website to apply

Discretionary Annual Incentive. Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans. Family Support: Maternal & Parental Leaves. Insurance Options: Auto & Home Insurance, Identity Theft Protection. Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement. Time Off: Vacation, Time Off, Sick Leave & Holidays. Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing. Salary Range: $100,000 - $125,000 a year