Data Engineer

1. Develop scalable batch and real-time data processing solutions using Spark and Scala.
2. Design and implement reusable ETL frameworks and ingestion pipelines.
3. Process large-scale structured and unstructured datasets.
4. Build optimized Spark jobs for high-performance distributed computing.
5. Tune Spark applications for performance and efficiency.
6. Optimize partitioning strategies and query execution plans.

7. Troubleshoot data skew, shuffle bottlenecks, and memory issues. Page 1 Final approved job description

8. Improve overall data pipeline reliability and scalability.
9. Design and develop real-time data pipelines using Spark Structured Streaming.
10. Integrate data from APIs, databases, flat files, and event streams.
11. Work with Kafka-based event-driven architectures.
12. Participate in Agile delivery activities, including sprint planning.
13. Conduct code reviews and mentor junior engineers.
14. Drive engineering best practices across teams

• Bachelor’s degree in Computer Science, Engineering.

• 10+ years of hands-on software engineering experience in deploying distributed systems and cloud-native solutions.

• 9+ years of experience operating in a consulting or advisory capacity, including structuring ambiguous problems, developing solution recommendations, and delivering against client or business objectives.

• Experience working with APIs, data pipelines, or distributed systems.
• Experience working with at least one cloud platform (e.g., AWS, Azure, or GCP).

• Demonstrated experience working in client-facing engagements, including partnering directly with stakeholders to understand needs, communicate solutions, and support delivery in real-world environments.

• Strong hands-on experience with Microsoft Azure and cloud-native data engineering solutions.

• Hands-on experience with Kubernetes, preferably Azure Kubernetes Service (AKS), for deploying, scaling, and operating containerized data-processing applications and services.

• Experience with Docker/containerization and Kubernetes concepts, including Pods, Deployments, Services, ConfigMaps, Secrets, resource management, autoscaling, and workload troubleshooting.

• Experience deploying and supporting data pipelines, APIs, microservices, or Spark-based workloads in Kubernetes/AKS environments, with a focus on reliability, scalability, and production readiness.

• Demonstrated advanced capability using AI-assisted software development and productivity tools.
• Proven ability to lead client engagements, including discovery, solution design, and delivery.
• Strong stakeholder management skills, with the ability to influence both technical and executive audiences.
• Experience structuring ambiguous problems and driving them to clear, actionable solutions.
• Experience working with enterprise-scale data platforms.

• Excellent communication and storytelling skills, with the ability to adapt messaging to different audiences. Programming & Scripting

• Languages: PySpark and Scala
• Database technologies: SQL and NoSQL, BACHELOR OF ENGINEERING

Salary Range: $110,000- $120,000 a year TCS Employee Benefits Summary: Discretionary Annual Incentive. Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans. Family Support: Maternal & Parental Leaves. Insurance Options: Auto & Home Insurance, Identity Theft Protection. Convenience & Professional Growth: Commuter Benefits & Certification & amp; Training Reimbursement. Time Off: Vacation, Time Off, Sick Leave & Holidays.