> Markdown version of [/jobs/ext/4602175-data-engineer](https://www.wearedevelopers.com/jobs/ext/4602175-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Data Engineer

- **Company:** Tata Consultancy Services Limited
- **Location:** Eden Prairie, MN, United States
- **Experience:** Expert
- **Salary:** $110,000 - $120,000
- **Contract:** Permanent contract
- **Skills:** Application Programming Interfaces (APIs), Amazon Web Services, Microsoft Azure, Cloud Computing, Code Review, Databases, Extract Transform Load (ETL), Distributed Systems, NoSQL, Scrum Methodology, Productivity Software, Cloud Services, Standard Sql, Software Engineering, Unstructured Data, Scripting, Cloud Platform System, Data Ingestion, Autoscaling, Apache Spark, AI Coding Agents, Event Driven Architecture, Containerization, Pyspark, Kubernetes, Information Technology, Apache Kafka, Azure AKS, Spark Streaming, Data Management, Api Design, Stream Processing, Data Pipelines, Docker, Microservices
- **Published:** November 6, 2026
- **Apply:** https://jobs.localjobnetwork.com/apply/add/88793912/1

## About the Role

* Bachelor's degree in Computer Science, Engineering.
 * 10+ years of hands-on software engineering experience in deploying distributed systems and cloud-native
solutions.

 * 9+ years of experience operating in a consulting or advisory capacity, including structuring ambiguous
problems, developing solution recommendations, and delivering against client or business objectives.

 * Experience working with APIs, data pipelines, or distributed systems.
 * Experience working with at least one cloud platform (e.g., AWS, Azure, or GCP).
 * Demonstrated experience working in client-facing engagements, including partnering directly with
stakeholders to understand needs, communicate solutions, and support delivery in real-world
environments.

 * Strong hands-on experience with Microsoft Azure and cloud-native data engineering solutions.
 * Hands-on experience with Kubernetes, preferably Azure Kubernetes Service (AKS), for deploying, scaling,
and operating containerized data-processing applications and services.

 * Experience with Docker/containerization and Kubernetes concepts, including Pods, Deployments, Services,
ConfigMaps, Secrets, resource management, autoscaling, and workload troubleshooting.

 * Experience deploying and supporting data pipelines, APIs, microservices, or Spark-based workloads in
Kubernetes/AKS environments, with a focus on reliability, scalability, and production readiness.

 * Demonstrated advanced capability using AI-assisted software development and productivity tools.
 * Proven ability to lead client engagements, including discovery, solution design, and delivery.
 * Strong stakeholder management skills, with the ability to influence both technical and executive audiences.
 * Experience structuring ambiguous problems and driving them to clear, actionable solutions.
 * Experience working with enterprise-scale data platforms.
 * Excellent communication and storytelling skills, with the ability to adapt messaging to different audiences.
Programming & Scripting

 * Languages: PySpark and Scala
 * Database technologies: SQL and NoSQL, BACHELOR OF ENGINEERING

## Description

1. Develop scalable batch and real-time data processing solutions using Spark and Scala.
 2. Design and implement reusable ETL frameworks and ingestion pipelines.
 3. Process large-scale structured and unstructured datasets.
 4. Build optimized Spark jobs for high-performance distributed computing.
 5. Tune Spark applications for performance and efficiency.
 6. Optimize partitioning strategies and query execution plans.
 7. Troubleshoot data skew, shuffle bottlenecks, and memory issues.
Page 1
Final approved job description

 8. Improve overall data pipeline reliability and scalability.
 9. Design and develop real-time data pipelines using Spark Structured Streaming.
 10. Integrate data from APIs, databases, flat files, and event streams.
 11. Work with Kafka-based event-driven architectures.
 12. Participate in Agile delivery activities, including sprint planning.
 13. Conduct code reviews and mentor junior engineers.
 14. Drive engineering best practices across teams