Data Engineer
Role details
Job location
Tech stack
Job description
-
Design, develop, and optimize ETL/ELT pipelines in Databricks.
-
Implement real-time and batch data processing solutions using Apache Spark and Delta Lake.
-
Develop PySpark/Scala-based data transformation scripts for large-scale data processing.
-
Ensure data quality, performance tuning, and cost optimization within Databricks.
-
Implement CI/CD pipelines for Databricks workflows using Terraform, GitHub Actions, or Azure DevOps.
-
Monitor, troubleshoot, and optimize Databricks clusters, jobs, and queries.
-
Collaborate with Data Architects, Business Analysts, and DevOps teams to align solutions with business needs.
Requirements
-
Hands on experience in Databricks (AWS) - Delta Lake, Medallion architecture, Coding skills etc.
-
Proficiency in PySpark for data engineering workflows.
-
Hands-on experience with file handling, Kafka, APIs, and streaming data processing.
-
Proficiency in SQL for querying and performance tuning.
-
Experience in DevOps and CI/CD pipelines for Databricks.
-
Understanding of Data Governance, Security, and Access Control in Databricks.
Mandatory Skills: DataBricks - Data Engineering .
Experience: 3-5 Years .
Benefits & conditions
The expected compensation for this role ranges from $45,000 to $121,000 .
Final compensation will depend on various factors, including your geographical location, minimum wage obligations, skills, and relevant experience. Based on the position, the role is also eligible for Wipro's standard benefits including a full range of medical and dental benefits options, disability insurance, paid time off (inclusive of sick leave), other paid and unpaid leave options.