Senior Data Engineer

Experteer Overview In this role you will design and optimize scalable big data pipelines using Spark, Python, Scala, and Spark SQL within the Databricks ecosystem. You will work closely with cross-functional teams to translate business needs into robust data solutions and drive data-driven decision making. You will ensure data quality, monitor cluster health, and continuously improve processing capabilities in a distributed environment. This position offers the opportunity to shape big data platforms and mentor junior engineers in best practices, within a collaborative, standards-driven team. Join us to advance scalable data processing and deliver impactful insights. Compensation / Benefits * Design, develop, and deploy complex data processing workflows with Apache Spark, Scala, Python, and SparkSQL * Write efficient, reusable, and maintainable Spark applications * Identify and resolve performance bottlenecks in Spark jobs and data pipelines * Collaborate with data engineers, data scientists, and stakeholders to map business requirements to technical solutions * Review code and mentor junior engineers on coding standards and practices * Participate in architectural discussions and contribute to evolving big data platforms * Monitor the health, availability, and reliability of Spark clusters and data services * Stay up-to-date with Spark, Scala, Python, and related big data technologies to continuously improve capabilities * Optimize Spark jobs for performance and efficiency in distributed environments * Ensure data quality and consistency across multiple data sources Tasks * 3 to 5 years of experience * Proficiency in Python, Scala, and Spark * Experience with Spark SQL and the Databricks Spark ecosystem * Ability to design and optimize scalable data processing pipelines * Strong collaboration and ability to translate business requirements into technical solutions * Experience troubleshooting performance bottlenecks and maintaining data quality Key requirements *

decision and stakeholders to map business requirements to technical solutions * Review code and mentor junior engineers on coding standards and practices * Participate in architectural discussions and contribute to evolving big data platforms * Monitor the health, availability, and reliability of Spark clusters and data services * Stay up-to-date with Spark, Scala, Python, and related big data technologies to continuously improve capabilities * Optimize Spark jobs for performance and efficiency in distributed environments * Ensure data quality and consistency across multiple data sources Tasks * 3 to 5 years of experience * Proficiency in Python, Scala, and Spark * Experience with Spark SQL and the Databricks Spark ecosystem * Ability to design and optimize scalable data processing pipelines * Strong collaboration and ability to translate business requirements into technical solutions * Experience troubleshooting performance bottlenecks and maintaining data quality Key requirements *