Lead Software Engineer - (Data Platforms)
Role details
Job location
Tech stack
Job description
Key Responsibilities:Build and optimise scalable data pipelines using Databricks and Apache Spark (PySpark).Ensure performance, scalability, and compliance.Collaborate on requirements, design, and backlog refinement.Promote engineering best practices including CI/CD, code reviews, and testing.Research and introduce new tools, technologies, and methodologies.
Requirements
Ideal experience:Experience with efficient, reliable data pipelines that improve time-to-insight.Knowledge of secure, auditable, and compliant data workflows.Know how on optimising performance and reducing costs through Spark and Databricks tuning.Be able to create reusable, well-documented tools enabling collaboration across teams.A culture of engineering excellence driven by mentoring and high-quality practices. Preferred ExperienceDatabricks in a SaaS environment, Spark, Python, and database technologies.Event-driven and distributed systems (Kafka, AWS SNS/SQS, Java, Python).Data Governance, Data Lakehouse/Data Intelligence platforms.AI software delivery and AI data preparation.