Data Engineer
Role details
Job location
Tech stack
Job description
Responsibilities Build and optimise scalable data pipelines using PySpark and SQLDesign and implement ETL/ELT processes for batch and streaming dataDevelop data solutions using Databricks Lakehouse and Delta LakeIngest and integrate data from internal and external sources (e.g. Kafka, CDC)Optimise Spark jobs and data workflows for performance, scalability, and cost efficiencyManage infrastructure and environments using Terraform (IaC)Ensure data quality, monitoring, and reliabilityImplement governance and access controls (e.g. Unity Catalog)Deliver clean, structured, and accessible data for analytics and business useCollaborate with cross-functional teams to support analytics, reporting, and AI/ML initiatives
Requirements
Requirements Demonstrated experience in data engineering, with a proven ability to build scalable data solutionsStrong proficiency in Python and SQLHands-on experience with Apache Spark (including Structured Streaming)Experience with Databricks (Workflows, Delta Live Tables, Lakehouse architecture)Experience with cloud platforms (AWS, Azure, or GCP) Experience with Terraform or similar infrastructure-as-code toolsExperience working with structured and semi-structured data (e.g. JSON)Familiarity with CI/CD, modular development, and code documentationStrong communication skills and ability to work independently with a high level of ownership Desired SkillsDatabricks Certified Data Engineer (Associate or Professional)Experience with Kafka, DBT, or similar data toolsKnowledge of Scala or other programming languagesExperience working in an Agile environmentInterest in AI, machine learning, and data-driven technologies Preferred QualificationsA degree in STEM or Computer Science