Data Engineer (Spark)

We are seeking a Data Engineer to support the implementation of a new CRM platform by designing, building, and maintaining scalable data pipelines and integrations. The role will play a key part in enabling customer insights, opportunity matching, reporting, analytics, and operational processes while ensuring data quality, governance, and scalability.

Key Responsibilities

• Analyse existing data sources and define data contracts with business consumers.
• Design, develop, and maintain data ingestion, transformation, and publishing pipelines.
• Build data models supporting customer profiles, opportunities, matching engines, tracking, and reporting.
• Implement data quality controls, reconciliation processes, and monitoring.
• Optimise ETL/ELT processes for performance, scalability, and cost efficiency.
• Work closely with Full Stack Developers to expose and consume data within the application.
• Document data lineage, models, business rules, dependencies, and recovery procedures.
• Ensure compliance with security, governance, and privacy standards.

Required Skills

• Strong experience with Apache Spark for large-scale data processing.
• Hands-on experience with AWS, including S3, RDS/PostgreSQL, IAM, and EMR.
• Solid knowledge of SQL for transformation, optimisation, and analytics.
• Experience designing and implementing ETL/ELT pipelines.
• Strong understanding of data modelling, data quality, metadata management, and traceability.
• Experience with Git, CI/CD, automated testing, monitoring, and alerting.
• Knowledge of security principles including IAM, encryption, secrets management, and least-privilege access.
• Experience working in Agile delivery environments.

• Strong experience with Apache Spark for large-scale data processing.
• Hands-on experience with AWS, including S3, RDS/PostgreSQL, IAM, and EMR.
• Solid knowledge of SQL for transformation, optimisation, and analytics.
• Experience designing and implementing ETL/ELT pipelines.
• Strong understanding of data modelling, data quality, metadata management, and traceability.
• Experience with Git, CI/CD, automated testing, monitoring, and alerting.
• Knowledge of security principles including IAM, encryption, secrets management, and least-privilege access.
• Experience working in Agile delivery environments.