PySpark Developer
Randstad UK
Charing Cross, United Kingdom
4 days ago
Role details
Contract type
Temporary contract Employment type
Full-time (> 32 hours) Working hours
Regular working hours Languages
English Experience level
Senior Compensation
£ 91KJob location
Remote
Charing Cross, United Kingdom
Tech stack
Amazon Web Services (AWS)
Amazon Web Services (AWS)
Continuous Integration
ETL
Data Mart
Software Debugging
Hive
Python
Modular Design
Performance Tuning
SAS (Software)
Macros
Spark
GIT
PySpark
Job description
- Code Conversion: Lead the end-to-end migration of SAS code (Base SAS, Macros, DI Studio) to PySpark using automated tools (SAS2PY) and manual refactoring.
- Pipeline Engineering: Design, build, and troubleshoot complex ETL/ELT workflows and data marts on AWS.
- Performance Tuning: Optimise Spark workloads for execution efficiency, partitioning, and cost-effectiveness.
- Quality Assurance: Implement clean coding principles, modular design, and robust unit/comparative testing to ensure data accuracy throughout the migration.
- Engineering Excellence: Maintain Git-based workflows, CI/CD integration, and comprehensive technical documentation.
Requirements
- PySpark (P3): 5+ years of hands-on experience writing scalable, production-grade PySpark/Spark SQL.
- AWS Data Stack (P3): Strong proficiency in EMR, Glue, S3, Athena, and Glue Workflows.
- SAS Knowledge (P1): Solid foundation in SAS to enable the understanding and debugging of legacy logic for conversion.
- Data Modeling: Expertise in ETL/ELT, dimensions, facts, SCDs, and data mart architecture.
- Engineering Quality: Experience with parameterisation, exception handling, and modular Python design.
Benefits & conditions
- Industry: Financial Services experience is highly desirable.
- Working Pattern: Fully remote with internal team collaboration days.
- Benefits: 33 days holiday entitlement (pro-rata).
