SC Cleared Python Developer

What is Citizen Event Analytics? Citizen Event Analytics (CEA) is a cross-benefit, cross-channel event history compiled from citizens' interaction, (telephony, face to face and digital), claim processing and support events.

CEA uses a pipeline that:

• Extracts event data from different sources
• Transforms the data into a usable and trusted resource
• Loads that data into the data asset that is accessible to Data users through the Uplifted Analytical Service (UAS).

Support DWP in the maintenance of the longitudinal event history data asset and associated data pipelines that forms Citizen Event Analytics. As directed by DWP, activities may include:

The resources will be expected to support:

• plan and lead development on sets of related stories

• have an understanding of the whole CEA system and take responsibility for teaching this to others(specific technical skills listed below in the 'Technical skill requirements' section

• work with other users, Product Owner and Business Analyst to understand what needs to be built

• coach and mentor more junior colleagues

• operate the ingest and publishing production pipelines/services, that are build and find ways to improve system robustness, resilience and stability

Key skills required:

• Understanding of data processing using Apache Spark
• Use of Python, SQL, and familiarity with PySpark
• Experience using Apache Airflow for task orchestration
• Understanding of EMR and reviewing output logs
• Use of Jupyter notebooks and/or Amazon Athena to query and validate data
• Data analysis to identify root cause of issues
• Understanding of dimensional data models and slowly changing dimensions/historic data capture
• Use of AWS console and services such as, but not limited to; CloudWatch, IAM, S3, Glue, ECR, EC2, EMR, Dynamo DB, LakeFormation
• Familiarity with Amazon Textract and Comprehend
• Understanding of both Server Side and client-side encryption
• Use of GitLab for source code management pipelines for CI/CD
• Use of GitLab Tags for component versioning in shared repositories
• Understanding of Docker and containerization of solutions
• IaC using Terraform
• Experience of understanding how customer expectations transition to applied functionality
• Familiarity with, and implementation of, DWP Engineering best practices
• Use of gitlab for release tagging and deployments
• Familiarity with basic data structures for constructing a solution

• Understanding of data processing using Apache Spark
• Use of Python, SQL, and familiarity with PySpark
• Experience using Apache Airflow for task orchestration
• Understanding of EMR and reviewing output logs
• Use of Jupyter notebooks and/or Amazon Athena to query and validate data
• Data analysis to identify root cause of issues
• Understanding of dimensional data models and slowly changing dimensions/historic data capture
• Use of AWS console and services such as, but not limited to; CloudWatch, IAM, S3, Glue, ECR, EC2, EMR, Dynamo DB, LakeFormation
• Familiarity with Amazon Textract and Comprehend
• Understanding of both Server Side and client-side encryption
• Use of GitLab for source code management pipelines for CI/CD
• Use of GitLab Tags for component versioning in shared repositories
• Understanding of Docker and containerization of solutions
• IaC using Terraform
• Experience of understanding how customer expectations transition to applied functionality
• Familiarity with, and implementation of, DWP Engineering best practices
• Use of gitlab for release tagging and deployments
• Familiarity with basic data structures for constructing a solution