Remote Data Engineer

• We design, build, and operate simple, repeatable ETL data pipelines across distributed processing environments, cloud platforms, and single-node environments.
• We develop prototyped and productionized code for ETL, data validation, and other data production processes.
• We develop understanding of the native tooling in GCP, Azure, AWS, or similar platforms.
• We produce code for data products including data matching, rule development, scans, and operational outputs.
• We participate in the development and maintenance of in-house code libraries.
• We undertake unit testing to support code development.
• We review business requirements and transform them into reusable production-ready code and effective data models.
• We resolve technical problems in databases, data processes, data products, and services.
• We initiate actions, monitor services, and identify trends to resolve issues.
• We apply correct techniques to normalize data and build robust relational structures.
• We perform basic data analysis for profiling, QA, and problem resolution.
• We support end users with data quality issues and communicate blockers and issues to develop solutions.
• We perform source system analysis and data profiling to confirm data quality and accurate metadata.
• We work with experts to develop validation frameworks for simple and complex data sources.
• We communicate with customers to provide progress updates and confirm requirements.
• We assist with the development of data pipelines, products, and automation processes using cloud data engineering tools.
• We contribute to tool performance logging and monitoring.
• We apply engineering best practice in data products and pipelines.
• We advise on technology changes in the engineering toolsets and platforms we work on.
• We support the transition to modern data platforms, including data warehouse and lakehouse environments.
• We engage with professional communities such as Data Science and Architecture to identify cross-community issues and escalate them as needed.
• We describe technical, data, pipeline, and production issues to colleagues with different specialisms.
• We communicate across teams to manage delivery expectations, blockers, priorities, and issues, and we escalate proactively.
• We work on very large-volume big data series using native engineering tools, practices, and coding approaches.
• We apply engineering standards across platforms and native toolsets and keep outputs up to date with those standards.

Technologies:

• Airflow
• AWS
• Azure
• Big Data
• BigQuery
• Bash
• Cloud
• CML
• Data Warehouse
• Databricks
• ETL
• GCP
• Git
• GitHub
• Hadoop
• Hive
• Informatica
• Support
• Java
• Kafka
• Kibana
• Machine Learning
• MySQL
• Neo4J
• NoSQL
• PostgreSQL
• Power BI
• Python
• PySpark
• Qlik
• SAP
• SAS
• SQL
• SQLite
• Scala
• Spark
• Tableau
• Unix
• NodeJS
• Data-Engineer

• We have 5+ years of data engineering experience.
• We have knowledge and experience in Azure or AWS cloud data solution provision.
• We are proficient in SQL.
• We can deliver complex visualisation, reporting, and dashboard solutions using tools such as Power BI.
• We have enterprise-scale experience with ETL tools such as Informatica or similar.
• We have experience in data modelling and transforming raw data into datasets and extracts.
• We have experience working in large-scale, complex organisations and migrating legacy capabilities.
• We have experience working in Agile environments.
• We can analyse information and evidence, identify problems and opportunities, and align recommendations with strategic business objectives.
• We have experience building team capability through role modelling, mentoring, and coaching.
• We can manage relationships with non-technical colleagues and work collaboratively and inclusively.
• We can design, write, and operate ETL pipelines in distributed processing environments.
• We understand distributed and cloud data processing principles and apply them to robust coding.
• We can write clean, efficient, and well-documented code for data processing tasks.
• We can use Git for version control and review merge requests.
• We can perform simple data and code analysis for quality assurance and issue resolution.
• We have experience with one or more programming languages such as Python/PySpark, SQL, Proc SQL, NoSQL, MySQL, SQLite, Spark SQL, Hive SQL, PostgreSQL, SAS, SAS E-guide, Scala, RegEx, Java, or R.
• We understand database structures, database integrity, and related practices.
• We have basic knowledge of applying database principles and SQL across tools such as SQL Server, Cloud SQL, BigQuery, Hive, and Athena.
• We can identify opportunities for innovation with new tools and uses of data.
• We have experience with BI development tools such as Plotly, R Shiny, Tableau, QlikView/Qlik Sense, Power BI, SAP, Business Objects, or MicroStrategy.
• We have experience with tools such as NiFi, HBase, Bash, PuTTY, Neo4J, Spark, Kafka, HDFS, Oozie, GitHub, Unix, Hadoop, Impala, DoJo, Flume, Elastic, Logstash, Kibana, Airflow, Glue, BigQuery, Athena, CML, Hive, Informatica, or CuteFTP.
• We can explain technical concepts in non-technical language.
• We can communicate effectively with internal and external stakeholders.
• We can build data products from multiple system feeds using different storage technologies and access methods.
• We can explain and apply data modelling concepts and principles.
• We can create and run simple unit tests.
• Desirable: certifications in AWS, Azure, Databricks, or related technologies.
• Desirable: experience with public sector data initiatives and compliance requirements.
• Desirable: knowledge of machine learning and artificial intelligence concepts.

We are Scrumconnect, a leading technology consultancy contributing to more than 20% of the UKs most significant citizen-facing public services. Our award-winning team has delivered more than 64 services in the past two years, reaching over 50 million citizens and helping save the taxpayer over £25 million. We are a collaborative community of consultants who value knowledge sharing and continuous learning as we solve complex challenges through advanced software engineering, human-focused design, and data-driven insights. This full-time Data Engineer role is based in the UK, with working time split across our HQ and hub locations, client sites, and home working depending on business, client, and team needs. We welcome candidates from all identities, attributes, and backgrounds and are committed to creating inclusive impact through our work.