Data Engineer - £500 (Inside IR35) - Remote - 6 months

• Build and maintain integrations with CDS source systems using REST APIs, webhooks and event streams.
• Develop event-driven ingestion pipelines using AWS API Gateway, Lambda, EventBridge and SQS.
• Implement reliable state and watermark tracking using DynamoDB.
• Develop reconciliation mechanisms to identify and recover from missed or failed webhook deliveries without creating unnecessary polling load on source systems.
• Onboard new source systems onto the platform’s standard ingestion architecture as the CDS estate expands.
• Integrate with enterprise platforms such as Confluence, JIRA and other collaboration/ITSM systems where required.
• Undertake data migration and backfill activities, including using technologies such as AWS Database Migration Service (DMS).
• Develop and maintain Python-based integration and data pipeline components.
• Work closely with Knowledge Engineers to ensure ingested data has the required quality, structure and metadata for downstream graph population and AI use cases.
• Implement monitoring, logging, error handling and alerting to maintain high levels of integration reliability.
• Investigate and resolve data ingestion failures and integration issues.
• Contribute to the development and continuous improvement of the platform’s ingestion patterns, standards and engineering practices.
• Ensure solutions meet appropriate enterprise security, governance and regulatory requirements.

Mandatory Skills & ExperienceCore Engineering

• Strong experience in Data Engineering/Data Integration.
• Proven experience building integrations against source-system APIs, including:
• REST APIs
• Webhooks
• Event streams
• Strong Python development experience for data pipelines and integrations.
• Experience designing and implementing event-driven ingestion architectures.
• Hands-on AWS experience, particularly:
• API Gateway
• Lambda
• EventBridge
• SQS
• DynamoDB
• Experience implementing state management, watermarking and reconciliation patterns.
• Experience identifying and recovering from data delivery gaps and integration failures.
• Practical experience with data migration and backfill, ideally using AWS DMS or equivalent technologies.
• Experience working with multiple source systems and onboarding new data sources into an enterprise data platform.

Data Reliability

• Understanding of data quality, completeness and consistency.
• Experience implementing monitoring, logging and alerting for data pipelines.
• Ability to design resilient integrations with appropriate retry, failure handling and recovery mechanisms.
• Understanding of idempotency and reliable event processing.