Senior Data Engineer

Retelligence

London, UK

1 day ago

Apply on www.collegerecruiter.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Languages

English

Job source

www.collegerecruiter.com

Tech stack

Airflow Amazon Web Services Databases Information Engineering Data Infrastructure Fault Tolerance Python (Programming Language) NoSQL Performance Tuning Standard Sql Data Streaming Workflow Management Systems

+10 more

Data Logging Scripting Google Cloud Cloud Platform System Data Ingestion Kubernetes Real Time Data Apache Kafka Video Streaming Data Pipelines

Job description

System Architecture & Development: Lead the design, development, and maintenance of scalable, high-performance, real-time data pipelines and core infrastructure components within the cloud environment.
Data Integration: Ensure seamless, low-latency data flow by integrating and unifying diverse data sources across the organization.
Data Modeling & Quality: Build and optimize robust data models specifically for complex querying and analytical use cases, and ensure data quality, integrity, and security across all systems.
Reliability & Monitoring: Construct highly available, fault-tolerant systems for high-volume data ingestion and processing, implementing effective monitoring, logging, and alerting mechanisms.
Performance Optimization: Continuously monitor, tune, and optimize pipeline scalability and performance to meet stringent throughput and latency standards.
Cross-Functional Alignment: Partner directly with technical and business teams to ensure the data infrastructure directly supports strategic objectives.

Requirements

Production Experience: Extensive hands-on experience in Data Engineering, specifically a proven track record of successfully architecting, building, and managing production-grade real-time data pipelines across multiple large-scale initiatives.
Cloud Platform Specialization: Mandatory deep, practical experience leveraging Google Cloud Platform (GCP) and its associated tools for the deployment and management of real-time data ingestion and processing services. (While AWS knowledge is valuable, GCP expertise is primary.)
Distributed Streaming: Strong familiarity with distributed streaming technologies, such as Kafka or similar platforms.
Programming & Scripting: Expert proficiency in Python for development, coupled with the ability to optimize and refactor data pipelines for improved performance and scalability.
Database Expertise: Advanced knowledge of SQL is mandatory, alongside a strong understanding of various database technologies (NoSQL, time-series databases) and deep competence in data modeling principles.
Workflow Orchestration: Strong working knowledge of industry-standard data orchestration tools (e.g., Apache Airflow or Kubernetes)., We are seeking a highly collaborative professional with strong problem-solving skills, capable of thriving in a fast-paced environment. Exceptional written and verbal communication skills in English are required for effective technical leadership and documentation.