Senior Data Engineer

Falcon Smart IT

London, UK

13 days ago

Apply on www.collegerecruiter.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Job source

www.collegerecruiter.com

Tech stack

Airflow Apache HTTP Server Computer Programming Continuous Integration Data as a Services Persistent Data Structure Relational Databases Python (Programming Language) PostgreSQL Metadata Meta-Data Management Cloud Services

+7 more

Workflow Management Systems Snowflake Apache Spark AWS Glue Performance Monitor Data Management Data Pipelines

Job description

Architect and Develop: Contribute to the platform’s architectural design and build integration, modelling, data persistence, and analytical systems.
Data Pipelines: Implement, maintain, and test robust data pipelines.
Metadata Management: Develop and manage metadata processes and tools.
Performance Monitoring: Ensure the stability and performance of data pipelines.
Data Quality: Implement tools for data curation, metadata management, and quality assurance.
Collaboration: Engage with business and technology teams to align the platform with organizational goals.

Requirements

Programming: 12+ years of experience in Python.
Cloud Expertise: Strong understanding of AWS services (e.g., Lambda, Step Functions, ECS).
Data Platforms: Hands-on experience with Snowflake and data stack technologies like Apache Iceberg and Spark.
Workflow Orchestration: Exposure to tools like Apache Airflow, Prefect, Dagster, or DBT.
Data Services: Familiarity with AWS Glue, Lake Formation, EMR, EventBridge, Athena, and similar services.
Metadata Tools: Experience with tools like Amundsen, Atlas, DataHub, OpenDataDiscovery, or Marquez.
RDBMS: Knowledge of PostgreSQL is a plus.
Industry Experience: Proven experience building enterprise-wide data and analytics systems, preferably in financial services or asset management.