Data Engineer

J&C Associates
London, UK
19 days ago

Role details

Contract type
Temporary contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
2 years minimum
Working hours
Regular working hours

Tech stack

Java (Programming Language) A/B Testing Application Programming Interfaces (APIs) Agile Methodology Airflow Amazon Web Services Microsoft Azure Databases System Configuration Continuous Integration Data Security Data Warehousing
+17 more
Relational Databases DevOps Python (Programming Language) PostgreSQL Cloud Services Web Analytics Scripting Git Pandas Data Lakes Pyspark Data Analytics Splunk Text Files Docker Jenkins Databricks

Job description

  • As a senior data platform engineer, you will build and maintain scalable, rock solid self-serve solutions for data analysts and support them by understanding the content and context of data as well as collaborating with them to figure out the best way to Extract, Transform, Load and access it.
  • The ideal candidate will have a passion for simplifying and automating workflows, for making things better, for being efficient and helping the team do more with less.

What you will do:

  • Work with data analysts to determine the best schemas to translate available data sources into business domains.
  • Build connectors to integrate with Postgres-based RDBMSs, cloud solutions like AWS, Azure.
  • Write scripts to retrieve data from RDBMSs, logging systems (e.g., Splunk), APIs, text files.
  • Work with external partners to facilitate data access.
  • Make data workflows more efficient and automate them when possible.
  • Lead sophisticated, large-scale projects in an Agile environment using Git, Jenkins, Docker.

Requirements

  • 5-10 years of Industry Experience.
  • Passion for continuous betterment (i.e., better workflows, automation, efficiency).
  • Experience with Data Warehouse or Data Lake, choosing database architectures, optimizing SQL queries.
  • Experience in a DevOps role, configuring and launching Docker containers, Jenkins for CI/CD, airflow for scheduling.
  • 2-5 years of proven experience coding in Python/Java or any scripting language.
  • Excellent in teamwork and communication skills.

Preferred skills:

  • Experience and knowledge of Digital Analytics or Digital Marketing products.
  • Experience with storage, compute services (e.g., databricks) on a major cloud vendor like AWS or Microsoft Azure is a big plus.
  • Knowledge of statistics or experience with pandas, pyspark or R, to better connect with analysts.
  • Knowledge of A/B testing or data-driven experimentation frameworks.
  • Experience in a B2B environment is a plus.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www1.jobdiva.co.uk

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

2:03 min

Accelerating pandas dataframes using cudf module plugins

Ankit Patel Ankit Patel · WWC 2024

2:38 min

Establishing comprehensive monitoring and log management

Michael Eder +1 · LIVE

6:21 min

Investigating push inefficiencies with upstream Git experts

Jonathan Creamer · Coffee With Developers

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

6:58 min

Analyzing production code coverage data using pandas

Markus Harrer Markus Harrer · WWC 2021

56 sec

Favorite git commands and the importance of patch commits

Eileen Uchitelle Eileen Uchitelle +1 · Coffee With Developers

Videos

See all

Related articles

See all