BigQuery Data Engineer
Role details
Job location
Tech stack
Job description
Checkout.com is looking for an ambitious Data Engineer to join our Data Platform Team. Our team's mission is to build a world-class data platform that powers our products and analytics.
The Data Platform team is here to ensure internal stakeholders can easily collect, store, process and utilise data to build reports or products aiming to solve business problems. Our focus is on maximising the amount of time business stakeholders spend on solving business problems and minimising time spent on technical details around implementation, deployment, and monitoring of their solutions.
The core tech stack we use is based on AWS and GCP, using:
- Kafka as our message transport
- Flink for (near) real time processing
- Datahub as our catalog
- BigQuery as our warehouse
- Airflow (GCP Composer) for scheduling
- DBT for data transformation
- Montecarlo for monitoring
The platform here encompasses the end-to-end, first for real time / streaming use cases but also for our analytical / warehouse needs.
We're building for scale. As such, much of what we design and implement today will be the technology/infrastructure which will serve hundreds of teams and petabyte-level volumes of data., * You'll work as part of the team to build enablement components across the platform, as well as monitor and support the capabilities we offer.
- Develop and maintain documentation for data systems and processes.
- Participate in code and design reviews and provide constructive feedback.
- Wherever possible, automate workflows and processes, we're aiming for the platform to be as self-sustaining as possible.
- Stay up-to-date with the latest data and streaming engineering technologies and trends.
- Use that knowledge and subject matter expertise to mentor the more junior members of the team, and work with other "application" teams to provide guidance and best practice.
- Build light weight tooling and associated reference patterns to foster the adoption of the platform by enabling upstream teams and systems to easily publish and manipulate data and deploy applications using industry best practices
- Implement all the necessary infrastructure to enable end users to build, host, monitor and deploy their own applications
- Provide consultancy across the technology organisation to drive the adoption of the platform and unlock use-cases
- Promote data quality and governance as a first class citizen of the platform
Requirements
- Strong engineering background with a track record of implementing and owning components of a data platform
- Strong experience working with Cloud data warehousing technologies, in particular with BigQuery optimisation and FinOps
- Experience working with modern cloud-based stacks such as AWS, Azure or GCP
- Excellent SQL skills
- Strong programming skills with at least one of Python, Java, Scala or C#
- You're a mentor, raising the bar for your colleagues.
- You're a collaborator, always ready to dive in and partner to solve tough problems.
- You're a listener, and seek to understand the underlying problems, before pitching solutions
- You are able to drive through best practices by taking teams and organisations as a whole with you.
- You are a thought leader, so we'd love to see articles, podcasts, meetups or conference talks if you've done them