> Markdown version of [/jobs/ext/3332442-data-engineer-commodities](https://www.wearedevelopers.com/jobs/ext/3332442-data-engineer-commodities). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Data Engineer - Commodities

- **Company:** Millennium Management LLC
- **Location:** London, UK
- **Experience:** Expert
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Airflow, Automation of Tests, Big Data, Databases, Information Engineering, Data Integrity, Extract Transform Load (ETL), Data Transformation, Data Warehousing, Github, Python (Programming Language), Standard Sql, SQL Databases, Git, Pytest, Production Code, Data Delivery, Data Pipelines
- **Published:** September 28, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=a5bbe81229aa39eb

## About the Role

* 5+ years of experience in data engineering, analytics engineering, or a similar role building and maintaining ETL pipelines.
 * Strong Python and SQL skills, including experience working with large datasets and complex data transformations.
 * Hands-on experience with Airflow or comparable workflow orchestration tools.
 * Experience with Git, CI/CD pipelines such as GitHub Actions, and automated testing frameworks such as PyTest.
 * Strong attention to detail, data quality, and documentation, with the ability to identify edge cases and protect data integrity.
 * Ability to work independently, communicate effectively with technical and non-technical stakeholders, and manage multiple concurrent initiatives.
 * Knowledge of commodities markets and data, including weather, supply and demand, storage, freight, or flows, is preferred.
 * Experience with data warehousing, columnar storage formats, analytic databases, data catalog or governance tools, or financial services, trading, or research-driven environments is preferred.

## Description

* Design, implement, and maintain end-to-end ETL workflows in Python and SQL to ingest and transform commodities data from multiple vendor and internal sources.
 * Build and maintain standardized data models, schemas, and metadata that make commodities datasets easy to understand, discover, and reuse.
 * Schedule, monitor, and manage data pipelines using Airflow or similar workflow orchestration tools to ensure reliable, timely data delivery.
 * Implement validation, reconciliation, and anomaly-detection controls to maintain data completeness, accuracy, and consistency.
 * Apply AI to automate schema inference across structured and semi-structured data sources, manage schema drift, and accelerate scalable ingestion development.
 * Use AI-driven data-quality, observability, and documentation capabilities to detect anomalies, monitor data health, and produce clear lineage and technical documentation.
 * Maintain high-quality, production-ready code and repeatable deployments through Git, GitHub Actions, PyTest, and automated testing practices.
 * Partner with commodities Portfolio Managers, researchers, data analysts, and data strategists to refine datasets, definitions, and documentation based on evolving business needs.


## Related Videos

- [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next)
- [pytest: Simple, rapid and fun testing with Python](https://www.wearedevelopers.com/videos/213-pytest-simple-rapid-and-fun-testing-with-python)
- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering)
- [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models)
- [Agents for the Sake of Happiness](https://www.wearedevelopers.com/videos/1387-agents-for-the-sake-of-happiness)

## Related Articles

- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)