> Markdown version of [/jobs/ext/2730078-artificial-general-intelligence](https://www.wearedevelopers.com/jobs/ext/2730078-artificial-general-intelligence). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Artificial General Intelligence

- **Company:** Agi
- **Location:** UK (Remote available)
- **Contract:** Permanent contract
- **Skills:** Clean Code Principles, Airflow, Computer Clusters, Continuous Integration, Data Deduplication, Distributed Data Store, Python (Programming Language), Machine Learning, Cloud Services, Prometheus, Web Crawlers, Large Language Models, Grafana, Git, Pyspark, Kubernetes, Dask, Slurm, Stream Processing, Docker
- **Published:** September 5, 2026
- **Apply:** https://startup.jobs/member-of-engineering-pre-training-data-engineering-poolside-7694404

## About the Role

* Strong background in building production-grade, distributed data systems for machine learning, with experience in:
 + Orchestration: Slurm, Airflow, or Dagster
 + Observability & Reliability: CI/CD, Grafana, Prometheus, etc.
 + Infra: Git, Docker, k8s, cloud managed services
 + Batch inference (ex: vLLM)
 + Performance obsession, especially with large-scale GPU clusters and distributed pipelines
 * Expert-level python knowledge and ability to write clean and maintainable code
 * Strong algorithmic foundations
 * Proficiency with libraries like Polars, Dask, or PySpark
 * Nice to have:
 + Experience in building trillion-scale SOTA pretraining datasets
 + Experience translating research to production at scale
 + Experience with OCR, web crawling, or evals
 + Prior experience pre-training LLMs

## Description

You will be a core member of our Pretraining Data team, responsible for building and scaling our Model Factory: our system for quickly training, scaling, and experimenting with our foundation models. This is a hands-on role where your #1 mission is to architect and maintain the high-performance pipelines that transform trillions of raw tokens into the high-quality dataset "fuel" our models require.

To enable us to conduct and implement latest research, you'll be engineering the ingestion, deduplication, and streaming systems that handle petabyte-scale data. You will bridge the gap between raw web crawls and our GPU clusters, directly influencing model performance through superior data modeling, algorithmic sorting, and distributed pipeline optimization. You will be closely collaborating with other teams like Pretraining, Postraining, Evals, and Product to generate high-quality datasets that map to missing model capabilities and downstream use cases., * Build and maintain high-performance pipelines for trillions of tokens.

 * Deliver diverse and high quality datasets for pre-training foundation models.
 * Closely work with other teams such as Pretraining, Posttraining, Evals and Product to to ensure alignment on the quality of the models delivered.


## Related Videos

- [Running Secure Life Science Research at Scale using Hybrid GPU HPC and Kubernetes 🧬](https://www.wearedevelopers.com/videos/100355-running-secure-life-science-research-at-scale-using-hybrid-gpu-hpc-and-kubernetes)
- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [Agentic AI in Go](https://www.wearedevelopers.com/videos/100274-agentic-ai-in-go)
- [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews)
- [AI Factories at Scale](https://www.wearedevelopers.com/videos/1139-ai-factories-at-scale)
- [Stop Committing Your Secrets - GIt Hooks To The Rescue!](https://www.wearedevelopers.com/videos/573-stop-committing-your-secrets-git-hooks-to-the-rescue)

## Related Articles

- [MLOps And AI Driven Development](https://www.wearedevelopers.com/magazine/82-mlops-and-ai-driven-development)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [What is Agentic Programming and Why Should Developers Care?](https://www.wearedevelopers.com/magazine/625-what-is-agentic-programming-and-why-should-developers-care)
- [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models)
- [Got AI ideas but no money? Here are 10 free ways to level up your AI skills with Google Cloud](https://www.wearedevelopers.com/magazine/600-got-ai-ideas-but-no-money-here-are-10-free-ways-to-level-up-your-ai-skills-with-google-cloud)
- [MLOps – What’s the deal behind it?](https://www.wearedevelopers.com/magazine/125-mlops-what-s-the-deal-behind-it)