> Markdown version of [/magazine/610-who-owns-your-content-in-the-age-of-llms](https://www.wearedevelopers.com/magazine/610-who-owns-your-content-in-the-age-of-llms). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Who Owns Your Content in the Age of LLMs? **By:** [Benjamin Ruschin](https://www.wearedevelopers.com/@benjamin-ruschin) **Published:** August 1, 2025 AI has changed the web forever. Large language models (LLMs) are changing how information is produced, shared and consumed on the web. In fact, estimates suggest that now [more than half of all web traffic is made up of bots](https://www.independent.co.uk/tech/bots-internet-traffic-ai-chatgpt-b2733450.html), with a sizable amount of that traffic coming from so-called bad bots, bots carrying out malicious or unwanted activity. For developers, this presents new kinds of challenges. In the past, the threat of content theft came only from humans. However, developers must now ensure their content (be it code, video content, tutorials or otherwise) isn’t silently indexed and ingested, repurposed, or even monetised - without their consent - by AI bots. In this article, we’ll show you how to find out what LLMs know about you, how to block bots from crawling your content, and how to monetise it by charging bots each time they crawl. ## 1. The Manual Way To start off, let’s do things the manual way. This approach is by no means scientific, but if you ask your chosen LLM, “_Without using search, relying only on indexed data, what do you know about [me / my website]?”_ or _“What can you tell me about GitHub user [your username]?”_ you’ll get an informal sense of whether your content has been indexed along the way. At this point, it’s worth saying that this approach won’t give you a dataset, but it’s a quick way to detect whether your content is already circulating inside LLM responses. Also, there’s a strong possibility the LLM will hallucinate or simply fabricate a few aspects of its response, so take it with a relatively large lump of salt. ## 2. Check If Your Code Is In Training Data [BigCode](https://www.bigcode-project.org/) is a data set widely used in the training of LLMs, and Hugging Face has a handy resource called [Am I in The Stack?](https://huggingface.co/spaces/bigcode/in-the-stack) that lets you check if your GitHub repositories (and the code within them) are included in it. ![Am I In the Stack?](https://wearedevelopers.imgix.net//magazine/articles/610/images/content/zNoZVvJ6IbPBfgLT0LBe-1754043795.png?w=984&auto=compress,format) Just search for your GitHub username, and if you find your code listed then you can file an opt-out request for future versions. It’s not perfect, but this is one of the few transparent ways to verify whether your open‑source contributions are part of the AI training ecosystem. ## 3. Inspect Logs to Detect AI Crawlers If you want more concrete evidence of AI crawling, your site’s access logs are a powerful resource, showing each request to your site, regardless of the source. Of course the simplest way to check is to manually review these logs, but there are lots of dedicated tools out there to make things easier. [Screaming Frog’s Log File Analyser](https://www.screamingfrog.co.uk/log-file-analyser/) lets you upload log files and quickly visualise which bots are visiting, how often, and which pages they’re targeting, or [GoAccess](https://goaccess.io/) provides an open-source real‑time dashboard from your logs, highlighting unusual spikes or suspicious crawlers. ## 4. Use robots.txt to Set Boundaries Once you know which AI crawlers are indexing your site (and where), you can establish clearer boundaries. Updating your robots.txt file allows you to - in theory - block or allow specific crawlers, using directives like: User-agent: GPTBot Disallow: / In addition, you can apply meta tags such as `<meta name="robots" content="noindex">` on sensitive pages to prevent them from being indexed. Unfortunately some rogue crawlers may ignore these signals, but major players are increasingly pressured to respect them, especially as scrutiny grows around how LLMs collect training data. ## 5. Monetise or Restrict Access Despite AI bots silently crawling (and monetising) existing content for the past few years, the tide might be about to turn with Cloudflare’s introduction of their [pay-per-crawl program](https://blog.cloudflare.com/introducing-pay-per-crawl/). Pay-per-crawl gives developers an enforceable way to control or monetise AI access, with crawlers that attempt to scrape sites running through their servers can be met with an HTTP 402 “Payment Required” response, which forces them to either comply with your terms or back off. Essentially, rather than AI silently crawling and extracting your content, you have a choice over whether to block it, allow it, or monetise access. For developers, this marks a significant shift: rather than being sidelined while your content fuels LLMs, you can finally establish real terms of engagement. ## Summing Up Relatively speaking, we’re in the early days of the AI age. It’s transformed our lives in numerous ways, including how we surf the web, but the days of LLMs scraping and ingesting content, unimpeded, appear to be coming to an end (or at least we can hope!). We hope this article gives you some insights into how your content is being used, and empowers you to have better control over it, too. ## Related Articles - [What Are Large Language Models?](https://www.wearedevelopers.com/magazine/304-what-are-large-language-models) - [How to Use Generative AI to Accelerate Learning to Code](https://www.wearedevelopers.com/magazine/530-how-to-use-generative-ai-to-accelerate-learning-to-code) - [The Web We Broke (And Why AI Agents Are Paying the Price) - AgentCon Berlin](https://www.wearedevelopers.com/magazine/735-the-web-we-broke-and-why-ai-agents-are-paying-the-price-agentcon-berlin) - [SEO in an AI world - Google vs. ChatGPT and survival tips for content creators](https://www.wearedevelopers.com/magazine/534-seo-in-an-ai-world-google-vs-chatgpt-and-survival-tips-for-content-creators) ## Related Videos - [Command the Bots: Mastering robots.txt for Generative AI and Search Marketing](https://www.wearedevelopers.com/videos/1165-command-the-bots-mastering-robots-txt-for-generative-ai-and-search-marketing) - [Fireside Chat with Cloudflare's Chief Strategy Officer, Stephanie Cohen (with Mike Butcher MBE)](https://www.wearedevelopers.com/videos/1366-fireside-chat-with-cloudflare-s-chief-strategy-officer-stephanie-cohen-with-mike-butcher-mbe) - [WeAreDevelopers Live: Browser Extensions, Honey Scam, Jailbreaking LLMs and more](https://www.wearedevelopers.com/videos/1286-wearedevelopers-live-browser-extensions-honey-scam-jailbreaking-llms-and-more) - [ChatGPT vs Google: SEO in the Age of AI Search - Eric Enge](https://www.wearedevelopers.com/videos/1287-chatgpt-vs-google-seo-in-the-age-of-ai-search-eric-enge) ## Related Jobs - [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/588393-machine-learning-engineer) at **Twilio** - [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat** - [Staff Software Engineer, Copilot Experiences](https://www.wearedevelopers.com/jobs/ext/164361-staff-software-engineer-copilot-experiences) at **GitHub** - [Machine Learning Engineer](https://www.wearedevelopers.com/jobs/ext/1355348-machine-learning-engineer) at **TWILIO** - [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub** - [Senior AI Agent Software Engineer (Go, Python) (m/f/x)](https://www.wearedevelopers.com/jobs/48277-senior-ai-agent-software-engineer-go-python-m-f-x) at **Dynatrace**