Robots and licence, explained

Two decisions get made once and then quietly govern everything: what crawlers may read, and what anyone may do with the source. Both are usually copied from somewhere. Ours are written down.

Crawling

Every host allows every crawler.

`` User-agent: * Allow: / ``

That is the whole policy, and it follows from what the tools are. There is no account area to keep out of an index, no customer data behind a path, and no page whose value depends on being hard to find. A restrictive robots.txt on a set of static utilities would be theatre.

Two things are worth knowing about how those rules are evaluated, because both are commonly got wrong.

First, robots.txt is not access control. It is a request that well-behaved crawlers honour. Anything that must not be read must not be served.

Second, matching is by longest rule, not by first rule. Given a Disallow: /admin/ followed by an Allow: /admin/public/, Google allows /admin/public/page, because the allow pattern is longer. Plenty of files are written as though the first matching line wins, and they do not behave the way their author expected.

AI training crawlers

We allow them, and we think the decision is worth stating rather than leaving to a default.

The content here is a set of utilities and the documents explaining them. It is published in order to be read, the source is public, and a model that has read this desk is not taking anything a human reader could not take. Blocking the training crawlers would cost us nothing and gain us nothing, which makes the block a gesture.

We also use an AI provider ourselves to build this toolchain (V-5). Taking from a commons while fencing our own corner of it would be a position we could not defend.

This is a decision, not a principle. If what we publish changes, the decision gets revisited, and the revision will be a commit on this page.

Licence

First-party repositories are private by default and licensed when they are opened. The choice is made from four questions rather than by habit:

1. Should work built on ours stay open?
2. Do we want an explicit patent grant?
3. Do we want attribution?
4. Is it a library or an application?

For these tools the answers are no, yes, yes, and application, which lands on Apache 2.0: permissive, with the patent grant written down rather than implied. MIT would be the shorter alternative and BSD 3-Clause the one that adds a no-endorsement clause.

A licence picker that asks exactly those four questions is being built as one of the tools. As with the controls page, it is not linked until the host verifies as live.

The bundled licence texts

Where a tool of ours reproduces a licence, the text comes verbatim from the SPDX licence list, with its origin and the date pulled recorded next to it. Licence texts are legal documents. They are not reflowed, summarised in place, or tidied.