RadioOS
RadioOSBot
RadioOSBot is the crawler behind RadioOS, software that radio stations use to run their day. It reads the public news feeds of local outlets and public agencies, so a host can see what a community is talking about before going on air.
If you run a site and want us gone, you can skip to How to stop us. It takes one line, and we will not ask why.
How to recognise it
Every request carries this name:
RadioOSBot/1.0 (+https://radioos.ca/bot)It also sends From: sources@radioos.ca. It comes only from the addresses below. Each one's reverse DNS is bot.trends.radioos.ca, and that name resolves back to the same addresses, so you can check a request the way you would check a search engine's crawler. Anything else using our name is not us.
167.114.66.156
2607:5300:205:200::1:117What it reads
robots.txt, and the two files that say what a site allows machines to do,/.well-known/tdmrep.jsonand/ai.txt, once a day.- One public RSS or Atom feed per outlet, and news sitemaps where a site has no feed.
- Your home page and your terms of use, about four times a year, to see whether the terms say anything about reading your feed. If they do, we stop, and a person reads them.
- Environment Canada's weather alerts, each one as its Datamart announces it, through the Datamart's own subscription service.
- Bluesky's public stream of new posts, and its trending topics once an hour, to count how often public figures and provinces are named.
- When we first come across a site, and again after 30 and 90 days and then twice a year if it had none, to find out whether it has a feed: its
robots.txt, its home page once, and the usual addresses for a feed or a news sitemap (/feed/,/rss,/rss.xml,/feed,/index.xml,/atom.xml,/news-sitemap.xml,/sitemap-news.xml). It stops at the first feed it finds. Across the whole web this finding-out is at most 1 request a second.
It never reads article pages. It never crosses a login, a paywall or a challenge page, and it does not run a browser.
What it keeps
From each item in a feed it reads four things: the title, the link, the date and the description. From each item in a news sitemap it reads three: the title, the link and the date. From each Environment Canada alert it reads three: the headline, the time it was sent and the provinces its areas lie in. It skips the rest, including full text and images, without reading them. From Bluesky it keeps only a count each minute and the trending topics that are a public figure's name or plain words: never a post, its text or who wrote it.
- On disk: the link, the outlet, the times, and facts worked out from the item, such as which places it mentions. No title, no description and no article text. An Environment Canada alert's headline is the one exception, because its licence allows it to be kept and shown unaltered.
- In memory only: the title, for up to 24 hours, and the description, for as long as it takes to process the item.
What a station sees is our own words about a story, with your outlet's name, the time and a link to you. We show a publisher's own headline or text only when a licence or the publisher's written yes allows it.
How it behaves
- At most one request to a host every 10 seconds, and fewer if your site asks.
- At most 2 requests a second to one server address, so sites sharing a server are not hit together. Across everything it reads, at most 50 requests a second.
- One connection to a publisher at a time.
robots.txtis read at most once every 24 hours. Rules forRadioOSBotand for*both apply. Ifrobots.txtanswers with a server error, we read nothing from that host until it recovers.Crawl-delayandRetry-Afterare honoured. A 429 or a 503 makes it back off, doubling each time, up to 24 hours between tries.- If one server address turns it away (a 401, 403 or 451) on 5 of the sites it hosts within 15 minutes, it waits on every site at that address, from a minute, doubling each time, up to an hour between tries.
- It asks for gzip, sends conditional requests, and adds nothing to your addresses to get around a cache. A host that stops resolving is tried weekly, then monthly.
How to stop us
Either of these works. Neither needs a reason or a conversation.
Add this to your robots.txt. It takes effect within 24 hours.
User-agent: RadioOSBot
Disallow: /Or email sources@radioos.ca with your domain. Reading stops the same business day, and it stays stopped until you tell us otherwise.
We also stop if your terms bar automated reading of your feed, or if you reserve text and data mining in a machine-readable way where the law gives that force.
Questions
sources@radioos.ca reaches a person. Tell us if the bot has done something this page says it does not.
Back to RadioOS