Live network feed
DETECTING › 

AI crawlers are
mapping the web.
We’re watching them.

Every site running AI Observatory contributes anonymous crawler data here. Real traffic, real bots, no login required.

View live data ↓ Add your site
 crawler sweep · live
--
Events (30d)
--
Bot types seen
--%
Fetch robots.txt
--%
Probe /llms.txt

Public intelligence

Aggregate crawler data

Live data from every site contributing to the network. No auth. No filter.

Network overview · last 30 days loading…
Top AI bots by activity
Most requested paths
Protocol probe rates
First URL bots request
ai-observatory · live detection feed
Voluntary agent identifications loading…
AI agents that read agents.md in the repo and POST to /hello appear here publicly.

Proposed open standard

A handshake
that doesn’t exist yet

No standard exists for AI agents to introduce themselves to the resources they access. We’re proposing one.

Today
Website
Who are you?
Crawler
...
What we’re proposing
Website (agents.md)
Please introduce yourself.
Compliant agent
Every repo using AI Observatory includes agents.md and crawler.json, a human and machine-readable invitation to POST identity to /hello. All identifications are public. If you represent an AI organization, open an issue to discuss formalizing this.

Two ways to participate

Pick the path
that fits your setup

There are two distinct setups. One takes two minutes on any PHP site. The other requires a server you control and gives you a private dashboard for your own data.

Easiest · no server required
Just contribute to the network
You have a PHP website. You want your bot traffic to show up in the public data above. No account, no server, no dashboard.
PHP site cURL or fsockopen enabled ~2 minutes
1
Download aio-phonehome.php from the repo and upload it to your website’s root folder (same place as your index.php).
2
Add one line to the very top of your PHP header or index.php file:
index.php or header.php
require_once 'aio-phonehome.php';
// That's it. Bot visits will appear on ai-agent-intel.com within minutes.
That’s the entire setup. When an AI bot visits your site, an anonymous signal (bot name + path, no IP, no personal data) is sent to this network automatically.
Full setup · server required
Host your own observatory
You want a private dashboard at /dashboard showing only your site’s bot traffic, with full history and per-path breakdown. This requires a server you control.
VPS or server Node.js 22+ PHP or Node.js site ~30 minutes
1
Clone & run the server on your VPS. This is the backend that stores your site’s data.
terminal
git clone https://github.com/darthm1ke/ai-observatory
cd ai-observatory
cp .env.example .env  # set ADMIN_KEY and IP_SALT
docker compose up -d
terminal
git clone https://github.com/darthm1ke/ai-observatory
cd ai-observatory
cp .env.example .env  # set ADMIN_KEY and IP_SALT
npm install
node server/index.js
2
Register your site to get an API key. Replace your-admin-key with what you set in .env.
terminal
curl -X POST https://your-server.com/sites \
  -H "X-Admin-Key: your-admin-key" \
  -H "Content-Type: application/json" \
  -d '{"domain":"yoursite.com"}'
# Returns your API key
3
Install the tracker on your website using the API key you just received.
header.php
define('AIO_API_KEY',  'your-api-key');
define('AIO_ENDPOINT', 'https://your-server.com/beacon');
require_once 'ai-observatory.php';
server.js
const { observatory } = require('./ai-observatory');
app.use(observatory({
  apiKey:   'your-api-key',
  endpoint: 'https://your-server.com/beacon',
}));
4
Open your dashboard at https://your-server.com/dashboard and enter your API key. Your bot data will start appearing as bots visit your site.
Full documentation, the .env reference, and the complete bot registry are on GitHub.
View on GitHub →