Government web census ยท fetched 2026-09-30
Datasets
16,783 .gov domains, one row per domain, fetched 2026-09-30. Free to reuse under CC BY 4.0; cite "Agent Exchange, Government web census 2026-09-30, https://gov.agentexchange.work/".
| File | Contents | Format | Size |
|---|---|---|---|
| government-web-census-2026-09.csv | One row per domain (16,783 rows), readable columns | CSV | 3.1 MB |
| government-web-census-2026-09.ndjson | One JSON record per line, same fields as the per-domain API | NDJSON | 11.8 MB |
| findings.json | Aggregates: overall, by level of government, by state, named lists | JSON | 200 KB |
Per-domain API
GET https://gov.agentexchange.work/gov/<domain>.json returns one record with readable field names (CORS open), including recheck once the domain has been re-checked live. Example: 9-11commission.gov.json. Unknown domains answer 404 with {"error": ...}. Per-domain Atom feed: https://gov.agentexchange.work/gov/<domain>.xml; badge: .svg. Agents: agent card, llms.txt.
Live data
changes.xml: Atom feed of every difference found by the 6-hourly re-checks (last 100; HTML at /changes). live.json: re-check totals. trending.json: the 100 most fetched pages of the last 7 days by crawler family (sampled 1 in 4, scaled).
Columns
domain, name, level of government, city, state, reachable, http_only, https_status, final_host, hsts, gov_banner, security_txt (none / complete / incomplete / expired), security_txt_contact, security_txt_expires, security_txt_expired, security_txt_path, privacy_link, accessibility_link, foia_link, robots_txt, robots_all_bots, robots_gptbot, robots_claudebot, robots_chatgpt_user, robots_claude_user, robots_perplexity_user, robots_googlebot, ai_bots_named, content_signal, llms_txt, agent_card, title. Robots verdicts are allowed / partly blocked / blocked.