◆ Semantic Web Catalogue

Problem thread

Why would anyone add structure?

Who bears the cost of publishing structured data, and what makes it worth their while?

issue
adoption & incentives
status
open
steps
12, 1998–2026
threads
all problem threads

The 2001 vision relied on publishers describing their content for machines. Critics said people would not do it honestly or at all, and five years later the vision's authors agreed it was largely unrealized. What changed adoption was a direct reward: richer results in search and better previews on social sites. By 2024 about half of crawled web pages carried structured data, mostly schema.org, while the biggest knowledge graphs are built by companies for their own products. Where incentives were weak, regulation has supplied them, from financial filings in XBRL to the EU's Digital Product Passport. AI agents that read pages for people may be the newest incentive.

Lineage

  1. 1998
    proposes standard
    XBRL

    Regulators, securities commissions and tax agencies require filings in XBRL, so companies publish structured financial data because the law asks for it, not because it pays. [src]

  2. May 2001
    proposes writing ★ seminal
    The Semantic Web

    Software agents act on machine-readable meaning that publishers put on the Web alongside their pages. [src]

  3. 26 Aug 2001
    poses writing ★ seminal
    Metacrap: Putting the torch to seven straw-men of the meta-utopia

    The 'meta-utopia' will never come: people lie, are lazy, cannot describe themselves accurately, and disagree on how to describe things. Metadata is useful only for rough assumptions. [src]

  4. May 2006
    measures writing ★ seminal
    The Semantic Web Revisited

    Five years on, the vision's authors find its simple idea still largely unrealized: bots exist but are built for narrow tasks. They point to standards and to ontologies in e-science as grounds for optimism. [src1][src2]

  5. 1 May 2007
    measures project ★ seminal
    Linked Open Data cloud diagram

    A diagram of the datasets published as Linked Data and the links between them, first drawn in 2007 with DBpedia at the centre. [src]

  6. 12 May 2009
    proposes project ★ seminal
    Rich Snippets

    A reward for markup: Google shows ratings, prices and other details in search results for pages that mark them up with microformats or RDFa. [src]

  7. 2010
    proposes standard
    Open Graph protocol

    Facebook's meta tags turn a page into a rich object when someone shares it, another direct reward for adding a little structure. [src]

  8. 2 Jun 2011
    settles standard ★ seminal
    Schema.org

    The major search engines agree on one vocabulary, so a publisher marks up once for all of them. [src]

  9. 13 Mar 2012
    measures project
    Web Data Commons

    Extracts structured data from the Common Crawl. In the October 2024 crawl about 52% of pages and 44% of websites carried it, with microdata and JSON-LD dominant. [src]

  10. 24 Jul 2019
    measures writing ★ seminal
    Industry-scale Knowledge Graphs: Lessons and Challenges

    Engineers from Microsoft, Google, Facebook, eBay and IBM compare how their companies build and run knowledge graphs for their own products, and what remains hard. [src]

  11. 18 Jul 2024
    proposes project
    Digital Product Passport

    The EU requires products to carry a digital record of their materials, repair and recycling, reached by a QR code and registered centrally, starting with batteries in 2027: structure by mandate. [src]

  12. 20 Oct 2026
    measures writing
    W3C and the State of the Web in the AI Era

    AI agents become a new reason to publish structure: the W3C finds that ordinary HTML falls short for agents, which prefer text and structured data, and that knowledge graphs help retrieval-augmented generation. [src]

Recent signals

Dated events that show the problem moving, newest last.

  1. 12 Jun 2025

    Google phases out several rich-result types, among them ClaimReview, Course Info and Vehicle Listing, saying they were rarely used.[src]

  2. 22 Jul 2025

    Pew finds Google users click a result in 8% of visits when an AI summary appears, against 15% without one.[src]

  3. Nov 2025

    Google says it still supports about 30 structured-data features and drops the retired ones from Search Console from January 2026.[src]

  4. 16 Sep 2026

    Schema.org 30.1 adds vocabulary for EU Digital Product Passports and retail product data. Schema.org[src]

Open now

Under each question, the answers people have put forward or built. The catalogue gives no odds on them.

  • Will publishers keep adding markup when the reward shrinks? [src]
    In 2023 Google limited FAQ rich results to well-known, authoritative government and health sites.
    • Publishers keep markup only for the features search engines still reward. Schema.org
      Google narrows its support to the structured data it finds useful and retires the rest.[src1][src2]
    • Markup is kept as an input for AI agents rather than for search snippets. W3C and the State of the Web in the AI Era
      The W3C's 2026 paper argues that structured data becomes more valuable as agents read it.[src]
  • Who gains from structured data published on the open Web: publishers, the public, or the few companies that collect it? [src1][src2]
    Half of web pages now carry structured data, while the largest knowledge graphs are private to the companies that build them.
    • The companies that collect it gain most, as answers replace visits.
      Pew's 2025 study and the Wikimedia Foundation both report fewer people clicking through to sources when search answers directly.[src1][src2]
    • Open crawls make the same data available to everyone. Web Data Commons
      Web Data Commons extracts the structured data in the Common Crawl and publishes it openly.[src]

Forces

Pressures that push this problem one way or another, as their sources argue them. All forces

How to contribute