Scale & performance

Does it work at the size needed?

tension: Completeness against speed: storing, indexing and answering over billions of statements, and queries that reach across many sources at once.
also: Performance, Scalability, Storage
scheme: design issues

Threads (2)

Long-running problems under this issue, told step by step. All threads.

Querying a web, not a database 2008–2025 open 7 steps
Can you query data spread across thousands of independent servers as if it were one database?
Who pays for public data infrastructure? 1995–2025 open 9 steps
Who pays to keep shared identifiers, query services and knowledge bases running after the launch money is spent?

Who answered it

This issue's row of the map: which traditions' works mainly answer it, and the earliest from each.

Works on this issue by tradition
tradition	works	earliest
Hypertext
Knowledge representation
Databases	6	Virtuoso 1998
Machine learning	1	Big data
Web standards	5	Virtuoso 1998
Search	1	MapReduce: Simplified Data Processing on Large Clusters 2004
Social web
Open knowledge	1	Wikidata Query Service 2015
Decentralized web	1	Triple Pattern Fragments: A Low-cost Knowledge Graph Interface for the Web 2016
Libraries, archives and museums
Life sciences
Media
Geospatial
Commerce and supply chain
Government and law
Finance
Industry and engineering

6 of 17 traditions have taken up this question as a work's main issue; a work in several traditions counts in each. Same counts as the full map.

Timeline

Works on this question, its main or a secondary one, each shown once in date order; filter by tradition to follow one community. Bars run from start to end where both are known; marks people, projects and organizations still active. 3 entries have no date and are not shown. Hover over or focus an entry to draw its links. Scroll sideways, or use the decades above.

Lineage (9)

Works whose main question this is, by type, oldest first.

Concepts 2

Big data concept
Data too large, fast-moving or varied for ordinary databases, and the methods for drawing value from it by analysis at scale; the term spread around 2010 as companies mined logs, clicks and sensor data.
Triplestore concept technique
Database built to store RDF triples and answer SPARQL queries; Virtuoso is one example.

Writings 4

MapReduce: Simplified Data Processing on Large Clusters 2004 writing ★ seminal
2004 OSDI paper by Jeffrey Dean and Sanjay Ghemawat describing how Google split computations over huge datasets across thousands of ordinary machines; it set the pattern for big data processing.
Executing SPARQL Queries over the Web of Linked Data 2009 writing technique
ISWC 2009 paper by Hartig, Bizer and Freytag proposing link-traversal querying: a SPARQL query is answered by following links and looking up URIs as it runs, so the data sources need not be known in advance.
SPARQL Web-Querying Infrastructure: Ready for Action? 2013 writing ★ seminal
ISWC 2013 paper that monitored hundreds of public SPARQL endpoints and found many of them often unavailable, slow, or limited in what they would answer: evidence that free public query services were hard to keep running.
Triple Pattern Fragments: A Low-cost Knowledge Graph Interface for the Web 2016 writing ★ seminal technique
2016 Journal of Web Semantics paper proposing Triple Pattern Fragments: servers answer only simple triple patterns and clients do the rest of the query, trading some speed for servers that are cheap to run and stay up.

Projects 3

Virtuoso 1998 project
OpenLink Software's database server for SQL and RDF data, used to publish DBpedia and many other Linked Data sets.
Apache Hadoop 2006 project
Open-source framework, started in 2006 by Doug Cutting and Mike Cafarella, that implemented MapReduce and a distributed file system and made big data processing available outside Google.
Wikidata Query Service 2015 project
Public SPARQL endpoint for Wikidata, run by the Wikimedia Foundation since 2015. In May 2025 it moved scholarly articles into a separate graph because its Blazegraph backend could no longer keep up with Wikidata's size.

Also bears on this (11)

Concepts 1

Internet of Things 1999 concept vision
Physical objects that identify themselves and report data over networks without people typing it in; the term was coined by Kevin Ashton in 1999 about RFID in supply chains.

Writings 3

Semantics and Complexity of SPARQL 2009 writing ★ seminal formalism
Paper by Pérez, Arenas and Gutiérrez giving SPARQL a compositional formal semantics and proving the complexity of evaluating it; general SPARQL patterns are PSPACE-complete.
Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion 2014 writing
2014 KDD paper by Xin Luna Dong and eight Google colleagues describing a knowledge base built by machine: facts extracted from text, tables and page structure across the Web, combined with existing knowledge bases and given probabilities of being true.
Industry-scale Knowledge Graphs: Lessons and Challenges 2019 writing ★ seminal
2019 Communications of the ACM article by engineers from Microsoft, Google, Facebook, eBay and IBM comparing how their companies build and run knowledge graphs, and what remains hard.

Standards 6

Satellite Catalog Number 1957 standard
Sequential number given by the US military to every object it tracks in orbit, starting with Sputnik 1's rocket (1) and satellite (2). It ran out of five digits on 11 July 2026, forcing the 'Alpha-5' workaround and a move to nine-digit formats.
Two-line element set 1980 standard
NORAD's fixed-width text format for an object's orbit, read with the SGP4 model published in 1980; still the most shared orbital data, though its five-digit catalogue field overflowed and the CCSDS Orbit Mean-Elements Message has been used alongside it since 2020.
OWL 2 2009 standard
Second version of OWL, adding profiles such as OWL 2 EL, QL and RL for tractable reasoning; a W3C Recommendation since October 2009.
CBOR 2013 standard
IETF binary data format with a data model close to JSON's, designed for small code and message size on constrained devices; first published as RFC 7049 in October 2013 and revised as Internet Standard RFC 8949 in December 2020.
CBOR-LD 2026 standard
Binary serialization of JSON-LD in CBOR that uses JSON-LD contexts to compress terms and values, reporting results more than 60% better than general-purpose compression; meant for storage- and bandwidth-constrained settings. A W3C JSON-LD Working Group First Public Working Draft since 28 April 2026.
Electronic Product Code standard
Identifier for each individual physical object, not just each product type, designed to be carried on RFID tags; it came out of MIT's Auto-ID Center and is maintained by GS1.

Projects 1

Canada Geographic Information System c. 1963–c. 1989 project
Early geographic information system built for the Government of Canada from the early 1960s to store and analyse the Canada Land Inventory; it ran until the late 1980s.

People and organizations (15)

Linked to works on this question; the number counts those works.

Olaf Hartig (2) · Aidan Hogan · Carlos Buil-Aranda · Christian Bizer · Doug Cutting · Jeff Dean · Johann-Christoph Freytag · Jürgen Umbrich · Kingsley Idehen · Mike Cafarella · OpenLink Software · Pierre-Yves Vandenbussche · Ruben Verborgh · Sanjay Ghemawat · Wikimedia Foundation