The Evolution of the Net: Predicting Network Infrastructure

1
The Evolution of the NetPredicting Network
Infrastructure
Bruce R. SchatzGraduate School of Library and
Information Science University of Illinois at
Urbana-Champaign (USA) Graduate School of
Informatics Kyoto University (August 2001 July
2002) schatz_at_uiuc.edu, www.canis.uiuc.edu
GSLIS Proseminar February 24, 2003
2
The Future of Infrastructure

• Niels Bohr on Quantum Theory
• Prediction is very Difficult, especially about
  the Future

3
What is Infrastructure?

• Infra means Inside
• Structure means Support
• Infrastructure is
• Essential Services for Everyday Life
• Universal Protocols to Help Users
• Infrastructure Evolves to Deeper Levels
• New Technologies become Generic Enough

4
Outline of Talk

• The Evolution of Infrastructure
• Amateurs become Experts via Technology
• The Waves of the Net
• Data
• Information
• Knowledge
• The Laws of the Net
• The Basic Units of Evolution

5
Evolution of Infrastructure

• Everyone becomes
• An Expert with
• Interactive Support

6
The Structures of Everyday Life

• Bodies (individuals)
• Food and Clothes
• Buildings (groups)
• Houses and Cities
• Transportation (physical interactions)
• Rails (trains) and Roads (cars)
• Communication (logical interactions)
• Phones (talking) and Computers (retrieving)

7
How Infrastructure Scales

• Experts are Effective but Expensive
• Initially All things done by Experts
• Eventually Most things done by Amateurs
• Volume forces Provider Pyramid
• A few experts at the top
• A lot of amateurs at the bottom
• Infrastructure supports Ordinary People
• Everyone becomes a little bit of an Expert
• Most interactions done by amateurs

8
Example Talking

• Transmission of Messages
• Generated by a person and Sent to another
• Network connects Operators to Operators
• The Telegraph of the 19th Century
• Station to Station, trained professionals
• Operator transmits coded message
• The Telephone of the 20th Century
• Person to Person, untrained amateurs
• Person speaks in ordinary words
• Infrastructure transmits coded message

9
Example Retrieving

• Search of Archives
• Generated by many and Retrieved by many
• Network connects Archives to Users
• Bibliographic Search of the 20th Century
• Station to Station, trained professionals
• Librarian queries central archives
• Web Search of the 21st Century
• Person to Person, untrained amateurs
• Person types in ordinary words
• Infrastructure queries central archives

10
Provider Pyramids Emerge

• Telephone Network was Station-Station
• Person talked to Telephone Operator
• E.g. get me a doctor!
• Operator did manual switching to Person
• Volume forced Network to Person-Person
• In 1900, everyone must become an operator
• In 2000, everyone is an operator!
• Amateurs use directories then dial numbers
• Infrastructure supports Everyday Activity
• Automatic switching scales to volume

11
The Evolution of Infrastructure

• Experts handle Everyday Structures
• Provide services that become essential
• Increased Volume forces Provider Pyramid
• Amateurs handle most Interactions
• Infrastructure supports Amateurs
• Free Labor at Acceptable Quality
• Evolution proceeds by Copying Experts
• New Services approximate Experts
• Infrastructure adopts new Protocols
• Generic Technologies that always work

12
The Waves of the Net

• Network Infrastructure
• Evolves from Data to
• Information to Knowledge

13
What is the Net?

• The Net connects Everyone to Everything
• Global Network Infrastructure
• Universal Protocols to support Interactions
• The Functionality of the Net Evolves
• Data to Information to Knowledge
• Deeper Levels with Stronger Technologies
• At Any Level, the User only sees Outside
• Infrastructure is Inside, thus Invisible
• The Web is just the Present not the Future
• Someday, the Net will be true Cyberspace

14
Waves of the Net - Levels

• Data The Past
• Transmission across Networks (Physical)
• Telephone Operators Switching Gateways
• Information The Present
• Retrieval across Spaces (Logical)
• Reference Librarians Federated Repositories
• Knowledge The Future
• Analysis across Models (Reality)
• Doctors Categories
• Wisdom The Goal

15
Waves of the Net - Services

• Data The Past
• File Transfer
• Basic Unit is Bits within Packets
• Information The Present
• Concept Navigation
• Basic Unit is Words within Documents
• Knowledge The Future
• Structure Correlation
• Basic Unit is Features within Vectors

16
Waves of the Net -- Protocols

• Data The Internet
• Packet Subwave TCP/IP (Transmission)
• File Subwave FTP (File Transfer Protocol)
• Information The Interspace
• Document Subwave HTTP/HTML (Search)
• Concept Subwave CSP (Concept Switching)
• Knowledge The Intermodel
• Vector Subwave VTP/VCPL (Cluster Pattern)
• Structure Subwave SMP (Structure Matching)

17
Transmitting Data

• Telephone Network of the 19th Century
• Wave 0, Multiplexed Telegraph
• Digital Bits
• basic Syntax for Transmission
• Packets are the paradigm
• Packets are grouped Bits
• Address and Routing for Packets
• Bits interpreted only by local computers

18
The Internet

• First Universal Physical Wave
• Wave 1, Protocols across Networks
• Universal across packet networks (IP)
• Files from computer to computer
• Distributed File Systems
• Files cluster Bits within Computers
• Assemble Files from Bits using Packets (TCP)
• Transmit Files across Computers (FTP)

19
Searching Information

• WorldWide Web of the 20th Century
• Wave 2, Hyperlink Transmission (HTTP)
• Digital Words
• basic Semantics for Search
• Documents are the paradigm
• Documents are grouped Words (HTML)
• Classification and Indexing for Documents
• Words interpreted only by local repositories

20
The Interspace

• First Universal Logical Wave
• Wave 3, Protocols across Spaces
• Universal across document spaces
• Concepts from repository to repository
• Distributed Concept Systems
• Concepts cluster Words within Repositories
• Assemble Concepts from Words using Documents
• Search Concepts across Repositories

21
Universal Information Units

• Basic Concepts of Natural Language
• Fundamental Grouping needed for Semantics
• Specialized to specific Community Collections
• Essential Technologies all Converging
• Document Representation (XML, RDF)
• Language Parsing (multiword noun phrases)
• Statistical Indexing (contextual frequencies)
• Peer-Peer Networking (local computations)
• Concept Switching (vocabulary translations)

22
Same Process, Deeper Level

• The Interspace, using Libraries
• Subject thesaurus, yields Concept space
• Subject classification, yields Concept map
• From words to concepts
• From networks to spaces
• The Intermodel, using Lifestyles
• Reality problem, yields Structure model
• Reality solution, yields Structure cluster
• From features to structures
• From spaces to models

23
Analyzing Knowledge

• Lifestyle Grid of the 21st Century
• Wave 4, Full-spectrum Questionnaires
• Digital Features
• basic Pragmatics for Analysis
• Vectors are the paradigm
• Vectors are grouped Features
• Comparison and Clustering for Vectors
• Features interpreted only by local situations

24
Healthcare Infrastructure

• Monitors for each Individual
• Capture everyday lifestyle status
• Customized questionnaires for healthcare
• Build detailed database of health vectors
• Managers for each Individual
• Daily Feature Record for each Individual
• Detailed Databases for whole Population
• Analyze Clusters of Similar Individuals

25
The Intermodel

• First Universal Representational Wave
• Wave 5, Protocols across Models
• Universal across vector models
• Features from individual to individual
• Distributed Structure Systems
• Structures cluster Features within Individuals
• Assemble Structures from Features using Vectors
• Analyze Structures across Individuals

26
Universal Knowledge Units

• Basic Structures of Everyday Life
• World Categories needed for Pragmatics
• Similar Structures from Many Subjects
• Medicine, categories of health status
• Computing, TCOL versus UNCOL
• Anthropology, cross-cultural universals
• Psychology, basic natural categories
• Architecture, design patterns for comfort life
• History, structures of everyday life

27
Realizing Wisdom

• Zen Buddhism
• The Ocean including all Waves
• Direct Interaction with Reality
• Beyond Words and Concepts
• Zen Koans
• Curriculum process of realization exercises
• Become one with the koan, achieve Satori
• Richi -- Reality Challenge, e.g. Mu
• Kikan Reality Interaction, e.g. Oak Tree

28
The Intermind

• First Universal Spiritual Wave
• Wave N, Protocols across Minds
• Universal across Mind Realities
• Hive Mind on the Net, There is no Self!
• Religions to model Lifestyles
• richi (koan) for monitor (health)
• kikan (koan) for manager (health)
• Mimic basic structures not details
• Interactive navigation through automated clusters
  not automated decision-making

29
Metaphors across Waves

• Capturing Operators yields Internet
• Capturing Librarians yields Interspace
• Capturing Doctors yields Intermodel
• Capturing Roshi-s yields Intermind

30
The Laws of the Net

• Predicting Units of
• Network Infrastructure
• for Deeper Evolution

31
Philosophy of Informatics

• Physics in the 20th Century
• Newtonian Mechanics is Western philosophy
• Deterministic Laws predict all motion
• Quantum Mechanics is Eastern philosophy
• Probabilistic Computations are uncertain
• Informatics in the 21st Century
• Database Retrieval is Western philosophy
• Deterministic Rules match correct results
• Information Retrieval is Eastern philosophy
• Statistical Clusters suggest useful results

32
Laws of Motion

• Isaac Newton, 1687, Principia Mathematica
•  
• I. Every body continues in its state of rest,
  or of uniform motion in a right straight line,
  unless it is compelled to change that state by
  forces impressed upon it.
• II. The change of motion is proportional to the
  motive force impressed and is made in the
  direction of the right line in which that force
  is impressed.
• III. To every action there is always opposed an
  equal reaction or, the mutual actions of two
  bodies upon each other are always equal, and
  directed to contrary parts.

33
Laws of Infrastructure

• Bruce Schatz, 2004, The Evolution of the Net 
• I. Every infrastructure undergoes a
  characteristic evolution, from station-station to
  person-person, from trained experts to untrained
  amateurs, to handle the volume of a universal
  service.
• II. Most of the volume of a universal service
  is handled by the users themselves, with
  infrastructure support provided by modeling an
  appropriate expert.
• III. Each new basic structure models a new
  expert in terms of previous structures, and each
  successful evolution brings the structures closer
  to approximating reality.

34
Levels of Abstraction

• Deeper Levels drive Later Waves
• Each Wave has a fundamental level
• Deeper composed from Shallower
• Each Level built upon the Previous Level
• units level N-2 ordered by (ob) units level N-1
• Context of N-2 within N-1 yields N
• Laws predict Future Units
• Information Space, Knowledge Model

35
Units of Abstraction

• Metaphors
• structures ob situations
• Situations
• categories ob structures
• Structures
• concepts ob categories
• Categories
• documents ob concepts
• Concepts
• terms ob documents
• Documents
• words ob terms

36
The Science of Informatics

• New Units from Old Units
• Predict Future Units from Current Units
• New Waves from Old Waves
• Predict Future Protocols from Current
• Same Processes, New Technologies
• Context as Meaning, Statistics as Truth
• Deeper Levels are Closer to Reality