CS 591 Complex Adaptive Systems Spring 2008 Measures of Complexity Professor Melanie Moses 1/28/07 - PowerPoint PPT Presentation


PPT – CS 591 Complex Adaptive Systems Spring 2008 Measures of Complexity Professor Melanie Moses 1/28/07 PowerPoint presentation | free to download - id: e5dd0-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

CS 591 Complex Adaptive Systems Spring 2008 Measures of Complexity Professor Melanie Moses 1/28/07


Wolfram and Langton's complexity of CAs. Why define complexity? To estimate how long a particular system will ... Per Bak: Sandpile model and the edge of chaos' ... – PowerPoint PPT presentation

Number of Views:33
Avg rating:3.0/5.0
Slides: 30
Provided by: valued1031
Learn more at: http://www.cs.unm.edu


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: CS 591 Complex Adaptive Systems Spring 2008 Measures of Complexity Professor Melanie Moses 1/28/07

CS 591 Complex Adaptive Systems Spring
2008 Measures of Complexity Professor Melanie
Moses 1/28/07
  • Why do we need formal definitions of complexity?
  • Complex Adaptive Systems
  • Definitions of Complexity
  • Holland
  • Flake
  • Computational Definitions of complexity
  • Running time complexity
  • Kolmogorov (or Algorithmic) complexity
  • Contrast to Shannon information and randomness
  • Logical (thermodynamic) Depth
  • Others
  • Murray Gell-Mann
  • Stuart Kauffman
  • Wolfram and Langtons complexity of CAs

Why define complexity?
  • To estimate how long a particular system will
    take to solve a problem
  • To estimate difficulty in engineering complex
  • To understand the limits of prediction,
    approximation, and simulation
  • To answer fundamental scientific questions
  • How can complexity arise when entropy should
    always increase?
  • Does (why does) complexity increase through
  • As evolution proceeded on the surface of the
    earth, there has been a progressive increase in
    size and complexity J. T. Bonner Evolution of
  • Evolution tends to push systems towards the edge
    of chaos, where complex, interesting behaviors
    such as life can occur? Chris Langton
  • The universe, the biosphere, the econosphere
    have all become more complex Stuart Kauffmann
  • Are these statements correct? Can we quantify
    increases in complexity over time

What are Complex Adaptive Systems?
  • Collections of agents
  • Molecules, cells, animals, planets, stars,
    economic agents.
  • Agents interact (locally) with one another and
    with their environment
  • No central controller.
  • Interaction rules may be trivial or nontrivial,
    i.e. nonlinear.
  • Chemical reactions, cellular interactions,
    mating, buy/sell decisions.
  • Unanticipated properties often result from the
  • Immune system responses, flocks of animals,
    settlement patterns, earthquakes, speculative
    bubbles and crashes.
  • Agents adapt their behavior to other agents and
    environmental constraints
  • Imitation, adaptation, learning.
  • System behavior evolves over time
  • Rules change, unfolding of complex behavior.

Example Complex Adaptive Systems
  • Biological systems
  • Cells
  • Organs
  • Immune systems
  • Organisms
  • ecosystems
  • Food webs
  • Symbioses
  • Colonies ants, bees, termites, wasps bacteria,
  • Social/Economic Systems
  • Nations
  • Firms
  • Households
  • Cities
  • Universities
  • Technological Systems
  • The Internet
  • Peer to peer networks
  • Botnets

Characteristics of Complex Systems
  • What makes a system complex?
  • Nonlinear interactions among components.
  • Multi-scale phenomena and hierarchy.
  • Evolution of underlying components and
  • How to measure a systems complexity?
  • By its unpredictability?
  • By how difficult it is to describe?
  • Length of most concise description.
  • No single model adequate to describe system---the
    more models that are required, the more complex
    the system. (Lee Segel)
  • By measuring how long before it halts, if ever?
    By how long until it repeats itself?
  • Entropy?
  • Multiple levels of organization?
  • Number of interdependencies?
  • Is complexity inherent in the system or in our
    understanding of it?

  • Some believe that there is no general science of
    complex systems
  • Its becoming apparent that a theory of
    complexity in some sense of a great and
    transforming body of knowledge that applies to a
    whole range of cases may be untenable. Sir
    Robert May (2001)

Hollands Definition of Complexity
  • CAS multiple diverse agents, stimulus response
    rules and adaptation
  • Match environmental conditions at various time
  • Stimulus causes an immediate change in action
  • Cumulative stimulus causes changes in rules
    (adaptation, learning, evolution)
  • 7 principles
  • Aggregation how we model CAS emergent behavior
    of CAS
  • Tags identifiers that allow interaction and
    aggregation of agents
  • Nonlinearity esp. interactions, e.g. predator
    prey linear math does not apply
  • Flows over networks of interactions multiplier
    effect vs. recycling effect
  • Diversity results from progressive, distributed
    adaptation perpetual novelty
  • Internal models (Schema) internal structure
    changes in response to stimulus
  • Building blocks repetition, higher level laws
    derive from laws of building blocks

G.W. Flakes Definition of Complexity
  • Flakes scientific view
  • An intermediate view between Reductionism and
  • The interactions of agentsbind one level of
    understanding to the next
  • Nature is Frugal Occams Razor
  • CAS characterized by
  • Emergence, self-similarity, unpredictability,
  • Collections/Multiplicity/Parallelism
  • Iteration, Recursion, Feedback
  • Adaptation, Learning, Evolution
  • Conversion of the sciences facilitated by

Representative Measures of Complexity
  • Computational complexity (Cook)
  • How long a program runs (or how much memory it
  • Asymptotic.
  • Language complexity
  • Classes of languages that can be computed
    (recognized) by different kinds of abstract
  • Decidability, computability.
  • Information-theoretic approaches
  • Algorithmic Complexity (Solomonoff, Komogorov,
    and Chaitin)
  • Length of the shortest program that can produce
    the phenomenon.
  • Mutual information (many authors)
  • Logical depth (Bennett).
  • Thermodynamic depth (Lloyd and Pagels)
  • Effective complexity (Gell-Mann and Lloyd)

Computational Complexity
  • Introduced by Steve Cook (1970).
  • Asymptotic running time, and/or memory
    consumption of an algorithm.
  • Worst-case versus average-case.
  • Important computational complexity classes
  • NP (Can be verified in polynomial time). O(p(n))
    on a non-deterministic Turing Machine.
  • NC (polylogarithmic time,Ologk n, using
    polynomial of parallel processors).
  • P (polynomial time using a single processor).
  • C,k are constant, and n is the size (length) of
    the input.
  • Polynomial time algorithms P are O(p(n))
  • for some polynomial function p Cnk where c and K
    are constants independent of n
  • Drawbacks
  • Says nothing about transient behaviors. Many
    interesting systems never reach asymptote
  • Constants can matter, special cases vs worst
    cases, approximations may suffice

Algorithmic Complexity (AC) (also known as
Kolmogorov-Chaitin complexity)
  • The Kolomogorov-Chaitin complexity K(x) is the
    length, in bits, of the smallest program that
    when run on a Universal Turing Machine outputs
    (prints) x and then halts.
  • Example What is K(x) where x is the first 10
    even natural numbers? Where x is the first 5
    million even natural numbers?
  • Possible representations
  • 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, (2n - 2)
  • for (j 0 j lt n j) printf(d\n, j 2)
  • How many bits?
  • Alternative 1 O(n log n)
  • Alternative 2 K(x) O(log n)
  • Two problems
  • Calculation of K(x) depends on the machine we
    have available (e.g., what if we have a machine
    with an instruction print the first 10 even
    natural numbers?)
  • In general, it is an uncomputable problem to
    determine K(x) for arbitrary x.

Algorithmic Complexity cont.
  • AC formalizes what it means for a set of numbers
    to be compressible and incompressible.
  • Data that are redundant can be more easily
    described and have lower AC.
  • Data that have no clear pattern and no easy
    algorithmic description have high AC.
  • What about random numbers? If a string is
    random, then it possesses no regularities
  • K(x) Print(x)
  • The shortest program to produce x is to input to
    the computer a copy of x and say print this.
  • Implication The more random a system, the
    greater its AC.
  • AC is related to entropy
  • The entropy rate of a symbolic sequence measures
    the unpredictability (in bits per symbol) of the
  • The entropy rate is also known as the entropy
    density or the metric density.
  • The average growth rate of K(x) is equal to the
    entropy rate
  • For a sequence of n random variables, how does
    the entropy of the sequence grow with n?

Measures of Complexity that Capture Properties
Distinct from Randomness
Structural Complexity
Measures of randomness do not capture structure
or organization.
Logical Depth
  • Bennett 19861990
  • The Logical depth of x is the run time of the
    shortest program that will cause a UTM to produce
    x and then halt.
  • Logical depth is not a measure of randomness it
    is small both for trivially ordered and random
  • Drawbacks
  • Uncomputable.
  • Loses the ability to distinguish between systems
    that can be described by computational models
    less powerful than Turing Machines (e.g.,
    finite-state machines).

Turing Machines
Shannon Information
  • Shannon Entropy H to measure basic information
  • For a random variable X with a probability mass
    function p(x), the entropy of X is defined as
  • Entropy is measured in bits.
  • H measures the average uncertainty in the random
  • Example 1
  • Consider a random variable with uniform
    distribution over 32 outcomes.
  • To identify an outcome, we need a label that
    takes on 32 different values, e.g., 5-bit

  • More generally, the entropy of a random variable
    is a lower bound on the average number of bits
    required to represent the random variable.
  • The uncertainty (complexity) of a random variable
    can be extended to define the descriptive
    complexity of a single string.
  • E.g., Kolmogorov (or algorithmic) complexity is
    the length of the shortest computer program that
    prints out the string.
  • Entropy is the uncertainty of a single random
  • Conditional entropy is the entropy of a random
    variable given another random variable.

Mutual Information
  • Measures the amount of information that one
    random variable contains about another random
  • Mutual information is a measure of reduction of
    uncertainty due to another random variable.
  • That is, mutual information measures the
    dependence between two random variables.
  • It is symmetric in X and Y, and is always
  • Recall Entropy of a random variable X is H(X).
  • Conditional entropy of a random variable X given
    another random variable Y H(X Y).
  • The mutual information of two random variables X
    and Y is

Measures of Simplicity, Flake ch 9
  • Algorithmic Regularity Simplicity?
  • Things that can be compressed are simpler
  • Formally quantified as Algorithmic Complexity
    (AC) or Kolomogorov Complexity the shortest
    program to produce a result
  • Redundant or enumerable data are simple
  • AC depends on the computer (or complex system)
    that runs the program
  • Random data are complex because you cannot
    represent them excatly except by explicitly
    listing them
  • AC translates between time complexity and space
  • Contrasts with Statistical simplicity
  • Randomness can be simple because you can
    approximate them statistically
  • Coin toss, random walks, Gaussian (normal
  • You can compress random numbers with statistical
    descriptions and only a few parameters

Other Definitions
  • Per Bak Sandpile model and the edge of chaos
  • 1. A powerlaw size distribution of extinction
    events many small and few large
  • 2. The powerlaw lifetime distribution of species
    and genera.
  • 3. The powerlaw distribution of species per
    genus, genera per family, and so on.
  • Stuart Kauffmann The Adjacent possible
  • Autonomous agents have to live the most complex
    game that they can
  • There is a tendency for self-constructing
    biospheres to enlarge their workspace, the
    dimensionality of their adjacent possible
  • Murray Gell-Mann Effective Complexity
  • Complexity is highest for things that are neither
    strictly regular nor strictly random
  • Kolmogorov Complexity is based on the length of a
    concise description of a set
  • Effective complexity is based on the length of a
    concise description of a sets regularities

Neither random nor regular
  • Complex systems occur at a transition point
    between two extremes
  • Often the only way to predict a future state of a
    complex system is to simulate it
  • It cannot be described by a tractable formula or
    short program
  • It cannot be described statistically
  • Some cellular automata are complex the results
    of running them are neither regular nor random
  • Langtons lambda parameter and Wolframs CA
    classification schemes go from regular to
    chaotic, with complex in the middle

Summary of Complexity Measures
  • Computational complexity
  • How many resources does it take to compute a
  • The language/machine hierarchy
  • How complex a machine is needed to compute a
  • Information-theoretic methods
  • Entropy
  • Algorithmic complexity
  • Mutual information
  • Logical depth
  • Run-time of the shortest program that generates
    the phenomena and halts.
  • Asymptotic behavior of dynamical systems
  • Fixed points, limit cycles, chaos.
  • Wolframs CA classification the outcome of
    complex CA can not be predicted any faster than
    it can be simulated.
  • Effective Complexity
  • Neither regular nor random

Suggested References
  • Computational Complexity by Papadimitriou.
    Addison-Wesley (1994).
  • Elements of Information Theory by Cover and
    Thomas. Wiley (1991).
  • Kaufmann, At Home in the Universe (1996) and
    Investigations (2002).
  • Per Bak, How Nature Works The Science of
    Self-Organized Criticality (1988)
  • Gell-Mann, The Quark and the Jaguar (1994)
  • Reading for Wednesday 1/30 (to be posted)
  • Lansing, Complex Adaptive Systems, Ann Rev
    Anthropology 2003
  • www.ic.arizona.edu/lansing/CompAdSys.pdf
  • Gell Mann, What is complexity? 1995
  • http//www.santafe.edu/mgm/complexity.html

(No Transcript)
What is a Random Variable?
  • A function defined on a sample space.
  • Should be called random function.
  • Independent variable is a point in a sample space
    (e.g., the outcome of an experiment).
  • A function of outcomes, rather than a single
    given outcome.
  • Probability distribution of the random variable
  • Example
  • Toss 3 fair coins.
  • Let X denote the number of heads appearing.
  • X is a random variable taking on one of the
    values (0,1,2,3).
  • PX0 1/8 PX1 3/8 PX2 3/8 PX3

Detour into Information Theory cont.
  • Example 2
  • A horse race with 8 horses competing.
  • The probabilities of 8 horses are
  • Calculate the entropy H of the horse race
  • Suppose that we wish to send a (short) message to
    another person indicating which horse won the
  • Could send the index of the winning horse (3
  • Alternatively, could use the following set of
  • 0, 10, 110, 1110, 111100, 111101, 111110, 11111.
  • Average description length is 2 bits (instead of

  • the study of cellular automata, from the early
    discoveries of Stanislaw Ulam and John von
    Neumann through to John Conway's Game of Life and
    the extensive work of Stephen Wolfram, made it
    clear that complexity could be generated as an
    emergent feature of extended systems with simple
    local interactions.
  • Over a similar period of time, Benoît
    Mandelbrot's large body of work on fractals
    showed that much complexity in nature could be
    described by certain ubiquitous mathematical
    laws, while the extensive study of phase
    transitions carried out in the 1960s and '70s
    showed how scale invariant phenomena such as
    fractals and power laws emerged at the critical
    point between phases.

Computers are complex systems
  • Even though we build them, we dont necessarily
    understand how they work.
  • Complexity is overwhelming design strategies
    based on functional decomposition (reductionism)
  • Context-free languages.
  • Object-oriented programming.
  • Conventional parallel and concurrent programming
  • Need new methods for engineering nonlinear
  • Neural networks, Genetic algorithms, Simulated
  • Emergent computation?
About PowerShow.com