Circuit Complexity, Kolmogorov Complexity, and Prospects for Lower Bounds

1
Circuit Complexity, Kolmogorov Complexity, and
Prospects for Lower Bounds

• DCFS 2008

2
Todays Goal

• To raise awareness of the tight connection
  between circuit complexity and Kolmogorov
  complexity.
• And to show that this is useful.
• To plant seeds of optimism, regarding the
  prospects of proving lower bounds in circuit
  complexity.

3
Kolmogorov Complexity

• C(x) mind U(d) x
• Important property
• Invariance The choice of the universal Turing
  machine U is unimportant.
• x is random if C(x) x.
• CA(x) mind UA(d) x

4
Circuit Complexity

• Let D be a circuit of AND and OR gates (with
  negations at the inputs). Size(D) of wires
  in D.
• Size(f) minSize(D) D computes f
• We may allow oracle gates for a set A, along with
  AND and OR gates.
• SizeA(f) minSize(D) DA computes f

5
K-complexity Circuit Complexity

• There are some obvious similarities in the
  definitions. What are some differences?
• A minor difference Size gives a measure of the
  complexity of functions, C gives a measure of the
  complexity of strings.
• Given any string x, let fx be the function whose
  truth table is the string of length 2logx,
  padded out with 0s, and define Size(x) to be
  Size(fx).

6
K-complexity Circuit Complexity

• There are some obvious similarities in the
  definitions. What are some differences?
• A minor difference Size gives a measure of the
  complexity of functions, C gives a measure of the
  complexity of strings.
• A more fundamental difference
• C(x) is not computable Size(x) is.
• The Minimum Circuit Size Problem (MCSP) (x,i)
  Size(x) i.

7
MCSP

• MCSP is in NP, but is not known to be
  NP-complete.
• MCSP is not believed to be in P.
• Factoring is in BPPMCSP.
• Every cryptographically-secure one-way function
  can be inverted in PMCSP/poly.

8
So how can K-complexity and Circuit complexity be
the same?

• C(x) SizeH(x), where H is the halting problem.
• For one direction, let U(d) x. We need a small
  circuit (with oracle gates for H) for fx, where
  fx(i) is the i-th bit of x. This is easy, since
  (d,i,b) U(d) outputs a string whose i-th bit
  is b is computably-enumerable.
• For the other direction, let SizeH(fx) m. No
  oracle gate has more than m wires coming into it.
  Given a description of D (size not much bigger
  than m) and the m-bit number giving the size of
  y in H y m, U can simulate DH and produce
  fx

9
So how can K-complexity and Circuit complexity be
the same?

• C(x) SizeH(x), where H is the halting problem.
• So there is a connection between C(x) and Size(x)
  
• but is it useful?
• First, lets look at decidable versions of
  Kolmogorov complexity.

10
Time-Bounded Kolmogorov Complexity

• The usual definition
• Ct(x) mind U(d) x in time t(d).
• Problems with this definition
• No invariance! If U and U are different
  universal Turing machines, CtU and CtU have no
  clear relationship.
• (One can bound CtU by CtU for t slightly
  larger than t but nothing can be done for
  tt.)
• No nice connection to circuit complexity!

11
Time-Bounded Kolmogorov Complexity

• Levins definition
• Kt(x) mindlog t U(d) x in time t(d).
• Invariance holds! If U and U are different
  universal Turing machines, KtU(x) and KtU(x) are
  within log x of each other.
• Let A be complete for E Dtime(2O(n)). Then
  Kt(x) SizeA(x).

12
Time-Bounded Kolmogorov Complexity

• Levins definition
• Kt(x) mindlog t U(d) x in time t(d).
• Why log t?
• This gives an optimal search order for NP search
  problems.
• Adding t instead of log t would give every string
  complexity x.
• So lets look at how to make the run-time be
  much smaller.

13
Revised Kolmogorov Complexity

• C(x) mind for all i x 1, U(d,i,b)
  1 iff b is the i-th bit of x (where bit i1 of
  x is ).
• This is identical to the original definition.
• Kt(x) mindlog t for all i x 1,
  U(d,i,b) 1 iff b is the i-th bit of x, in time
  t(d).
• The new and old definitions are within O(log x)
  of each other.
• Define KT(x) mindt for all i x 1,
  U(d,i,b) 1 iff b is the i-th bit of x, in time
  t(d).

14
Kolmogorov Complexity is Circuit Complexity

• C(x) SizeH(x).
• Kt(x) SizeE(x).
• KT(x) Size(x).
• Other measures of complexity can be captured in
  this way, too
• Branching Program Size KB(x) mind2s
  for all I x 1, U(d,i,b) 1 iff b is the
  i-th bit of x, in space s(d).

15
Kolmogorov Complexity is Circuit Complexity

• C(x) SizeH(x).
• Kt(x) SizeE(x).
• KT(x) Size(x).
• Other measures of complexity can be captured in
  this way, too
• Formula Size KF(x)
  mind2t for all I x 1, U(d,i,b) 1
  iff b is the i-th bit of x, in time t(d), for
  an alternating Turing machine U.

16
but is this interesting?

• The result that Factoring is in BPPMCSP was first
  proved by observing that, in PMCSP, one can
  accept a large set of strings having large KT
  complexity (and by making use of many important
  results in the theory of pseudorandom generators
  and derandomization).
• (Basic Idea) There is a pseudorandom generator
  based on factoring, such that factoring is in
  BPPT for any test T that distinguishes truly
  random strings from pseudorandom strings. MCSP
  is such a test.

17
This idea has many variants.

• Consider RKT, RKt, and RC.
• RKT is in coNP, and not known to be coNP hard.
• RC is not hard for NP under poly-time many-one
  reductions, unless PNP.
• How about more powerful reductions?
• Is there anything interesting that we could
  compute quickly if C were computable, that we
  cant already compute quickly?
• Proof uses PRGs, Interactive Proofs, and the fact
  that an element of RC of length n can be found in
  
• But RC is undecidable! Perhaps H is in P
  relative to RC?

18
This idea has many variants.

• Consider RKT, RKt, and RC.
• RKT is in coNP, and not known to be coNP hard.
• RC, is not hard for NP under poly-time many-one
  reductions, unless PNP.
• How about more powerful reductions?
• PSPACE is in P relative to RC.
• NEXP is in NP relative to RC.
• Proof uses PRGs, Interactive Proofs, and the fact
  that an element of RC of length n can be found in
  poly time, relative to RC BFNV.
• But RC is undecidable! Perhaps H is in P
  relative to RC?

19
Relationship between H and RC

• Perhaps H is in P relative to RC?
• This is still open. It is known that there is a
  computable time bound t such that H is in
  DTime(t) relative to RC Kummer.
• but the bound t depends on the choice of U in
  the definition of C(x).
• We also know that H is in P/poly relative to RC.

20
This idea has many variants.

• Consider RKT, RKt, and RC.
• What about RKt?
• RKt, is not hard for NP under poly-time many-one
  reductions, unless ENE.
• How about more powerful reductions?
• EXP NP(RKt).
• RKt is complete for EXP under P/poly reductions.
• Open if RKt is in P!

21
Kolmogorov Complexity is Circuit Complexity

• C(x) SizeH(x).
• Kt(x) SizeE(x).
• KT(x) Size(x).
• Other measures of complexity can be captured in
  this way, too
• Formula Size KF(x)
  mind2t for all I x 1, U(d,i,b) 1
  iff b is the i-th bit of x, in time t(d), for
  an alternating Turing machine U.

22
Kolmogorov Complexity is Circuit Complexity

• C(x) SizeH(x).
• Kt(x) SizeE(x).
• KT(x) Size(x).
• Other measures of complexity can be captured in
  this way, too
• A similar definition captures depth k threshold
  circuit size.
• This is the clever transition to start the
  discussion of circuit lower bounds

23
Big Complexity Classes

• NP
• P
• .
• .
• NC
• L (Deterministic Logspace)

24
The Main Objects of InterestSmall Complexity
Classes

• TC0 O(1)-Depth Circuits of MAJ gates
• AC0 6
• NC1 Log-Depth Circuits
• AC0 cant compute Mod 2 FSS,A
• AC0 O(1)-Depth Circuits of AND/OR gates

25
The Main Objects of InterestSmall Complexity
Classes

• TC0 O(1)-Depth Circuits of MAJ gates
• AC0 6
• NC1 Log-Depth Circuits
• AC0 cant compute Mod 2 FSS,A
• AC0 O(1)-Depth Circuits of AND/OR gates

26
The Main Objects of InterestSmall Complexity
Classes

• TC0 O(1)-Depth Circuits of MAJ gates
• NC1 Log-Depth Circuits
• AC0 2 cant compute Mod 3 R,S
• AC0 2
• AC0 O(1)-Depth Circuits of AND/OR gates

27
The Main Objects of InterestSmall Complexity
Classes

• NC1 Log-Depth Circuits
• TC0 O(1)-Depth Circuits of MAJ gates
• AC0 6
• AC0 2
• AC0 O(1)-Depth Circuits of AND/OR gates

28
The Main Objects of InterestSmall Complexity
Classes

• NC1 poly-size formulae
• TC0 O(1)-Depth Circuits of MAJ gates
• AC0 6
• AC0 2
• AC0 O(1)-Depth Circuits of AND/OR gates

29
Complete Problems

• NP has complete sets (under polynomial time
  reducibility P)
• These small classes have complete sets, too
  (under AC)

30
Reductions

• A AC B means that there is a constant-depth
  circuit computing A that has the usual AND and OR
  gates, and also has oracle gates for B.

B
31
Complete Problems

• NC1
• TC0
• AC0 6
• AC0 2
• AC0

• sorting, multiplication, division
• Naor,Reingold Pseudorandom Generator

32
Complete Problems

• NC1
• TC0
• AC0 6
• AC0 2
• AC0

• BFE Balanced Boolean Formula Evaluation
  (AND,OR,XOR)
• Word problem over S5

33
The Word Problem Over S5

• A regular set complete for NC1

34
Complexity Classes are not Invented Theyre
Discovered

• NP (SAT, Clique, TSP,)
• P (Linear Programming, CVP, )
• NL (Connectivity, Shortest Paths, 2SAT, )
• L (Undirected Connectivity, Acyclicity, )
• NC1 (BFE, Regular Sets)
• TC0 (Sorting, Multiplication, Division)

Were interested in NC1 (for instance) not
because we want to build formulae for these
functions
35
Complexity Classes are not Invented Theyre
Discovered

• NP (SAT, Clique, TSP,)
• P (Linear Programming, CVP, )
• NL (Connectivity, Shortest Paths, 2SAT, )
• L (Undirected Connectivity, Acyclicity, )
• NC1 (BFE, Regular Sets)
• TC0 (Sorting, Multiplication, Division)

but because we want to know if the blocks of
this partition are distinct.
36
Complexity Classes are not Invented Theyre
Discovered

• NP (SAT, Clique, TSP,)
• P (Linear Programming, CVP, )
• NL (Connectivity, Shortest Paths, 2SAT, )
• L (Undirected Connectivity, Acyclicity, )
• NC1 (BFE, Regular Sets)
• TC0 (Sorting, Multiplication, Division)

These classes are real. Theyre important.
37
Longstanding Open Problems

• Is P NP?
• Is AC06 NP?
• Is depth 3 AC06 NP?

Well focus on questions such as Is BFE in
TC0? Is BFE in AC06?
38
How Close Are We to Proving Circuit Lower Bounds?

• Conventional Wisdom Not Close At All!
• No new superpolynomial size lower bounds in over
  two decades.
• Razborov and Rudich Any natural argument
  proving a lower bound against a circuit class C
  yields a proof that C cant compute a
  pseudorandom function generator.
• Since the Naor, Reingold generator is
  computable in TC0, this is bad news.

39
More Modest Goals

• Problems requiring formulae of size n3 Håstad
• Problems requiring branching programs of size
  nearly n loglog n Beame, Saks, Sun, Vee
• Problems requiring depth d TC0 circuits of size
  n1c Impagliazzo, Paturi, Saks
• Time-Space Tradeoffs Fortnow, Lipton, Van
  Melkebeek, Viglas
• There is little feeling that these results bring
  us any closer to separating complexity classes.

40
How Close Are We to Proving Circuit Lower Bounds?

• How close are the following two statements?
• TC0 Circuits for BFE must be of size n1O(1)
• For some cgt0, TC0 Circuits for BFE must be of
  size n1c.

41
How Close Are We to Proving Circuit Lower Bounds?

• How close are the following two statements?
• TC0 Circuits for BFE must be of size n1O(1)
• For some cgt0, FTC0 Circuits for BFE must be of
  size n1c

This is known IPS97
This implies TC0 ? NC1 A, Koucky
42
Self-Reducibility

• A set B is said to be self-reducible if B P B

43
Self-Reducibility

• A set B is said to be self-reducible if B P B
  via a reduction that, on input x, does not ask
  about whether x is in B.
• Very well-studied notion.
• For example, f is in SAT if and only if
  (f0 is in SAT) or (f1 is in SAT)

44
Self-Reducibility

• Many of the important problems in (or near) NC1
  have a special self-reducibility property

45
Self-Reducibility

• Many of the important problems in (or near) NC1
  have a special self-reducibility property
  Instances of length n are AC0-Turing (or
  TC0-Turing) reducible to instances of length n½
  via reductions of linear size.
• Examples
• BFE
• the word problem over S5
• MAJORITY
• Iterated Product of 3-by-3 Integer Matrices

46
Self Reducibility

• BFE

A subformula near the root
Subformulae near inputs
47
Self Reducibility

• S5

48
Self Reducibility

• The self-reduction of S5, on inputs of size n,
  uses (n½ 1) oracle gates of size n½.
• Thus if S5 has TC0 circuits of size nk, it also
  has circuits of size (n½ 1)nk/2 O(n(k1)/2).
• Similar arguments hold for other classes (such as
  AC06 and NC1).
• More complicated self-reductions can be presented
  for MAJORITY and Iterated Product of 3-by-3
  matrices.

49
A Corollary

• If BFE has TC0 or AC06 circuits, then it has
  such circuits of nearly linear size.
• If S5 has TC0 or AC06 circuits, then it has
  such circuits of nearly linear size.
• If MAJ has AC06 circuits, then it has such
  circuits of nearly linear size. (Etc.)
• Thus, e.g., to separate NC1 from TC0, it suffices
  to show that BFE requires TC0 circuits of size
  n1.0000001.

50
A Corollary

• If BFE has TC0 or AC06 circuits, then it has
  such circuits of nearly linear size.
• If S5 has TC0 or AC06 circuits, then it has
  such circuits of nearly linear size.
• If MAJ has AC06 circuits, then it has such
  circuits of nearly linear size. (Etc.)
• How widespread is this phenomenon? Is it true
  for SAT? (I.e., can we show NP ? TC0 by proving
  that SAT requires TC0 circuits of size
  n1.0000001?)

51
Limitations of Self-Reducibility

• Any problem for which instances of length n are
  TC0-Turing reducible to instances of length n½
  via poly-size reductions lies in NC.
• Thus there is no obvious way to apply these
  techniques to SAT or to problems complete for P.
• but perhaps, rather than showing directly that
  SAT has this strong form of self-reducibility,
  one can argue that if SAT is in TC0 then it has
  TC0 circuits of nearly-linear size.

52
Limitations of Self-Reducibility

• Any problem for which instances of length n are
  TC0-Turing reducible to instances of length n½
  via poly-size reductions lies in NC.

53
Limitations of Self-Reducibility

• Any problem for which instances of length n are
  TC0-Turing reducible to instances of length n½
  via poly-size reductions lies in NC.

d levels of oracle gates
54
Limitations of Self-Reducibility

• Any problem for which instances of length n are
  TC0-Turing reducible to instances of length n½
  via poly-size reductions lies in NC.

d2 levels of oracle gates
55
Limitations of Self-Reducibility

• Any problem for which instances of length n are
  TC0-Turing reducible to instances of length n½
  via poly-size reductions lies in NC.

After log log rounds, the depth is logO(1)n
d3 levels of oracle gates
56
Prospects for Progress

• We have seen that existing techniques prove
  bounds that are nearly good enough to separate
  NC1 and TC0. Some of these proofs are natural.
• Dont the results of Razborov Rudich indicate
  that further progress will require very different
  approaches?
• Not necessarily!

57
Prospects for Progress

• The Razborov Rudich framework of natural
  proofs assumes that a natural proof of a lower
  bound will make use of a combinatorial property
  that (among other things) is shared by a large
  fraction of the functions on n bits.
• In contrast, we are making use of a
  self-reducibility property that allows us to
  boost a n1e lower bound to a superpolynomial
  lower bound. This self-reducibility property
  holds for only a vanishingly small fraction of
  all functions.

58
Prospects for Progress

• These observations are simple, but
• they have forever changed the way that we look at
  quadratic (and smaller) lower bounds.
• We are not claiming to have found a way around
  the obstacles identified by Razborov Rudich.
  (Such a claim will have to wait until someone
  proves that NC1 ? TC0.) But we do believe that
  this avenue deserves further exploration.

59
Conclusion

• There are good reasons to develop and explore the
  connections between Kolmogorov complexity and
  circuit complexity.
• There are many open problems in this area that I
  will be delighted to discuss with you in more
  detail.
• There are two bad typos in the proceedings
  version of the paper. (P should be NP.) A
  corrected version is available at my home page.

60
Speculation
Connections between Kolmogorov Complexity and
Circuit Complexity might be relevant to the
question of whether NEXP is contained in
(non-uniform) TC0 (depth 3).
61
Speculation

• IKW showed that NEXP is in P/poly iff
  NEXP MA
  iff MA cannot be derandomized
• The proof shows that NEXP is in P/poly iff
  every set in P contains strings of KT-complexity
  O(log n) iff
  NEXP IPP/poly.

62
Speculation

• Similar techniques show
• NEXP is in nonuniform NC1
  iff every set in P contains strings of
  KF-complexity O(log n)
  iff NEXP MIPNC1
  iff MIPNC1 cannot be derandomized.
• NEXP is in nonuniform TC0
  iff every set in P contains strings of small
  complexity
  iff NEXP MIPTC0
  iff MIPTC0
  cannot be derandomized.

63
Speculation

• What else happens in such a collapse?
• If NP uniform TC0, then P is not contained in
  non-uniform TC0 (so NEXP is not in non-uniform
  TC0).
• So lets consider NEXP MIPTC0 and
  NP ? uniform TC0. If this hardness assumption
  were sufficient to derandomize MIPTC0 then this
  would give the desired lower bound on NEXP
• Fortnow, Klivans, van Melkebeek, Santhanam