Programming Languages for Compressing Graphics

1
Programming Languages for Compressing Graphics

• Morgan McGuire
• Shriram Krishnamurthi
• John F. Hughes
• Brown University
• morgan sk jfh_at_cs.brown.edu

2
Encoding Images as Programs
3
Describing Images (1)

• setPixel(0,0,BLUE)
• setPixel(1,0,BLUE)
• setPixel(2,0,BLUE)
• setPixel(50,0,WHITE)
• setPixel(100,0,RED)
• setPixel(101,0,RED)

4
The Cost of Bandwidth

• Major cost of doing business on the web
• Yahoo! 65,000,000 pages/day
• Valve 1M /software patch
• Image compression is a one-shot activity
• Low vs. high bandwidth users
• Multiresolution

5
Describing Images (2)

• repeat 50 times setNextPixel(BLUE)
• repeat 50 times setNextPixel(WHITE)
• repeat 50 times setNextPixel(RED)
• nextRow()
• repeat 50 times setNextPixel(BLUE)

6
American Flag
Doesnt compress as well as the French flag in
the repeat n times language.
7
Adding More Primitives

• drawRectangle(BLUE, )
• drawRectangle(RED, )
• drawRectangle(WHITE, )
• drawStar(WHITE, )

Is this enough?
8
Describing Complex Images
9
Describing Complex Images

• JPEG language consists of frequency domain
  instructions
• GIF language consists of setPixel and dictionary
  lookup
• Preferred format depends on the image

10
Observations

• Images are programs
• Even within one language, many possible
  descriptions produce similar images
• Lossy compression
• Description length depends on language and image
  complexity
• Best compression when the language matches the
  image

11
The Obvious Compression Scheme

• Compress the image in several formats
• TGA, GIF, JPG, SVG, SWF
• Choose the best
• Add a byte to the front of the file specifying
  the compression language

12
Problems with the Obvious Scheme

• None of the formats may be particularly good for
  our image
• Even JPEG tops out around 501, has serious
  artifacts
• Lacking ideal features like
• Multiresolution
• Time/Space tradeoff
• Introduction of new formats requires new browser
  plug-ins

13
Describing Complex Images
How can we do better than the obvious approach
for images like this?
14
Using an Expressive Language
(compose (vertical-gradient BLUE WHITE)
(polygon DARK-BLUE ) (polygon BLACK ) (
(polygon ) (blur tree-texture)
YELLOW))
15
Using an Expressive Language

• What if we design a really expressive language
  for representing images?
• Because the data is a program the decompressor
  is part of the data
• Each image gets its own custom format
• High compression
• We have control over multiresolution, perceptual
  artifacts
• Package for the web as a plug-in
• We only need to upgrade the plug-in when the
  language itself changes

Whats wrong with this plan?
16
Example
setPixel Language
Expressive Language
?
setPixel(0,0,WHITE) setPixel(1,0,WHITE) setPixel
(2,0,WHITE)
(blur ( (rotate (banana)) (distort ( RED
(circle))) )
2 Mb
2 kb
17
The Encoding Problem
?
Pierre cant code!
18
The Encoding Problem
Photoshop menu
EMACS
setPixel(0,0,WHITE) setPixel(1,0,WHITE) setPixel
(2,0,WHITE)
(blur ( (rotate (banana)) (distort ( RED
(circle))) )
2 Mb
2 kb
19
Where is the Save as Code Menu?

• It is easy to convert from an image to programs
  in the GIF/JPG language
• More expressive language harder conversion
• How much harder?

20
Much Harder!

• Converting the image into a program that produces
  it is a search problem
• The search space is the space of all possible
  programs
• This is an infinitely large space

21
Tempering Expressiveness

• Good compression languages are ones where
• Expressive power is large
• Searching is easy
• How do we make searching easy?

22
Steerable Search Techniques

• Genetic Algorithms
• Inject domain information through fitness
  function
• Metropolis Search
• Inject domain information through transition
  probabilities
• Simulated Annealing
• Inject domain information through gradient
  estimation

23
Perceptual Fitness Function
e (i ) edge filter, b (i ) convolve with
Gaussian, ix color magnitude

• Tweaking this is the domain-experts job
• Perfect fitness function not necessary (or
  possible!)

24
Designing the Language

• Desirable language properties for compression
• Expresses many images compactly
• There are many programs for which another,
  shorter program exists that produces visually
  similar output
• Desirable search properties
• Mutations safety
• All programs terminate

25
The Evolver Language
Scalar real between 0 and 1 Vector Scalar
x Scalar x Scalar Matrix Vector Value
Matrix Scalar Vector Operator Add
Collage Blur Noise Call Operator x
Expr Expr Call Scalar Vector
26
The Evolver Language

• Automatic coercion between Matrix, Vector, Scalar
• Every operator has the same domain and range
• Primitives include stock images and textures
• No looping constructs
• No functions!

27
l Not the Ultimate Compressor!
28
l Not the Ultimate Compressor!

• Copying code is sometimes good
• Multiple instances of a pattern in an image often
  differ slightly
• Hard to evolve both definition and multiple
  applications

29
Added Benefits of Search

• Artist can save the image immediately
• Webmaster uses Evolver toolkit to search for an
  equivalent program
• It is easy to find a large program quickly
• Webmaster lets Evolver continue to run
• Upload new, smaller encodings as they are found
• More time less space less cost
• Multiple constraints size, time, artifacts,
  decompression time

30
Results
31
Proof of Concept
TGA 11 (128x128)
32
Proof of Concept
JPG 241 (64x64)
33
Proof of Concept
Evolver 501, infinite detail resolution
34
Proof of Concept
TGA
Evolver
JPG
35
Lake Matheson
Original
36
Lake Matheson
Original
Compressed 501
37
Gradient
Original
Compressed
38
Aspens
Original
39
Aspens
Original
Compressed 541
40
Maples
Original
41
Maples Code
Collage(HueShift(HueShift(Min(RockImage(),
Collage(Rotate(0.22693111, 0.4339271,
-0.060890462, -0.14983156), 0.9689341,
-0.31166452, 1.0, EnvironmentMap(Interpolate(Deri
vative(0.5260445, -0.9943271, -0.83629435),
FishImage(), 0.22693111), Collage(VSplit(VGradient
(), Interpolate(Min(RockImage(),
Collage(0.22693111, 0.90638816, -0.3161332,
1.0, EnvironmentMap(0.3538252, -0.11179591,
0.76402247, 0.3538252, -0.11179591,
0.76402247))), -0.75621074, Derivative(HueShift(L
owColorNoise(), -0.60136193, -0.9961748,
0.956824)))), -0.6025814, -0.5151359,
-0.2444776, MiniBlur(Blur(0.48381335,
0.37744927, 0.18049468)))))), LeafImage()),
LeafImage()), Rotate(Rotate(Interpolate(Max(Rotate
(LeafImage(), Cosine(0.13357106, -0.48899084,
0.46273336)), LeafImage()), 0.26036343,
-0.2474052, 0.3318561, Add(Rotate(Interpolate(Max
(LowNoise(), LeafImage()), 0.26036343,
-0.2474052, 0.3318561, Add(Rotate(Interpolate(0.
26036343, -0.2474052, 0.3318561, 0.26036343,
-0.2474052, 0.3318561, Max(Rotate(Blur(Noise()),
FrequencyStars()), BitAnd(VSplit(LeafImage(),
-0.3782095, 0.06973941, 0.7708523),
Collage(Blur(0.19214037, 0.7060751, 0.9632803),
-0.94875985, 0.9535051, 0.9628181,
-0.94875985, 0.9535051, 0.9628181)))),
FrequencyStars()), Zoom(0.84155905, 0.44450688,
-0.6368634, Interpolate(0.97325927, 0.43938103,
0.8003519, -0.8865588, FrequencyStars())))),
FrequencyStars()), Zoom(1.0, 0.35874906,
-0.42753658, 1.0, 0.35874906, -0.42753658))),
FrequencyStars()), 0.30287892, -0.7879979,
0.756324), Distort(Distort(HueShift(Distort(Disto
rt(Distort(HueShift(Distort(Rotate(Distort(Distort
(HueShift(Distort(Rotate(Distort(Distort(Blur(Dist
ort(Distort(RockImage(), ArcTangent(ExpandRange(Co
lorNoise()))), ArcTangent(ExpandRange(ColorNoise()
)))), ArcTangent(ExpandRange(ColorNoise()))),
ArcTangent(ColorNoise())), RockImage()),
ArcTangent(ColorNoise())), RockImage()),
ColorNoise()), Blur(Add(0.214469, -0.05106278,
-0.8334819, Blur(Blur(-0.44914088, 0.86714524,
-0.038012877))))), Distort(Distort(Blur(Distort(D
istort(-0.32682085, SunriseImage()),
ArcTangent(0.84263784))), LeafImage()),
ArcTangent(ColorNoise()))), ArcTangent(ColorNoise(
))), ExpandRange(RockImage())), 0.315652),
ColorNoise()), Blur(ExpandRange(Add(0.44753784,
0.15750253, -0.9017423, 0.214469, -0.05106278,
-0.8334819)))), ExpandRange(RockImage())),
ColorNoise()), Blur(Rotate(0.10588625,
0.2359776, -0.20337643, 0.2809281, -0.97692156,
-0.49766022))))
42
Maples
Original
Compressed 561
43
Multi-resolution
JPEG 141
Evolver 561
44
Multi-resolution
JPEG 141
Evolver 561
45
Related work

• Searching for programs
• Massalins Superoptimizer
• Frigo Johnsons FFTW
• Palsberg, Lucier Mamillapalli
• Karl Sims
• Programmatic image compression
• Fractal compression
• MPEG-7
• Steerable search techniques in graphics
• MLT
• Radiosity

46
Conclusions

• It is tractable to search the space of all
  programs!
• The visually similar criterion makes computer
  graphics an interesting domain
• Keep using JPEG for now

47
Future directions

• Improve image quality/compression
• How can the design of searchable languages be
  formalized?
• How do expressive constructs affect the search
  problem?
• Other interesting domains animation, audio
  compression, image search, robot controllers

48
Questions

• http//www.cs.brown.edu/people/morgan/evolver