PACKMAN Texture Compression for Mobile Phones

1
PACKMAN Texture Compression for Mobile Phones
Jacob Ström, Tomas Akenine-Möller Ericsson
Research
2
Outline

• Motivation and Previous work
• Design Goals
• Basic Idea
• Decompression of a Block
• Compression
• Results

3
Why 3D Graphics on a Mobile Phone?

• Man-Machine Interfaces

• Screen Savers

• Games
• Maps, Messaging, Browsing and more...

4
Why is 3D Graphics Hard on a Mobile Phone?

• Limited resources
• Small amount of memory
• Little memory bandwidth
• Little chip area for special purpose
• Powered by batteries

5
Texture Compression Helps

• Small amount of memory
• More texture data can fit in the limited amount
  of memory
• Little memory bandwidth
• More texturing possible for same amount of
  bandwidth
• Little chip area for special purpose
• A texture cache using compressed data can be made
  smaller
• Powered by batteries
• Reduced bandwidth means lower energy consumption

6
Previous Work
7
Previous Work
8
Previous Work
9
Previous Work
10
Design Goals

• Major Design Goals
• Minimal decompression complexity

• Acceptable quality
• Minor Design Goals
• Low compression complexity
• Small block size
• Reasonable compression (around 4 bpp)

11
Early Design Decisions

• Block size of 32 bits was chosen
• Few bits -gt small bit widths after texture cache
• Fits 32-bit wide buses on systems without texture
  cache
• No indirect addressing of colors
• LUT of colors increases latency and complicates
  hardware
• Implications
• 2x4 was the only reasonable block size, 4 bpp

12
Basic Idea

• The Human Visual System (HSV) is more sensitive
  to luminance than to chrominance
• In video and still image coding, chrominance
  information is most often subsampled in the x-
  and y- direction (MPEG, JPEG, H263, H264 etc).
• Our scheme has basically only one color per 2x4
  block. The rest is luminance information

13
Basic Idea

• Use only 12 bits to specify a general color for
  a 2x4 block

12-bit general color
14
Basic Idea

• Use only 12 bits to specify a general color for
  a 2x4 block
• Modify the luminance for each pixel in the block

12-bit general color
per-pixel luminance
15
Basic Idea

• Use only 12 bits to specify a general color for
  a 2x4 block
• Modify the luminance for each pixel in the block

12-bit general color
per-pixel luminance
resulting image
16
Luminance modification
Luminance is modified additively Example
general block color is R17, G34, B
51 Luminance modifier for the pixel is 220, The
new pixel value is R 17220 237G 34220
254B 51220 255 (after clamping)
17
How to specify luminance

• Two bits per pixel are used to specify the
  luminance modifier is one out of four values.
• Problem Small values -8, -2, 2, 8 ?

18
How to specify luminance

• Two bits per pixel are used to specify the
  luminance modifier is one out of four values.
• Problem Small values -8, -2, 2, 8 ?
• smooth transitions OK

19
How to specify luminance

• Two bits per pixel are used to specify the
  luminance modifier is one out of four values.
• Problem Small values -8, -2, 2, 8 ?
• smooth transitions OK
• sharp edges bad

20
How to specify luminance

• Two bits per pixel are used to specify the
  luminance modifier is one out of four values.
• Problem Small values -8, -2, 2, 8 ?
• smooth transitions OK
• sharp edges bad
• Big values -255, -127, 127, 255 ?

21
How to specify luminance

• Two bits per pixel are used to specify the
  luminance modifier is one out of four values.
• Problem Small values -8, -2, 2, 8 ?
• smooth transitions OK
• sharp edges bad
• Big values -255, -127, 127, 255 ?
• smooth transitions bad

22
How to specify luminance

• Two bits per pixel are used to specify the
  luminance modifier is one out of four values.
• Problem Small values -8, -2, 2, 8 ?
• smooth transitions OK
• sharp edges bad
• Big values -255, -127, 127, 255 ?
• smooth transitions bad
• sharp edges OK

23
How to specify luminance

• Two bits per pixel are used to specify the
  luminance modifier is one out of four values.
• Problem Small values -8, -2, 2, 8 ?
• smooth transitions OK
• sharp edges bad
• Big values -255, -127, 127, 255 ?
• smooth transitions bad
• sharp edges OK
• Solution Codebook of tables, one/block.

24
Modifier Codebook

• We started with random values and optimized by
  minimizing the error for a set of images

25
Modifier Codebook

• We started with random values and optimized by
  minimizing the error for a set of images

26
Modifier Codebook

• We started with random values and optimized by
  minimizing the error for a set of images

• simulated annealing
• modified version of the Generalized Lloyd
  algorithm (Linde Buzo Gray 80)
• Symmetry was added to reduce on-chip codebook size

27
Modifier Codebook

• We started with random values and optimized by
  minimizing the error for a set of images

• simulated annealing
• modified version of the Generalized Lloyd
  algorithm (Linde Buzo Gray 80)
• Symmetry was added to reduce on-chip codebook size

28
Modifier Codebook

• We started with random values and optimized by
  minimizing the error for a set of images

• simulated annealing
• modified version of the Generalized Lloyd
  algorithm (Linde Buzo Gray 80)
• Symmetry was added to reduce on-chip codebook size

29
Modifier Codebook

• We started with random values and optimized by
  minimizing the error for a set of images

• simulated annealing
• modified version of the Generalized Lloyd
  algorithm (Linde Buzo Gray 80)
• Symmetry was added to reduce on-chip codebook size

30
Decompressing a Block

• First 12 bits is RGB444 which gives the general
  color for the entire block. It is extended to 24
  bits.

153 153 85
extend to 24 bits
12 bit RGB444
31
Decompressing a Block

• Next 4 bits select a table from the code book

5
12 bit RGB444
32
Decompressing a Block

• Next 4 bits select a table from the code book

5
12 bit RGB444
33
Decompressing a Block

• The next 2 bits modify the first pixel according
  to the table

5
10
12 bit RGB444
34
Decompressing a Block

• The next 2 bits modify the first pixel according
  to the table

5
10
12 bit RGB444
35
Decompressing a Block

• The next 2 bits modify the first pixel according
  to the table and so on

5
11
10
12 bit RGB444
36
Decompressing a Block

• The next 2 bits modify the first pixel according
  to the table and so on

5
11
10
01
12 bit RGB444
37
Decompressing a Block

• The next 2 bits modify the first pixel according
  to the table and so on

5
11
10
01
01
12 bit RGB444
38
Decompressing a Block

• The next 2 bits modify the first pixel according
  to the table and so on

5
11
10
01
01
10
12 bit RGB444
39
Decompressing a Block

• The next 2 bits modify the first pixel according
  to the table and so on

5
11
10
01
01
10
01
12 bit RGB444
40
Decompressing a Block

• The next 2 bits modify the first pixel according
  to the table and so on

5
11
10
01
01
10
01
11
12 bit RGB444
41
Decompressing a Block

• The next 2 bits modify the first pixel according
  to the table and so on

5
11
10
01
01
10
01
11
11
12 bit RGB444
42
Simple Decompression

• Only three adders and some muxes.
• Only twelve adders for four parallel units needed
  for bilinear interpolation
• S3TC 60 adders
• PVRTC 60 adders

43
Simple Decompression

• The correct texel is selected

44
Simple Decompression

• The correct texel is selected
• The modifier value is looked up

45
Simple Decompression

• The correct texel is selected
• The modifier value is looked up
• The general color is extended to 24 bits

46
Simple Decompression

• The correct texel is selected
• The modifier value is looked up
• The general color is extended to 24 bits
• The modifier value is added

47
Simple Decompression

• The correct texel is selected
• The modifier value is looked up
• The general color is extended to 24 bits
• The modifier value is added
• The result is clamped

48
Compression

• To compress a block, we need to find
• general color

general color
49
Compression

• To compress a block, we need to find
• general color
• table

general color
table
50
Compression

• To compress a block, we need to find
• general color
• table
• pixel indices.

pixel indices
general color
table
51
Average Compression

• In average compression, the average color of the
  2x4 block is used as general color.

• Exhaustive search is used for the table and the
  pixel indices.
• 30 milliseconds for a 64 x 64 texture.

pixel indices
general color
table
52
Exhaustive Compression

• In exhaustive compression, exhaustive search is
  used for the general color as well (optimal
  compression).
• One minute for a 64x64 texture.
• On average, about 1.5 dB better than average
  compression.

53
Exhaustive Compression

• Why is Exhaustive Compression better than
  Average?
• Often they represent the same 24 bit color, but
  it has been quantized differently.

54
Quantization of General Color

• When Quantizing to 12 bits, we go from many
  positions in RGB space to relatively few.

G
R
55
Quantization of General Color

• When Quantizing to 12 bits, we go from many
  positions in RGB space to relatively few.

G
R
56
Quantization of General Color

• When Quantizing to 12 bits, we go from many
  positions in RGB space to relatively few.
• Ordinary Quantizing just chooses the closest one

G
R
57
Quantization of General Color

• When Quantizing to 12 bits, we go from many
  positions in RGB space to relatively few.
• Ordinary Quantizing just chooses the closest one

G
R
58
Quantization of General Color

• However, the per-pixel luminance modification can
  compensate for errors in the direction (1,1,1)

G
R
59
Quantization of General Color

• However, the per-pixel luminance modification can
  compensate for errors in the direction (1,1,1)
• We can reach (almost) all points on the dotted
  lines

G
R
60
Quantization of General Color

• However, the per-pixel luminance modification can
  compensate for errors in the direction (1,1,1)
• We can reach (almost) all points on the dotted
  lines

G

• The best quantization point is therefore
  different.

R
61
Quantization of General Color

• However, the per-pixel luminance modification can
  compensate for errors in the direction (1,1,1)
• We can reach (almost) all points on the dotted
  lines

G

• The best quantization point is therefore
  different.

R
62
Quantization of General Color

• However, the per-pixel luminance modification can
  compensate for errors in the direction (1,1,1)
• We can reach (almost) all points on the dotted
  lines

G

• The best quantization point is therefore
  different.

R
63
Combined Quantization Compression

• We call this type of quantization combined
  quantization, since the R, G and B values are
  now quantized together to the closest line.

pixel indices
general color
table
64
Combined Quantization Compression (cont.)

• On average, Combined Quantization Compression
  yields 1.0 dB better PSNR than average Compression

65
Combined Quantization Compression (cont.)

• On average, Combined Quantization Compression
  yields 1.0 dB better PSNR than average
  Compression
• Only 0.5 dB worse than Exhaustive search

PSNR
0.5 dB
Average
Exhaustive
Combined
66
Combined Quantization Compression (cont.)

• On average, Combined Quantization Compression
  yields 1.0 dB better PSNR than average
  Compression
• Only 0.5 dB worse than Exhaustive search
• Still only 30 milliseconds to compress a 64x64
  texture.

PSNR
0.5 dB
Average
Exhaustive
Combined
67
Examples
68
Error Metric

• When selecting one way to compress a block over
  another, an error metric is used to tell which is
  better.
• An obvious error metric to use is to sum the
  squared error over the block
• e2 (R-R)2 (G-G)2 (B-B)2
• However, since the eye is more sensitive to
  errors in green than in red and blue, it makes
  sense to add more weight to the green component
• e2 wR(R-R) 2 wG(G-G) 2 wB(B-B) 2

69
Error Metric (cont.)
70
Image Quality

• Peak Signal to Noise Ratio (PSNR) was measured
  for a number of images.
• Proposed scheme on average 1 dB worse than S3TC
• However, luminance component on average 0.5 dB
  better

71
Game Example
Scene with original textures
72
Game Example
Scene with PACKMAN compressed textures
73
Game Example
Scene with original light maps
74
Game Example
Scene with PACKMAN compressed light maps
75
Game Example
Scene with original textures and light maps
combined
76
Game Example
Scene with PACKMAN compressed textures and light
maps combined
77
In the end, the rendered images should be
compared.
original textures
PACKMAN textures
78
Weaknesses

• Block cannot contain two different hues
• E.g., A block with red and blue of same
  luminance.

79
Conclusion

• Emphasis on low complexity
• 20 of the adders of rivaling schemes
• Quality is about 1 dB worse than S3TC
• A fast and near optimal compression scheme has
  been developed
• Perceptual quality can be enhanced using weighted
  error metrics
• PACKMAN is implemented in Bitboys latest
  graphics processors for mobile phones (announced
  yesterday).

80
Thanks

• www.gametutorials.com for their Quake3 scene and
  BSP-culler.
• Timo Aila for feedback on the sketch.

81
(No Transcript)