Bit Sequential bSQ Data Model and Peano Count Trees Ptrees Department of Computer Science North Dako

1
Bit Sequential (bSQ) Data ModelandPeano Count
Trees (P-trees)Department of Computer
ScienceNorth Dakota State University, USA(the
bSQ and P-tree technology is patented by NDSU)
2
Background on Spatial Data

• Pixel a point in a space
• Band feature attribute of the pixels
• Value usually one byte (0255)
• Images have different numbers of bands
• TM4/5 7 bands (B, G, R, NIR, MIR, TIR, MIR2)
• TM7 8 bands (B, G, R, NIR, MIR, TIR, MIR2, PC)
• TIFF 3 bands (B, G, R)
• Ground data individual bands (Yield, Moisture,
  Nitrate level, Temperature, elevation)

3
RSI dataset example
RSI data can be viewed as collection of pixels.
Each pixel has a value for each feature attribute
TIFF image
Yield Map
For example, the RSI dataset above has 320 rows
and 320 columns of pixels (102,400 pixels) and 4
feature attributes (B,G,R,Y). The (B,G,R)
feature bands are in the TIFF image and the Y
feature is color coded in the Yield Map.
4
Spatial Data Formats

• Existing formats
• BSQ (Band Sequential)
• BIL (Band Interleaved by Line)
• BIP (Band Interleaved by Pixel)
• New format
• bSQ (bit Sequential)

5
Spatial Data Formats (Cont.)

• BAND-1
• 54 127
• (1111 1110) (0111 1111)
• 4 193
• (0000 1110) (1100 0001)

• BAND-2
• 7 240
• (0010 0101) (1111 0000)
• 00 19
• (1100 1000) (0001 0011)

BSQ format (2 files) Band 1 254 127 14
193 Band 2 37 240 200 19
6
Spatial Data Formats (Cont.)

• BAND-1
• 54 127
• (1111 1110) (0111 1111)
• 4 193
• (0000 1110) (1100 0001)

• BAND-2
• 7 240
• (0010 0101) (1111 0000)
• 00 19
• (1100 1000) (0001 0011)

BSQ format (2 files) Band 1 254 127 14
193 Band 2 37 240 200 19
BIL format (1 file) 254 127 37 240 14 193
200 19
7
Spatial Data Formats (Cont.)

• BAND-1
• 54 127
• (1111 1110) (0111 1111)
• 4 193
• (0000 1110) (1100 0001)

• BAND-2
• 7 240
• (0010 0101) (1111 0000)
• 00 19
• (1100 1000) (0001 0011)

BSQ format (2 files) Band 1 254 127 14
193 Band 2 37 240 200 19
BIL format (1 file) 254 127 37 240 14 193
200 19
BIP format (1 file) 254 37 127 240 14
200 193 19
8
Spatial Data Formats (Cont.)

• BAND-1
• 54 127
• (1111 1110) (0111 1111)
• 4 193
  
• (0000 1110) (1100 0001)

• BAND-2
• 7 240
• (0010 0101) (1111 0000)
• 00 19
• (1100 1000) (0001 0011)

BSQ format (2 files) Band 1 254 127 14
193 Band 2 37 240 200 19
BIL format (1 file) 254 127 37 240 14 193
200 19
BIP format (1 file) 254 37 127 240 14 200
193 19
9
bSQ Format

• Split each band into eight separate files, one
  for each bit position.
• Reasons of using bSQ format
• Different bits contribute to the value
  differently.
• bSQ format facilitates the representation of a
  precision hierarchy (from 1 bit up to 8 bit
  precision).
• bSQ format facilitates the creation of an
  efficient data structure P-tree, P-tree algebra
  and T-cube.

10
The tabular formats (inverted list)

• BSQ and bSQ are tabular formats
• BSQ consist of a separate table for each feature
  band
• bSQ consist of a separate table for each bit of
  each band
• One can view it this way
• The data set is initially one relation or
  table, R(K1,..,Kk, A1, A2, , An) where K1,..,Kk
  are the structure attributes and each Ai is a
  feature attribute.
• The structure attributes of a 2-D image are the X
  and Y coordinates of the pixels (rows).
• The feature attributes are the bands, B,G,R, NIR,
  
• In BSQ we separate each feature into a separate
  file and suppress the structure attributes
  altogether (under the assumption that the pixels
  are always arranged in raster order.
• In bSQ we separate each bit of each feature into
  a separate file (same raster order assumption)

11
Peano Count Tree (P-tree)

• P-tree represents spatial bSQ data bit-by-bit in
  a recursive quadrant-by-quadrant arrangement.
• An P-tree is a lossless representation of the
  original data.
• A P-tree is a compressed structure.
• A P-tree is count pre-computed.

12
An example of Ptree

• Peano or Z-ordering
• Pure (Pure-1/Pure-0) quadrant
• Root Count

• Level
• Fan-out
• QID (Quadrant ID)

13
An example of Ptree
?Level-3
?Level-2
?Level-1
?Level-0

• Peano or Z-ordering
• Pure (Pure-1/Pure-0) quadrant
• Root Count

• Level
• Fan-out
• QID (Quadrant ID)

14
P-tree variation PM-tree

• Peano Mask tree (PM-tree) uses mask instead of
  count.
• 1 denotes pure-1, 0 denotes pure-0 and m denotes
  mixed.
• It provides an efficient way for ANDing.

15
Ptree Algebra

• And
• Or
• Complement
• Other (XOR, etc)

Ptree 55
____________/ / \ \___________ /
___ / \___ \
/ / \
\ 16 ____8__
_15__ 16
/ / \ / \ \
3 0 4 1 4 4 3 4
//\ //\
//\ 1110 0010
1101
Complement 9
____________/ / \ \___________ /
___ / \___ \
/ / \
\ 0 ____8__
__1__ 0
/ / \ / \ \
1 4 0 3 0 0 1 0
//\ //\
//\ 0001 1101
0010
16
Ptree ANDing Operation
Using Depth-first Pure-1 path code
0 100 101 102 12 132 20 21 220 221 223 23 3
0 20 21 22 231 ? RESULT 0

0 ? 0
20
20 ?
20 21
21
? 21
220 221 223
22 ? 220 221 223
23
231 ? 231
17
Alternative forms for Ptrees (all lossless)
1 means quadrant is pure-1, 0 otherwise (pure0
if no sub-tree ptrs, otherwise mixed)
1 means quadrant is pure-0, 0 otherwise
1 means quadrant is Not pure-Zero, 0 otherwise
(Note PM PNZ XOR P1 )
00
01
11
10
00
01
00
01
10
10
11
11
Vector forms (A table entry for each mixed inode
containing its qid and its children bit-vector
Eliminate need for subtree pointers)
Since there is no qid01.01 in the table we
know its pure0, not mixed
18
Basic, Value and Tuple Ptrees
Basic Ptrees (i.e., P11, P12, , P18, P21, ,
P28, , P71, , P78)
AND
Value Ptrees (i.e., P1, 001 P11 AND P12 AND
P13)
AND
Tuple Ptrees (i.e., P001, 010, 111 P1, 001
AND P2, 010 AND P3, 111)
19
Example
B13
B12
B11
Blue Band, B1, with 3-bit values
Redundant! Since no mixed at this level.
RC(P 1,6)RC(P1,110 )RC(P 11P 12P13). At
P11,6 P11P12PNZ13 (since P13 is
P0V13PNZ13). Calculate COUNTP1Cnt4level
PNZ1,6PNZ11PNZ12P113 (since PN113 P113).
Recursively get P1Cnt4level from each
mixed child (PMV1,6P01,6P11,6). L-2 P1
1001100010001000, contrib 14216, PNZ
111110101110 1010, P1 1000,
PM 0010 L-1 P110 1101111000010000,
contrib 0410, PNZ101111111110011001
, P1100000, PM101001 L-0 P110.00
1111111101110111, contrib 340 3 Level-0
P110.11 1111011111110111, contrib 340 3
TOT22
20
Distributed P trees?
P11
P13
P12
Assume a 5-computer cluster NodeC, Node00,
Node01, Node10, Node11. Send to Nodeij if qid
ends in ij
Bp qid NZ P1 00 1101.00 1110 1310.00 100
0
Bp qid NZ P1 01 1101 1011 0010 1301 1111
1110
Bp qid NZ P1 C 11 1111 1001 12
1010 1000 13 0111 0001
Bp qid NZ P1 10 1110 1111 1101 1110.10 1
101 1210 1111 1110 1310 1110 0110
Bp qid NZ P1 11 1101.11 0010 1210.11 011
1 1301.11 0110
Distributed Request Send request, RC(P 1,6)
to NodeC (if Root Count for a subquadrant
only, send to that qid, eg, 10 RC(P 1,6)
NodeC does P1 1001100010001000.
Count14216. PNZ 1111101011101010. P1
1000. PM 0010. Send request RC(P
1,6) to Node10. Node10 does P110
1101111000010000. Count0.
PNZ101111111110011001. P1100000.
PM101001. Send request RC(P 1,6) to
Nodes00 Node11 Node00 does P110.00
1111111101110111, contribution is 340 3
Send (3) to NodeC (all nodes return counts to
C) Node11 does P110.11 1111011111110111,
contribution is 3403 Sends (3) to
NodeC NodeC does add up Counts 16 3 3
22.
All nodes return counts to the original root (in
this case, NodeC)