Design and Implementation of the Recursive Virtual Address Space Model for Small Scale Multiprocessor Systems

1
Design and Implementation of the Recursive
Virtual Address Space Model for Small Scale
Multiprocessor Systems

• Diploma Thesis
• Marcus Völp
• Supervisor Volkmar Uhlig
• Universität Karlsruhe

2
recursive virtual address space model
Policy S2a
Policy S2b
Policy S1a
Policy S1b
Policy S1
Policy S2
Policy S0
microkernel
3
recursive virtual address space model
map
4
recursive virtual address space model
unmap
5
recursive virtual address space model
grant
6
recursive virtual address space model
Policy S2a
Policy S2b
Policy S1a
Policy S1b
Policy S1
Policy S2
Policy S0
microkernel
7
recursive virtual address space model
s0

• mapping database
• keep track of mappings
• to implement unmap
• No long interrupt latencies
• No unbounded priority inversion
• No helping

Hazelnut Pistachio unmap with interrupts
disabled
8
recursive virtual address space model
s0

• mapping database
• keep track of mappings
• to implement unmap
• No long interrupt latencies
• No unbounded priority inversion
• No helping

unbounded priority inversion Hazelnut
Pistachio (if interrupts enabled) Fiasco
(if helping disabled)
9
recursive virtual address space model
s0

• mapping database
• keep track of mappings
• to implement unmap
• No long interrupt latencies
• No unbounded priority inversion
• No helping

• Helping problematic for SMP
• xcpu time donation
• migration to helper cpu

10
Outline

• Motivation
• Problem analysis
• Unbouned priority inversion inherent to
• pre-order tree traversal
• (if not helped out)
• Unbounded priority inversion inherent
  tonon-post-order tree traversal
• Mapping database
• Data Structures
• Locking Scheme
• Restart Point Tracking
• Performance
• Conclusion

11
Problem Analysis

• Pre-Order tree traversal
• tb in sb unmap ( b )
• remove node c
• preemption
• ta unmap ( a )
• Priority ta gt tb
• Solutions
• roll forward unmap (b)
• help tb
• keep node c
• (last thread removes it)

a
b
c
D
12
Problem Analysis
a

• Unbound priority inversion is inherent to non
  post-order tree traversal
• General traversal algorithm
• i child nodes before parent
• k child nodes after parent
• Post order traversal
• k 0 for all nodes

b


1
i
1
k
13
Conclusion from problem analysis

• Post order tree traversal
• Find leaf node to start with
• Collapse tree from leaf to root of subtree

14
Mapping Database

• Datastructure
• fast insertion (map)
• find start-leafnode
• post order traversal
• Synchronization
• SMP safe
• avoid Priority inversion
• avoid Starvation
• gt Restart Point Tracking

15
Data Structure

• LL

Post order sorted doubly linked list
0
a
1
b
c
d
e
g
f
2
h
i
k
3
Number of Pointers 3 depth Find first leaf
O (N)
16
Data Structure

• Tree

(left child, right sibling)
a
b
c
d
e
g
f
h
i
k
17
Data Structure

• LL Tree

a
b
c
d
e
g
f
h
i
k
Number of Pointers 5 Find first leaf O (D)
18
Data Structure

• LL O1

a
b
c
d
e
g
f
h
i
k
Number of Pointers 6 Find first leaf O (1)
best O (D) worst
19
Data Structure

• LL O1

a
b
c
d
e
g
f
h
i
k
Number of Pointers 6 Find first leaf O (1)
best O (D) worst
20
Data Structure
Representation Number of Pointers Find first leaf
LL 3 depth O (N)
LL Tree 5 O (D)
LL O1 6 O (1) best O (D)
21
Synchonization

• Datastructure consistency
• Atomic modifiaction
• No Priority Inversion
• gt Scheduler Conscious Locks
• Scalability
• Maximal possible parallelism
• gt Frame granular locking

22
Synchronization
acquire Lock
release Lock
unpreemptable
Roll forward to Preemption point
23
Synchronization

• Frame granular locking (the real world)

4MB
a
1MB
b
c
d
e
f
g
h
i
j
k
l

24
Restart Point Tracking

• Why Restart Point Tracking ?
• Required Properties
• Low overhead on map / unmap
• Guarentee fwd progress to avoid starvation

a
(rw)
t1
b
c
(rw)
(rw)
d
(r)
25
Restart Point Tracking

• Why Restart Point Tracking ?
• Required Properties
• Low overhead on map / unmap
• Guarentee fwd progress to avoid starvation

a
(rw)
t1
b
c
(rw)
(rw)
d
(r)
26
Restart Point Tracking

• Update restart point
• Find restart point information
• Update restart point information
• On Wakeup
• Find node to proceed
• ! Guarantee progress of find !

27
Restart Point Tracking

• Token Based Preempted Thread List

4
2
3
28
Restart Point Tracking

• Token Based Preempted Thread List

1
2
4
2
3
29
Restart Point Tracking

• Token Based Preempted Thread List

1
2
4
2
3
30
Restart Point Tracking

• Rootnode overrun detection

1
2
4
2
3
check overrun flag in mdb node
31
Summary Mapping Database

• Post order traversal
• Data Structures
• LL
• LL-Tree
• LL-O1
• Synchronization
• Scheduler Conscious Locks
• Roll forward to preemption point
• Token based preempted thread list
• Guarantee progress of preempted operations
• Root overrun detection

size
search time for first leaf
32
Performance Evaluation
Repre-sentation pointer Search time leaf node Mapping node size
LL 3 depth O(N) 32 byte (24 byte)
LL-Tree 5 O(D) 32 byte
LL-O1 6 O(1) best O(D) worst 40 byte (32 byte)

• Comparable performance
• for map systemcall
• (prepare IPC, map page,
• resume faulting instruction)

on PIII 500 MHz, 256 KB L2 Cache
33
Performance Evaluation

1
2
n

• Unmap wide mapping tree
• Performance of unmap systemcall

34
Overhead to Interrupt Latencies

• Map
• insert mapping
• update PTE
• Unmap
• 3 Preemption Points
• 1 stepsearch start point
• remove mappingupdate PTE
• finish unmap

• Mapping Database responsible for 13ms interrupt
  latency
• (largest distance between preemption points)

35
Conclusion

• Search time optimization for leaf node does not
  pay
• Good preemptability without helping
• No unbounded priority inversion
• No need for prevention !!!
• approx. 12.2 - 26 loss of performance for unmap
• Open
• explaination of Fiasco performance
• bounded tree-size N for real-time unmap

36
End of slide show, click to exit
37
Page Reference Information / Overmapping
rw

• Increase rights omitting prior flush
• same source address space
• same virtual source address
• same virtual destination address
• same physical page

r
rw
r
r
r
38
Page Reference Information /Overmapping
0
pa rw
a
pb rw
pc w
c
b
ltWgt
pd w
d
ltRgt
39
THE REAL END
40
Motivation

• Problems in Existing Implementations
• Long Interrupt Latencies
• Orangepip, Hazelnut
• Unbounded Priority Inversion
• Calypso, Hazelnut (two phase lock)
• Complex Helping or Timeslice Donation
• Fiasco, L4 Alpha

41
Guaranteeing Forward Progress

• Why Restart Point Tracking ?

a
b
c
d
e
g
f
h
i
k
42
Guaranteeing Forward Progress

• Why Restart Point Tracking ?

a
(rw)
t1
b
c
(rw)
(rw)
d
(r)
43
Guaranteeing Forward Progress

• Restart Point Tracking

44
Guaranteeing Forward Progress

• Restart Point Tracking

45
Guaranteeing Forward Progress

• Restart Point Tracking

46
Guaranteeing Forward Progress

• Restart Point Tracking
• Token Based Preempted Thread List

4
2
3
47
Experimental Results

• Performance
• Map 4KB to handle Pagefault
• Unmap Deep Wide subtree
• Interrupt Latencies
• Time OPs are rolled forward

Intel Pentium III 500 MHz , 512 KB L2, 2x 16 KB
L1, 64 / 32 entry 4KB TLB (data / code), 8 / 2
entry 4MB TLB
48
Experimental Results
49
Experimental Results
50
Experimental Results
51
Experimental Results

• Worst case interrupt latency

52
Experimental Results
53
Interrim Measurements
54
Interrim Measurements