Title: Chord: A Scalable Peertopeer Lookup Service for Internet Applications
1Chord A Scalable Peer-to-peer Lookup Service for
Internet Applications
- Robert Morris
- Ion Stoica, David Karger,
- M. Frans Kaashoek, Hari Balakrishnan
- MIT and Berkeley
2A peer-to-peer storage problem
- 1000 scattered music enthusiasts
- Willing to store and serve replicas
- How do you find the data?
3The lookup problem
N2
N1
N3
Internet
Keytitle ValueMP3 data
?
Client
Publisher
Lookup(title)
N6
N4
N5
4Centralized lookup (Napster)
N2
N1
SetLoc(title, N4)
N3
Client
DB
N4
Publisher_at_
Lookup(title)
Keytitle ValueMP3 data
N8
N9
N7
N6
Simple, but O(N) state and a single point of
failure
5Flooded queries (Gnutella)
N2
N1
Lookup(title)
N3
Client
N4
Publisher_at_
Keytitle ValueMP3 data
N6
N8
N7
N9
Robust, but worst case O(N) messages per lookup
6Routed queries (Freenet, Chord, etc.)
N2
N1
N3
Client
N4
Lookup(title)
Publisher
Keytitle ValueMP3 data
N6
N8
N7
N9
7Structured Overlay Networks / DHTs
Main representativesChord, Pastry, Tapestry,
CAN, Kademlia, P-Grid, Viceroy
Set of Nodes
Keys of Nodes
Common Identifier Space
Hashing
ConnectThe nodes Smartly
Set of Values/Items
Keys of Values
Keys of Values
Hashing
Node IdentifierValue Identifier
8The Principle Of Distributed Hash Tables
- A dynamic distribution of a hash table onto a set
of cooperating nodes
- Basic service lookup operation
- Key resolution from any node
9Routing challenges
- Define a useful key nearness metric
- Keep the hop count small
- Keep the tables small
- Stay robust despite rapid change
- Freenet emphasizes anonymity
- Chord emphasizes efficiency and simplicity
10Chord properties
- Efficient O(log(N)) messages per lookup
- N is the total number of servers
- Scalable O(log(N)) state per node
- Robust survives massive failures
- Proofs are in paper / tech report
- Assuming no malicious participants
11Chord overview
- Provides peer-to-peer hash lookup
- Lookup(key) ? IP address
- Chord does not store the data
- How does Chord route lookups?
- How does Chord maintain routing tables?
12Chord IDs
- Key identifier SHA-1(key)
- Node identifier SHA-1(IP address)
- Both are uniformly distributed
- Both exist in the same ID space
- How to map key IDs to node IDs?
13Consistent hashing Karger 97
Key 5
K5
Node 105
N105
K20
Circular 7-bit ID space
N32
N90
K80
A key is stored at its successor node with next
higher ID
14Basic lookup
N120
N10
Where is key 80?
N105
N32
N90 has K80
N90
K80
N60
15Simple lookup algorithm
- Lookup(my-id, key-id)
- n my successor
- if my-id lt n lt key-id
- call Lookup(id) on node n // next hop
- else
- return my successor // done
- Correctness depends only on successors
16Finger table allows log(N)-time lookups
½
¼
1/8
1/16
1/32
1/64
1/128
N80
17Finger i points to successor of n2i
N120
112
½
¼
1/8
1/16
1/32
1/64
1/128
N80
18Lookup with fingers
- Lookup(my-id, key-id)
- look in local finger table for
- highest node n s.t. my-id lt n lt key-id
- if n exists
- call Lookup(id) on node n // next hop
- else
- return my successor // done
19Lookups take O(log(N)) hops
N5
N10
N110
K19
N20
N99
N32
Lookup(K19)
N80
N60
20Another example
- Identifier circle
- Keys assigned to successor
- Evenly distributed keys and nodes
- Finger table logN
- ith finger points to first node that succeeds n
by at least 2i-1
1
54
8
58
10
14
47
21
- Find
- Map key to node
- Join, Leave, or Failure
- Update the immediate neighbors
- Successor and predecessor
- Stabilize eventually propagate the info
- Reliability
- Log(N) successors data replication
42
38
32
38
24
30
21Chord Key Location
- Lookup in finger table the furthest node that
precedes key - Query homes in on target in O(logN) hops
22Problem
- Nodes close on ring can be far in the network.
Figure from http//project-iris.net/talks/dht-to
ronto-03.ppt
23Chord node insertion
Insert node N40
Locate node
Add fingers
Update successor pointers and other nodes
fingers (max in-degree O(log2n) whp)
Time O(log2n) Stabilization protocol for
refreshing links
N40
24Yet another example
0
15
1
14
2
13
3
12
4
11
5
10
6
9
7
8
Nodes
Values
25Chord Routing (1/4)
Get(15)
0
15
1
2
14
13
3
12
4
11
5
10
6
9
- Routing table size M, where N 2M
- Routing entries log2(N)
- log2(N) hops from any node to any other node
7
8
26Chord Routing (2/4)
Get(15)
0
15
1
2
14
13
3
12
4
11
5
10
6
9
7
8
27Chord Routing (3/4)
Get(15)
0
15
1
2
14
13
3
12
4
11
5
10
6
9
7
8
28Chord Routing (4/4)
Get(15)
0
15
1
- From node 1, only 3 hops to node 0 where item 15
is stored - For 16 nodes, the maximum is log2(16) 4 hops
between any two nodes
2
14
13
3
12
4
11
5
10
6
9
7
8
29Joining linked list insert
N25
N36
1. Lookup(36)
K30 K38
N40
30Join (2)
N25
2. N36 sets its own successor pointer
N36
K30 K38
N40
31Join (3)
N25
3. Copy keys 26..36 from N40 to N36
N36
K30
K30 K38
N40
32Join (4)
N25
4. Set N25s successor pointer
N36
K30
K30 K38
N40
Update finger pointers in the background Correct
successors produce correct lookups
33Failures might cause incorrect lookup
N120
N10
N113
N102
Lookup(90)
N85
N80
N80 doesnt know correct successor, so incorrect
lookup
34Solution successor lists
- Each node knows r immediate successors
- After failure, will know first live successor
- Correct successors guarantee correct lookups
- Guarantee is with some probability
35Choosing the successor list length
- Assume 1/2 of nodes fail
- P(successor list all dead) (1/2)r
- I.e. P(this node breaks the Chord ring)
- Depends on independent failure
- P(no broken nodes) (1 (1/2)r)N
- r 2log(N) makes prob. 1 1/N
36Lookup with fault tolerance
- Lookup(my-id, key-id)
- look in local finger table and successor-list
- for highest node n s.t. my-id lt n lt key-id
- if n exists
- call Lookup(id) on node n // next hop
- if call failed,
- remove n from finger table
- return Lookup(my-id, key-id)
- else return my successor // done
37Chord status
- Working implementation as part of CFS
- Chord library 3,000 lines of C
- Deployed in small Internet testbed
- Includes
- Correct concurrent join/fail
- Proximity-based routing for low delay
- Load control for heterogeneous nodes
- Resistance to spoofed node IDs
38Experimental overview
- Quick lookup in large systems
- Low variation in lookup costs
- Robust despite massive failure
- See paper for more results
- Experiments confirm theoretical results
39Chord lookup cost is O(log N)
Average Messages per Lookup
Number of Nodes
Constant is 1/2
40Failure experimental setup
- Start 1,000 CFS/Chord servers
- Successor list has 20 entries
- Wait until they stabilize
- Insert 1,000 key/value pairs
- Five replicas of each
- Stop X of the servers
- Immediately perform 1,000 lookups
41Massive failures have little impact
(1/2)6 is 1.6
Failed Lookups (Percent)
Failed Nodes (Percent)
42Related Work
- CAN (Ratnasamy, Francis, Handley, Karp, Shenker)
- Pastry (Rowstron, Druschel)
- Tapestry (Zhao, Kubiatowicz, Joseph)
- Chord emphasizes simplicity
43Chord Summary
- Chord provides peer-to-peer hash lookup
- Efficient O(log(n)) messages per lookup
- Robust as nodes fail and join
- Good primitive for peer-to-peer systems
- http//www.pdos.lcs.mit.edu/chord
44(No Transcript)
45Join lazy finger update is OK
N2
N25
K30
N36
N40
N2 finger should now point to N36, not
N40 Lookup(K30) visits only nodes lt 30, will
undershoot
46CFS a peer-to-peer storage system
- Inspired by Napster, Gnutella, Freenet
- Separates publishing from serving
- Uses spare disk space, net capacity
- Avoids centralized mechanisms
- Delete this slide?
- Mention distributed hash lookup
47CFS architecturemove later?
Block storage Availability / replication Authentic
ation Caching Consistency Server
selection Keyword search Lookup
Dhash distributed block store
Chord
- Powerful lookup simplifies other mechanisms