The Design and Implementation of Declarative Networks

1
The Design and Implementation of Declarative
Networks

• Boon Thau Loo
• University of Pennsylvania, University of
  California-Berkeley

This dissertation was completed as a graduate
student at the University of California- Berkeley
2
Declarative Networking

• A declarative framework for networks
• Network protocols are declaratively specified
  using a database query language
• Distributed query engine executes specifications
  to implement network protocols
• Success of database research
• 70s today Database research has
  revolutionized data management
• Today Similar opportunity to revolutionize the
  Internet architecture

3
Motivation

• Internet faces many challenges today
• Unwanted, harmful traffic
• Complexity/fragility in Internet routing
• Proliferation of new applications and services
• Efforts at improving the Internet
• Evolutionary App-level Overlay networks
• Revolutionary Clean-slate designs
• NSF GENI initiative, FIND program

Opportunity Software tools that can
significantly accelerate network innovation
4
A Declarative Network
messages
Dataflow
Dataflow
messages
Dataflow
messages
Dataflow
Dataflow
Distributed recursive query
Dataflow
Traditional Networks
Declarative Networks
5
The Case for Declarative Networking

• Ease of programming
• Compact and high-level representation of
  protocols
• Orders of magnitude reduction in code size
• Declarative Chord DHT is 48 lines instead of
  10,000.
• Easy customization
• Safety
• Queries are sandboxed within query processor
• Potential for static analysis of safety
• What about efficiency?
• No fundamental overhead when executing standard
  routing protocols
• Application of well-studied query optimizations
• Note Same question was asked of relational
  databases in the 70s.

6
Main Contributions of Dissertation

• Declarative Routing SIGCOMM 05
• Extensible Routers balance of flexibility,
  efficiency and safety
• Declarative Overlays SOSP 05
• Rapid prototyping of new overlay networks
• Database Fundamentals SIGMOD 06
• Network specific query language and semantics
• Distributed recursive query execution strategies
• Query optimizations, classical and new

7
A Breadth of Use Cases

• Example implementations to date
• Textbook routing protocols
• Chord DHT
• Narada mesh for end-system multicast
• Distributed Gnutella/Web crawlers
• Pastry DHT
• Replication protocols
• Lamport/Chandy snapshots
• Paxos distributed consensus
• Overlays for host mobility
• Sensor network protocols
• P2 declarative networking system
• http//p2.cs.berkeley.edu

8
Outline

• Background
• The Connection Routing as a Query
• Execution Model
• Path-Vector Protocol Example
• Query specification ? protocol implementation
• Query Processing
• Beyond routing Declarative Overlays
• Ongoing work _at_ Penn

9
Traditional Router
Routing Protocol
Control Plane
Forwarding Plane
Traditional Router
10
Declarative Router
Query Engine
Routing Protocol
Control Plane
Forwarding Plane
Declarative Router
Traditional Router
11
All-Pairs Reachability
R1 reachable(S,D) ? link(S,D)
R2 reachable(S,D) ? link(S,Z), reachable(Z,D)
For all nodes S,D, If there is a
link from S to D, then S can reach D.
link(a,b) there is a link from node a to node
b
reachable(a,b) node a can reach node b

• Input link(source, destination)
• Output reachable(source, destination)

12
All-Pairs Reachability
R1 reachable(S,D) ? link(S,D)
R2 reachable(S,D) ? link(S,Z), reachable(Z,D)
For all nodes S,D and Z, If there is
a link from S to Z, AND Z can reach D, then S
can reach D.

• Input link(source, destination)
• Output reachable(source, destination)

13
Towards Network Datalog

• Specify tuple placement
• Value-based partitioning of tables
• Tuples to be combined are co-located
• Rule rewrite ensures body is always single-site
• All communication is among neighbors
• No multihop routing during basic rule execution
• Link-restricted rules Enforced via simple
  syntactic restrictions

14
Network Datalog
R1 reachable(_at_S,D) ? link(_at_S,D) R2
reachable(_at_S,D) ? link(_at_S,Z), reachable(_at_Z,D)
Query reachable(_at_a,N)
Query reachable(_at_M,N)
link
link
link
link
_at_S D
_at_c b
_at_c d
_at_S D
_at_b c
_at_b a
_at_S D
_at_a b
_at_S D
_at_d c
Input table
b
d
c
a
reachable
reachable
reachable
reachable
Output table
_at_S D
_at_a b
_at_a c
_at_a d
_at_S D
_at_b a
_at_b c
_at_b d
_at_S D
_at_c a
_at_c b
_at_c d
_at_S D
_at_d a
_at_d b
_at_d c
Query reachable(_at_a,N)
15
Path Vector in Network Datalog
R1 path(_at_S,D,P) ? link(_at_S,D), P(S,D).
path(_at_Z,D,P2),
?
link(_at_Z,S),
path(_at_S,D,P)
PS?P2.
R2
Query path(_at_S,D,P)
Add S to front of P2

• Input link(_at_source, destination)
• Query output path(_at_source, destination,
  pathVector)

16
Query Execution
R1 path(_at_S,D,P) ? link(_at_S,D), P(S,D). R2
path(_at_S,D,P) ? link(_at_Z,S), path(_at_Z,D,P2), PS?P2.
Query path(_at_a,d,P)
link
link
link
link
Neighbor table
_at_S D
_at_b c
_at_b a
_at_S D
_at_c b
_at_c d
_at_S D
_at_a b
_at_S D
_at_d c
b
d
c
a
_at_S D P
_at_S D P
_at_c d c,d
_at_S D P
_at_S D P
Forwarding table
17
Query Execution
R1 path(_at_S,D,P) ? link(_at_S,D), P(S,D). R2
path(_at_S,D,P) ? link(_at_Z,S), path(_at_Z,D,P2), PS?P2.
Query path(_at_a,d,P)
link
link
link
link
Neighbor table
_at_S D
_at_b c
_at_b a
Communication patterns are identical to those in
the actual path vector protocol
_at_S D
_at_c b
_at_c d
_at_S D
_at_a b
_at_S D
_at_d c
b
d
c
a
path(_at_a,d,a,b,c,d)
path(_at_b,d,b,c,d)
_at_S D P
_at_S D P
_at_S D P
_at_c d c,d
_at_S D P
_at_b d b,c,d
_at_S D P
_at_a d a,b,c,d
Forwarding table
18
Other Routing Examples

• Best-Path Routing
• Distance Vector
• Dynamic Source Routing (Wireless)
• Policy Decisions
• QoS-based Routing
• Link-state
• Multicast Overlays (Single-Source CBT)

19
Outline

• Background
• The Connection Routing as a Query
• Query Processing
• Beyond routing Declarative Overlays
• Sampling of ongoing work

20
Recursive Query Evaluation

• Semi-naïve evaluation
• Iterations (rounds) of synchronous computation
• Results from iteration ith used in (i1)th

9
7
5
2
10
4
1
8
0
3
6
Path Table
Link Table
Network
Problem Unpredictable delays and failures
21
Pipelined Semi-naïve (PSN)

• Fully-asynchronous evaluation
• Computed tuples in any iteration pipelined to
  next iteration
• Natural for network protocols

9
10
7
9
5
6
2
10
4
1
3
Relaxation of semi-naïve
8
0
8
5
3
2
7
6
4
1
Path Table
Link Table
Network
22
Pipelined Evaluation

• Challenges
• Does PSN produce the correct answer?
• Is PSN bandwidth efficient?
• I.e. does it make the minimum number of
  inferences?
• Proofs for
• Basic technique local timestamps

23
Execution Plan
Strands
Network Out
Network In
Messages
Messages
Single Node

• Nodes in execution plan (operators)
• Network operators (send/recv, cc, retry, rate
  limitation)
• Relational operators (selects, projects, joins,
  aggregates)
• Flow operators (mux, demux, queues)

24
Localization Rewrite

• Rules may have body predicates at different
  locations

R2 path(_at_S,D,P) ? link(_at_S,Z), path(_at_Z,D,P2),
PS?P2.
Rewritten rules
R2a linkD(S,_at_D) ? link(_at_S,D)
R2b path(_at_S,D,P) ? linkD(S,_at_Z), path(_at_Z,D,P2),
PS?P2.
25
Localized Rule Compilation
R2b path(_at_S,D,P) ? linkD(S,_at_Z), path(_at_Z,D,P2),
PS?P2.
Execution Plan
Network In
Network Out
26
Optimizations

• Traditional evaluate in the NW context
• Aggregate Selections
• Magic Sets rewrite
• Predicate Reordering
• New motivated by NW context
• Multi-query optimizations
• Query Results caching
• Opportunistic message sharing
• Cost-based optimizations
• Neighborhood density function
• Hybrid rewrites

27
Beyond Routing Declarative Overlays

• Language extensions to support events and
  soft-state predicates
• Chord Routing, including
• Multiple successors
• Stabilization
• Optimized finger maintenance
• Failure detection
• 48 rules
• 11 table definitions
• MIT-Chord x100 more code
• Another example
• Narada mesh in 22 rules

10 pt font
28
Outline

• Background
• The Connection Routing as a Query
• Query Processing
• Beyond routing Declarative Overlays
• Ongoing work _at_ Penn

29
Ongoing Work _at_ Penn

• Declarative secure networking
• Difficult to design/implement/reason about secure
  networks
• Network Datalog logic-based security languages
  NetDB 07
• Authenticated path vector protocol, DNSSEC,
  secure DHTs,
• Moving forward
• Exploit fine-grained control over networks and
  security policies
• Data-centric querying and routing in
  heterogeneous networks
• Internet Wired infrastructure with wireless
  clouds at the edges
• Flexible network support for mobility ACM
  MobiArch 07
• Declarative queries for addressing and naming
  mobile hosts
• Session-aware customizable QoS routing
• Moving forward
• Declarative wireless ad-hoc networks
• Cost-based query optimizations to adapt protocols

30
Summary

• Declarative networking
• Declarative Routing
• Extensible routing infrastructure
• Declarative Overlays
• Rapid prototyping overlay networks
• Database fundamentals
• Query language
• New distributed query execution strategies and
  optimizations
• Semantics in dynamic networks
• P2 declarative networking system
  (http//p2.cs.berkeley.edu)

31
Many Thanks

• Advisors Joseph M. Hellerstein, Ion Stoica
• Collaborators
• UC Berkeley Tyson Condie, Ryan Huebsch
• Intel Research David Gay, Petros Maniatis,
  Timothy Roscoe
• Yahoo! Research Minos Garofalakis, Raghu
  Ramakrishnan
• Rice University Atul Singh
• Many others