Distributed Programming in Mozart - 2

1
Distributed Programming in Mozart - 2

• Per Brand

2
Mozart

• This lectureHow to deal with fundamental aspects
  of distributed programming in the Mozart
  programming system
• Distribution control, fault-tolerance, resource
  control, security etc.
• Most of the basic primitives (except fault
  module) have already been presented
• Show how a number of useful abstractions can be
  built on top
• Later
• Discuss how well (or not) Mozart meets our
  criteria for a distributed programming platform
• Transparency/awareness/control
• Comparison with other systems for distributed
  programming

3
Abstractions for Distribution Control

• Is it enough to have one distribution strategy
  per type of entity
• Object-state is always mobile
• Records are always replicated eagerly
• Object-records are always replicated lazily
• Answer is almost but not quite
• More can be achieved than is apparent at first
  sight
• Abstractions can be build for control that is
  lacking
• Examples
• Stationary objects and other entities
• Lazy records
• 1st part of this lecture

4
How to deal with failure, resources, security

• Failure
• The Fault module introduced
• Fault detection
• Enables the construction of a number of
  fault-tolerant abstractions
• Resources
• Dynamic scoping
• Static scoping
• Resource control by use of virtual sites
• Security
• Language security
• Security in connection with resources
• Implementation security

5
Making stateful entities stationary

• It is possible to create abstraction that make
  stateful entities that are either mobile by
  default (e.g. objects) or resources by default
  (e.g. dictionaries) stationary with network
  transparency
• Begin by considering objects
• Remember we might want objects to be stationary
  for a number of reasons
• If the pattern of use is that any one site rarely
  makes repeated invocations before another site
  requests the object then there is no advantage
  with having the object mobile
• Easier to deal with failure and security

6
Stationary Objects

• Stationary object (server) A stationary object
  remains on the site at which it was created.
• Each method invocation uses one message to start
  the method and one message to synchronize with
  the caller when the method is finished.
• Exceptions are raised in the caller's thread.
• Each method executes in a new thread created for
  it on the object's site. This is reasonable since
  threads in Mozart are extremely lightweight
  (millions can be created on one machine).

7
Stationary Objects
T1 O m(X) send m(X) to O
T2 O m(X)
XValue
T1
time
8
Stationary Objects

• declare ObjectNewStat Class Init
• When executed on a site, the procedure NewStat
  takes a class and an initial message and creates
  an object that is stationary on that site. We
  define NewStat as follows.
• declare fun NewStat Class Init MakeStat
  New Class Initend
• The procedure MakeStat takes an object or a
  one-argument procedure and returns a one-argument
  procedure that obeys exactly the same language
  semantics and is stationary.
• For procedures it implements Remote Procedure
  Call (RPC)

9
ltMakeStat definitiongt

• fun MakeStat PO S StatP PNewPort S
  NNewName in proc StatP M R in
  returned procedure Send P MR if
  RN then skip else raise R end end end
  thread ForAll S proc MR
  thread try PO M
  RN catch X then RX end
  end end end StatPend

10
Client Side

• fun MakeStat PO S StatP PNewPort S
  NNewName in proc StatP M R in
  returned procedure Send P MR if
  RN then skip else raise R end end end

- The client uses StatP exactly as it would have
used PO (same interface) - Using StatP the
client is actually sending a message. For
example instead of Obj m(ask A) the client
does Send P m(ask A)R - The client waits for R
to be bound - object invocation is synchronous -
Normally R is bound to the unique name N and the
client thread can continue executing. - However
R may be bound to something else in which case it
is the exception that is to be raised
11
Server Side

• thread ForAll S proc MR
  thread try PO M
  RN catch X then RX end
  end end end

- At the server side a thread is continually
responsible for listening to the stream. - A new
thread executes the method in a try-catch. -
Normally R is bound to N (the common name in both
server/client side abstractions - If an
exception is raised R is bound to that exception
12
Variation 1- server side

• thread ForAll S proc MR
  thread try PO M
  RN catch X then RX end
  end end end

- Make the server an active object by having only
one serving thread - Only one thread inside
object at a time
13
Sequential Asynchronous Stationary Objects

• Each method invocation uses one message only and
  does not wait until the method is finished.
• All method invocations execute in the same
  thread, so the object is executed in a completely
  sequential way.
• Non-caught exceptions in a method are ignored by
  the caller.

14
Variation 2 - Client Side

• fun MakeStat PO S StatP PNewPort S
  NNewName in proc StatP M R in
  Send P MR if RN then skip else
  raise R end end end

- The client invokes object asynchronously -
Exceptions ignored
15
Variation 3 - Client Side

• fun MakeStat PO S StatP PNewPort S
  NNewName in proc StatP M R T in
  Send P MR TThread.this
  thread if RN then skip else
  Thread.injectException T R end
  end end

- The client invokes object asynchronously -
Exceptions caught and raised in thread (no effect
if terminated) - Useful to terminate thread on
repeated invocations if one invocation goes wrong
16
Wrappers

• Stationary objects were implemented by wrapper
  technique
• Given a procedure/object/module/class with a give
  interface - wrap the entity and provide the same
  interface
• Can also be used for non-distributable entities
  that are treated as if they were resources - to
  make them stationary
• Arrays, dictionaries, threads, etc.
• Note that all references to the entity might not
  be wrapped
• E.g. give some sites reference to the object and
  gives others to the stationary object
• Only privileged users can interfere with quick
  home access to the object by moving it.

17
Does this qualify as network transparent

• Short answer Not quite
• Non-transparent aspects
• Type tests (inspection)
• An object is of type object while a stationary
  object is of type procedure
• Equality tests (if both kinds of references
  exist)
• Subtleties
• Re-entrant locking
• Feature access
• Also wrapping has a significant cost for local
  execution
• Conclusion by Mozart consortium
• more distinction of type mobile/stationary need
  to be made primitive

18
Compute Servers

• One of the promises of distributed computing is
  making computations go faster by exploiting the
  parallelism inherent in networks of computers.
• A first step is to create a compute server, that
  is, a server that accepts any computation and
  uses its computational resources to do the
  computation.
• declare class ComputeServer meth init skip
  end meth run(P) P end end CNewStat
  ComputeServer init
• The compute server can be made available through
  a URL as shown before.

19
Clients

• declare fun Fib N if Nlt2 then 1 else Fib
  N-1Fib N-2 end end
• Do first computation remotelylocal F in C
  run(proc FFib 30 end) Browse Fend
• Do second computation locallylocal F in
  FFib 30 Browse Fend

20
Dealing with Resources

• declare fun Fib0 N if Nlt2 then 1 else
  Fib N-1Fib N-2 end endfun Fib N
  Browser.browse fib(N Fib0 N) end local F
  in C run(proc FFib 30 end) end

Exception is raised application of non-procedure
in statement Resource Browser.browse fib(N
Fib0 N) is actually Resource fib(N Fib0 N)
21
Resources or sited entities

• Entities that can be used only on one site are
  called sited. We call this site their owner site
  or home site.
• References to these entities can be passed to
  other sites, but they do not work there. They
  work only on their owner site.
• Entities that can be used on any site are called
  unsited.
• In Mozart, all sited entities are modules
  (components created from functors).
• Not all modules are sited, though.
• The base modules are unsited
• System modules, that are part of the system but
  loaded only when needed. they are sited.

22
Resources

• Browser.browse is an abstraction that makes us of
  the Tk module which in turn uses the window
  system
• This actual window commands reference a system
  resource
• For convenience the whole Browser is made sited
• More coarse-grained
• Less waster of computation
• Application programmer can also make procedures,
  classes sited
• see documentation - not often needed
• Three possibilities in dealing with resources
• References should not leave site - resource
  exceptions the right way to go
• Static scoping (the original resource)
• Dynamic scoping (the corresponding resource on
  remote site)

23
Static scoping

• Can be realized by techniques shown so far
• Instead of sited procedure P construct wrapper
  that invokes corresponding method in stationary
  object

class B meth browse(X) Browse X end
ObjNew B init MyBrowserexport(browseproc
X O browse(X) end)
24
Dynamic scoping-1

• Requires receiving site to plug in his resources
• you dont know them
• Method 1 - clumsy
• explicitly parameterize over the needed resources
• Example
• instead of procP Tk end
• parameterize over Tk procP Tk Tk end
• Method 2- functors
• making use of naming convention of Mozart systems

25
An example with agents

• Assume that we have N sites
• Each site has an external port (Oz-port) XPI with
  the associated stream XSI
• Each site knows about a subset of other sites
  -neighbors
• takes the form of a list of external ports
• e.g. XP2 XP8 XP11
• Sites may have been connected by tickets or some
  registry service.
• Program in full on web pages
• Each site should let agents come into the site,
  get information about the neighbors and other
  agents currently living on the site.
• Agents are give access to the window system(Tk)

26
A little mobile agent using functors

• funCreateAgentFunctor State functor export
  Start import Tk imports Window System
  define class AgentClass end
  procStart Center Obj in State is in
  closure ObjNew AgentClass init(State)
  Send Center entering(Id Obj) Obj start
  Send Center leaving(Id) end end

27
Server-side

• procStartCenter XS NeighborPorts IS
  DictNewDictionary IPortNewPort IS
  ModManNew Module.manager init
  procExternalServe Msg case Msg of agent(A)
  then I in thread ModMan apply(A
  I) I.start Iport end end
  procExternalServe Msg case Msg of
  entering(Id A) then Dict.A P
  leaving(Id) Dictionary.remove Dict Id
  getAgents(A) Dictionary.items Dict A
  getNeighbors(N) NNeighbors end
  end

28
Server-side

• thread for X in XS do
  ExternalServe S end for X in IS do
  InternalServe S end end end

29
Starting the agent

• funAgent Id State funCreateAgentFunctor
  State ... endend
• an agent can be started from a site if it
  knows at least one center
• declareFAgent MyId OriginalStateSend
  KnownPort agent(F)

30
Agent Functionality

• class AgentClass feat center attr center
  meth init(Center) statelt-State
  self.centerCenter end meth start As
  Ns Next in Send self.center
  getAgents(As) communicate with other
  agents do graphics Send
  self.center getNeighbors(Ns) self
  chooseNextDestination(Ns Next)
  FCreateAgentFunctor _at_state Send Next
  agent(F) end meth proposal(X)
  routine used by other agents end end

31
Additional Varieties

• Agent reports back to home site after each move
• E.g. send a message on an encapsulated port
• Agent functor encapsulates the agent object -
  created once only
• have to take care that the methods do not use Tk
  directly, i.e. parameterized over Tk
• note that with the mobile object-state protocol
  that after each move the agent will fetch the
  object state from the last site visited but this
  is only done once
• We still have to deal with failure

32
Failure Model

• Distributed systems have the partial failure
  property, that is, part of the system can fail
  while the rest continues to work.
• Partial failures are not at all rare.
• Properly-designed applications must take them
  into account.
• This is both good and bad for application design.
  
• The bad part is that it makes applications more
  complex.
• The good part is that applications can take
  advantage of the redundancy offered by
  distributed systems to become more robust.
• Mozart recognizes permanent site failures that
  are instantaneous (fail-stop) and both temporary
  and permanent communication failures.
• In local area networks these are possible to
  detect.
• In global networks only temporary failure makes
  sense.
• Mozart provides only the primitive operations
  needed to detect failure and reflect it in the
  language.

33
Temporary faults

• The fault state tempFail exists to allow the
  application to react quickly to temporary network
  problems. It is raised by the system as soon as a
  network problem is recognized.
• It is fundamentally different from a time-out.
  For example, TCP gives a time-out after some
  minutes to give probable guarantees that the
  connection is not working.
• The purpose of tempFail is quite different from a
  time-out. It is to inform the application of
  network problems, not to mark the end of a
  connection.
• Example A client getting tempFail trying to
  connect to a server may use this info to connect
  to another server.
• The application has to make the decision not the
  network layer.

34
Remote faults

• An entity can also contain information about the
  fault states on other sites.
• Incomplete information !!!
• Example the current site is waiting for a
  variable binding, but the remote site that will
  do the binding has crashed. The current site can
  find this out.
• At least one of the other sites referencing the
  entity can no longer perform operations on the
  entity
• remoteProblem(permSome)
• All of the other sites referencing the entity can
  no longer perform operations on the entity
• remoteProblem(permAll)
• remoteProblem(tempSome)
• remoteProblem(tempAll)

35
Remote faults

• Even if there exists a remote problem, it is not
  always possible to return a remoteProblem fault
  state.
• This happens if there are problems with a proxy
  that the owner site does not know about.
• This also happens if the owner site is
  inaccessible. In that case it might not be
  possible to learn anything about the remote sites
  (since owner site mediates information to its
  proxies).

36
Basic Model

• Exception based.
• The basic model allows to enable or disable
  synchronous exceptions on language entities.
• Attempting to perform operations on entities
  with faults will either block or raise an
  exception without doing the operation.
• The fault detection can be enabled separately on
  each of two levels a per-site level or a
  per-thread level.
• Exceptions can be enabled on logic variables,
  ports, objects, cells, and locks.
• All other entities, will never raise an exception
  since they have no fault states.

37
Basic Model

• Enables/disable exceptions on entity faults
• Fault.enable Entity Level Fstates ?Bool
• Entity (any Oz-entity)
• Level site, thread(T) where T is a
  first-class thread reference or the atom this
• Fstates a list that may contain one or more of
  permFail, tempFail, remoteProblem(permSome),rem
  oteProblem(tempSome), remoteProblem(permAll),
  remoteProblem(tempAll)
• Bool returns true if entity could be enabled -
  false otherwise
• you cannot enable twice on the same entity and
  level
• Fault.disable Entity Level ?Bool
• Fault.defaultEnable Fstates ?Bool
• Currently Fault.defaultEnable permFail
  tempFail _
• May want to change to permFail only
• Fault.defaultDisable ?B

38
Advanced Model

• Allows to install or deinstall handlers and
  watchers on entities.
• These are procedures that are invoked when there
  is a failure.
• Handlers are invoked synchronously (when
  attempting to perform an operation)
• Watchers are invoked asynchronously (in their own
  thread as soon as the fault state is known).

39
Advanced Model

• Installing handlers
• Fault.install Entity Level Fstates
  HandlerProc?Bool
• HandlerProc is a procedure of 3 arguments
• If the fault occurs HandlerProc Entity
  ActualFaultStates Op is called
• Op is the attempted operation
• if HandlerProc defined as shown below equivalent
  to enabling
• procHandlerProc E As Op
• raise system(dp(entityE conditionsAs opOP))
  end
• end
• Installing watchers
• Fault.installWatcher Entity Level Fstates
  WatcherProc?Bool
• WatcherProc is a procedure of 2 arguments
• If the fault occurs WatcherProc Entity
  ActualFaultStates is called in its own thread

40
Fault Tolerant Examples

• Pattern 1 SafeSendAndWait
• Sending message to port with variable for return
  value.
• Raise exception on failure
• For variable we must enable failure on permSome
  tempSome
• We know that there is only one other site
  referencing the variable
• If there were many sites then permSome might not
  be important
• We are the manager so permFail/tempFail are not
  observable for us
• We enable on the port unnecessary - the port is
  enabled by default.

41
SafeSendAndWait

• procSafeSendAndWait P Msg Ret Fault.enable
  P thread(this) permFail tempFail _
  Fault.enable Ret thread(this) permFail
  tempFail remoteProblem(tempSome) remoteProble
  m(permSome) _ try Send P Msg
  Wait Ret catch _ raise safeSendAndWaitError
  end endend

42
Fault Tolerant Examples

• Pattern 2 SafeSendAndWaitTime
• Sending message to port with variable for return
  value.
• Raise exception on permFail and timeout on temp
• Assuming the timeout is long and temporary fault
  conditions are fast
• First variety uses an extra thread to track time
  at each send
• Second example only uses an extra thread when a
  temp fault condition detected (i.e. no extra
  thread created on normal operation).

43
SafeSendAndWaitTime

• procSafeSendAndWaitTime P Msg TimeOut Ret
  TThread.this CNewCell notOkin thread
  Delay TimeOut if Access CnotOk then
  Thread.injectException T error end
  end Fault.enable P thread(this) permFail
  _ Fault.enable Ret thread(this) permFail
  remoteProblem(permSome) _ try
  Send P Msg Wait Ret Update C ok catch
  _ raise error end endend

44
SafeSendAndWaitTime2

• procSafeSendAndWaitTime2 P Msg TimeOut Ret
  procHP E C Op SafeSendAndWait P Msg Ret
  end Fault.install P thread(this) permFail
  tempFail HP _ Fault.install Ret
  thread(this) permFail remoteProblem(permSome
  ) tempFail remoteProblem(tempSome) HP _
  Send P Msgend

45
Fault Tolerant Examples

• Pattern 3 Watcher
• Watchers give eager detection of failure
• Example use watcher to stop computations on a
  compute server if client goes down

• procServe Stream case Stream of msg(Request
  Answer)More thread Fault.installWatche
  r Answer tempFail permFail proc E C Op
  Thread.injectException Thread.this e
  end try big computation catch
  _ then skip end end Serve More
  end end

46
Security

• Language security
• Language contains the necessary support for
  enabling arbitrary security levels (no holes)
• Implementation security
• Mechanisms for guaranteeing security on untrusted
  sites (fake implementations)
• Resource security
• Security for OS-resources (files, etc.)
• Language entity security
• Security for language entities
• Mozart
• no implementation security
• language security
• resource security

47
Language Security for Language Entities-1

• Mechanisms for language security
• Oz-names are unforgeable but unique
• Example - Trusted can use the secret method but
  untrusted cannot
• local Secret in NNewName class X
  meth public end meth !Secret end
  endObjNew X initSend Trusted
  ObjSecretSend Untrusted Obj

48
Language Security for Language Entities-2

• Mechanisms for language security
• Encapsulation (wrappers)
• Example - Only method inc is accessible through P
• procP X Obj inc(X)end class
  BankAccount meth inc(X) end meth
  dec(X) end end ObjNew X initSend
  Untrusted PSend Trusted Obj

49
Language and Implementation Security for Resources

• Specialized Module Managers to use with untrusted
  components (applets/ozlets) etc.
• For each system module service
• if class then use dummy class that raises
  exception on initialization
• if procedure then use dummy procedure that raises
  and exception
• e.g. Open.file is a class, OS.getDir is a
  procedure

• declare procDummyGetDir A B raise
  securityViolation(OS.getDir) end end
  class DummyFile meth init( )
  raise securityViolation(Open.file) end
  end end

50
Security for Resources

• declare ModManNew Module.manager init
  ModMan enter(nameOpen DummyOpen)
  procServe S case S of do(Functor)More
  thread ModMan apply(Functor ).start
  end Serve More end end

51
Implementation Security

• Desired Properties
• Non-forgeable references/names (non-guessable)
• Secure communication (no eavesdropping)
• Language security holds even in the presence of
  hostile sites
• Orthogonal control of security - independent of
  other aspects of distribution
• Mozart
• Currently no implementation security whatsover!!