Implementation: Charm

1
ImplementationCharm
Orion Sky Lawlor olawlor_at_acm.org
2
Background Roadmap

• Your code sits on Charm
• Interface Translator
• .ci file, CProxy_, parameter marshalling
• Arrays (Orion)
• Reductions, broadcasts, migrations, pup
• Groups and Chares (Milind)
• Basic communication calls atop Converse
• Basics ()
• Registration, readonlies, messages

3
Where is it?

• Charm core source code in charm/src/Common/ck-co
  re/
• Gets soft-linked over and made in
  charm/net-linux/tmp/
• Headers in charm/net-linux/include/
• Code gets linked into the library
  charm/net-linux/lib/libck.a
• This all gets used when you pass "-language
  charm" to charmc

4
Basics

• We need to be able to call some C method bar
  on user object foo
• But the cores never heard of foo
• Solution register a function pointer to a
  "call-method" function

void _call_foo_bar(void msg,foo obj)
obj-gtbar(msg) ...at startup... int
barEpIdx CkRegisterEp(_call_foo_bar,..)
5
Registration Startup

• ck-core/register.h .C (300 lines)
• At startup, each module registers its methods,
  which end up in a big entry method table
• An entry point's index in this table is its int
  "epIdx", used everywhere
• Translator writes this registration code (the
  call-method function, "__register" function) in
  .def file

6
Groups Chares (Milind)

• charm.h, ck.C (1000 lines)
• Lowest layer of Charm
• CkCreateChare, CkCreateGroup
• CkSendMsg, CkSendMsgBranch
• Quite simple, thin wrapper on Converse
• Very stable-rarely changes
• Callable from C (!)

7
CkSendMsg(int epIdx, void msg,CkChareID to)

• envelope env UsrToEnv(msg)
• Finds Charm msg fields (see envelope.h)
• env-gtsetMsgtype(ForChareMsg)
• env-gtsetObjPtr(to-gtobjPtr)
• env-gtsetEpIdx(epIdx)
• env-gtsetSrcPe(CkMyPe())
• CmiSetHandler(env, _charmHandlerIdx)
• CldEnqueue(to-gtonPE, env, _infoIdx)

8
CkSendMsg Clutter

• Various unrelated calls clutter up the Charm
  core (FIXME?)
• Keep track of message sends give "Message being
  re-sent..." diagnostic
• _CHECK_USED(env)
• _SET_USED(env, 1)
• Trace message sends for projections
• _TRACE_CREATION_1(env)
• Notify quiescence detection of send
• CpvAccess(_qd)-gtcreate()

9
Arrays (Orion)

• Many interesting features
• Arbitrary indexed collection of Chares
• Connects to load balancer
• Supports migration, broadcasts, reductions
• Not even clear some operations even possible
• But much more complicated
• Highly fragmented, er, factored code
• Array Manager, Location Manager, Reduction
  Manager, Broadcaster, Reducer

10
Arrays Diagram
CkReductionMgr-- reductions
CkLocMgr-- migration
CkArray-- coordinator
ArrayElement
ArrayElement
ArrayElement
ArrayElement-- user's work
11
CkReductionMgr

• ckreduction.h .C .ci (1300 lines)
• Accept contributions from local members of a
  migrating set of reduction "contributors"
• Implemented as Group
• Actually "IrrGroup", since normal groups inherit
  from CkReductionMgr.
• Exactly one per array but zero dependence on
  arrays
• Arrays are just a client of CkReductionMgr

12
Location-- CkLocMgr

• ck-core/cklocation.h .C .ci (2000)
• Keep track of location of migrating elements
• Handles messaging and migration
• Each array has a location manager
• But a location manager may have several "bound"
  arrays, which will migrate together
• Implemented as a group

13
Array Manager-- CkArray

• ck-core/ckarray.h .C .ci (1300 lines)
• Keeps track of a set of local array elements
• Implemented as a group
• Glue that connects pieces of array system
  together
• Inherits from CkReductionMgr
• Implements CkArrMgr (for location mgr)
• Interfaces with proxies
• Controls ArrayReducer, ArrayBroadcaster

14
Pup

• util/pup.h .C (1000 lines)
• Used to migrate, parameter marshall
• Heavy C overload the operator
• "Easy to use, tough to understand"
• Bottom line (user view)
• Say "px" for all your x's.
• Bottom line (system view)
• We get a "void " for each of the users x.

15
Translator

• xlat-i/xi-symbol.h .C (3000 lines)
• Turn .ci files into .decl/.def files
• Standalone program "charmxi"
• Tokenize .ci with lexx xi-scan.l
• Parse .ci with yacc xi-grammar.y
• Turn parsed structures into .decl/.def
• Almost completely routines named "gen"--
  code-generating routines
• E.g., "EntrygenDecls"-- generate code in .decl
  file for an entry method

16
Not Covered

• Startup Shutdown (init.C)
• Quiescence detection (qd.C)
• Dynamic load balancer (ck-ldb/)
• Callbacks, futures (for sync methods), debug,
  TeMPO
• Libraries (charm/src/Common/lib/)
• TCharm, FEM, AMPI, MBlock, AMR, comm, ...
• Details! Read manuals, headers, our minds...

17
Conclusion

• Interface Translator
• .ci file, CProxy_, parameter marshalling
• Arrays (Orion)
• Reductions, broadcasts, migrations, pup
• Groups and Chares (Milind)
• Basic communication calls atop Converse
• Basics ()
• Registration, readonlies, messages