CS294-1 Deeply Embedded Networks TinyOS Sept 4, 2003

1
CS294-1 Deeply Embedded NetworksTinyOSSept 4,
2003

• David Culler
• Fall 2003
• University of California, Berkeley

2
Characteristics of Network Sensors

• Small physical size and low power consumption
• Concurrency-intensive operation
• multiple flows, not wait-command-respond
• Limited Physical Parallelism and Controller
  Hierarchy
• primitive direct-to-device interface
• Asynchronous and synchronous devices
• Diversity in Design and Usage
• application specific, not general purpose
• huge device variation
• gt efficient modularity
• gt migration across HW/SW boundary
• Robust Operation
• numerous, unattended, critical
• gt narrow interfaces

3
Classical RTOS approaches

• Responsiveness
• Provide some form of user-specified interrupt
  handler
• User threads in kernel, user-level interrupts
• Controlled Scheduling
• Static set of tasks with specified deadlines
  constraints
• Generate overall schedule
• Doesnt deal with unpredictable events (comm)
• Threads synchronization operations
• Sophisticated scheduler to coerce into meeting
  constraints
• Priorities, earliest deadline first, rate
  monotonic
• Priority inversion, load shedding, live lock,
  deadlock
• Sophisticated mutex and signal operations
• Communication among parallel entities
• Shared (global) variables ultimate unstructured
  programming
• Mail boxes (msg passing)
• dynamic allocation, storage reuse, parsing
• external communication considered harmful
• Fold in as RPC
• Requires multiple (sparse) stacks

4
Alternative Starting Points

• Event-driven models
• Easy to schedule handfuls of small, roughly
  uniform things
• State transitions (but what storage and comm
  model?)
• Usually results in brittle monolithic dispatch
  structures
• Structured event-driven models
• Logical chunks of computation and state that
  service events via execution of internal threads
• Threaded Abstract machine
• Developed as compilation target of inherently
  parallel languages
• vast dynamic parallelism
• Hide long-latency operations
• Simple two-level scheduling hierarchy
• Dynamic tree of code- block activations with
  internal inlets and threads
• Active Messages
• Both parties in communication know format of the
  message
• Fine-grain dispatch and consume without parsing
• Concurrent Data-structures
• Non-blocking, lock-free (Herlihy)

5
TinyOS design goals

• Simple framework for resource constrained
  concurrency
• Single stack
• Flexible hardware/software and system boundary
• Expressive enough to build sophisticated,
  application specific system structures
• Avoid arbitrary constraints on optimization
• Communication is integral to execution
• Asynchrony is first class
• Promote robustness
• Modular
• Static allocation
• Explicit success/fail at all interfaces
• Reuse
• Ease of interpositioning

6
Tiny OS Concepts

• Scheduler Graph of Components
• constrained two-level scheduling model tasks
  events
• Component
• Commands
• Event Handlers
• Frame (storage)
• Tasks (concurrency)
• Constrained Storage Model
• frame per component, shared stack, no heap
• Very lean multithreading
• Efficient Layering
• Events can signal events

Events
Commands
send_msg(addr, type, data)
power(mode)
init
Messaging Component
Internal State
internal thread
TX_packet(buf)
Power(mode)
TX_packet_done (success)
init
RX_packet_done (buffer)
7
Application Graph of Components
Route map
router
sensor appln
application
Active Messages
Radio Packet
Serial Packet
packet
Temp
photo
SW
Example ad hoc, multi-hop routing of photo
sensor readings
HW
UART
Radio byte
ADC
byte
3450 B code 226 B data
clocks
RFM
bit
Graph of cooperating state machines on shared
stack Execution driven by interrupts
8
TOS Execution Model

• commands request action
• ack/nack at every boundary
• call cmd or post task
• events notify occurrence
• HW intrpt at lowest level
• may signal events
• call cmds
• post tasks
• Tasks provide logical concurrency
• preempted by events
• Migration of HW/SW boundary

data processing
application comp
message-event driven
active message
event-driven packet-pump
crc
event-driven byte-pump
encode/decode
event-driven bit-pump
9
Dynamics of Events and Threads
bit event gt end of byte gt end of packet gt
end of msg send
thread posted to start send next message
bit event filtered at byte layer
radio takes clock events to detect recv
10
Programming TinyOS - nesC

• TinyOS 1.x is written in an extension of C,
  called nesC
• Applications are too!
• just additional components composed with the OS
  components
• Provides syntax for TinyOS concurrency and
  storage model
• commands, events, tasks
• local frame variables
• Rich Compositional Support
• separation of definition and linkage
• robustness through narrow interfaces and reuse
• interpositioning
• Whole system analysis and optimization

11
Event-Driven Sensor Access Pattern
command result_t StdControl.start() return
call Timer.start(TIMER_REPEAT, 200) event
result_t Timer.fired() return call
sensor.getData() event result_t
sensor.dataReady(uint16_t data)
display(data) return SUCCESS
SENSE
LED
Photo
Timer

• clock event handler initiates data collection
• sensor signals data ready event
• data event handler calls output command
• device sleeps or handles other activity while
  waiting
• conservative send/ack at component boundary

12
TinyOS Commands and Events
... status call CmdName(args) ...
command CmdName(args) ... return status
event EvtName)(args) ... return status
... status signal EvtName(args) ...
13
Split-phase abstraction of HW

• Command synchronously initiates action
• Device operates concurrently
• Signals event(s) in response
• ADC
• Clock
• Send (UART, Radio, )
• Recv depending on model
• Coprocessor
• Higher level (SW) processes dont wait or poll
• Allows automated power management
• Higher level components behave the same way
• Tasks provide internal concurrency where there is
  no explicit hardware concurrency
• Components (even subtrees) replaced by HW and
  vice versa

14
TinyOS Execution Contexts

• Events generated by interrupts preempt tasks
• Tasks do not preempt tasks
• Both essential process state transitions

15
Storage Model

• Local storage associated with each component (or
  instance of)
• Internally managed
• Only objects passed by reference are message
  buffers

16
Data sharing

• Passed as arguments to command or event handler
• Dont make intra-node communication heavy-weight
• If queuing is appropriate, implement it
• Send queue
• Receive queue
• Intermediate queue
• Bounded depth, overflow is explicit
• Most components implement 1-deep queues at the
  interface
• Dont force marshalling/unmarshalling through
  generic queue mechanism
• If you want shared state, created an explicit
  component with interfaces to it.

17
TASKS

• provide concurrency internal to a component
• longer running operations
• are preempted by events
• able to perform operations beyond event context
• may call commands
• may signal events
• not preempted by tasks

... post TskName() ...
task void TskName ...
18
Typical application use of tasks

• event driven data acquisition
• schedule task to do computational portion

event result_t sensor.dataReady(uint16_t data)
putdata(data) post processData()
return SUCCESS task void processData()
int16_t i, sum0 for (i0 i maxdata
i) sum (rdatai 7)
display(sum shiftdata)

• 128 Hz sampling rate
• simple FIR filter
• dynamic software tuning for centering the
  magnetometer signal (1208 bytes)
• digital control of analog, not DSP
• ADC (196 bytes)

19
Tasks in low-level operation

• transmit packet
• send command schedules task to calculate CRC
• task initiated byte-level data pump
• events keep the pump flowing
• receive packet
• receive event schedules task to check CRC
• task signals packet ready if OK
• byte-level tx/rx
• task scheduled to encode/decode each complete
  byte
• must take less time that byte data transfer
• i2c component
• i2c bus has long suspensive operations
• tasks used to create split-phase interface
• events can procede during bus transactions
• Timer
• Post task in-critical section, signal event when
  current task complete

Make SW look like HW
20
Example Radio Byte Operation

• Pipelines transmission transmits single byte
  while encoding next byte
• Trades 1 byte of buffering for easy deadline
• Separates high level latencies from low level
  real-time requirements
• Encoding Task must complete before byte
  transmission completes
• Decode must complete before next byte arrives

Encode Task
Byte 2
Byte 1
Byte 3
Byte 4
Bit transmission
Byte 1
Byte 2
Byte 3
start
RFM Bits
Hardware accelerators in MICA eliminate bit pumps
21
Task Scheduling

• Currently simple fifo scheduler
• Bounded number of pending tasks
• When idle, shuts down node (except clock)
• Uses non-blocking task queue data structure
• Simple event-driven structure control over
  complete application/system graph
• instead of complex task priorities and IPC

22
Structured Events vs Multi-tasking

• Storage
• Control Paradigm
• Always block/yield rely on thread switching
• Never block rely on event signaling
• Communication Coordination among potentially
  parallel activities
• Threads global variables/mailboxes, mutex,
  signaling
• Preemptive handle many potential races
• Non-premptive
• All interactions protected by costs system synch
  ops
• Events signaling
• Scheduling
• Complex threads require sophisticating scheduling
• Collections of simple events ??

23
Communication

• Essentially just like a call
• Receive is inherently asynchronous
• Dont introduce potentially unbounded storage
  allocation
• Avoid copies and gather/scatter (mbuf problem)

24
Tiny Active Messages

• Sending
• Declare buffer storage in a frame
• Request Transmission
• Name a handler
• Handle Completion signal
• Receiving
• Declare a handler
• Firing a handler
• automatic
• behaves like any other event
• Buffer management
• strict ownership exchange
• tx done event gt reuse
• rx must rtn a buffer

25
Sending a message

• Refuses to accept command if buffer is still
  full or network refuses to accept send command
• User component provide structured msg storage

26
Send done event
event result_t IntOutput.sendDone(TOS_MsgPtr
msg, result_t success) if
(pending msg buf) pending
FALSE signal IntOutput.outputComplete(success)
return SUCCESS

• Send done event fans out to all potential senders
• Originator determined by match
• free buffer on success, retry or fail on failure
• Others use the event to schedule pending
  communication

27
Receive Event
event TOS_MsgPtr ReceiveIntMsg.receive(TOS_MsgPtr
m) IntMsg message (IntMsg )m-gtdata
call IntOutput.output(message-gtval)
return m

• Active message automatically dispatched to
  associated handler
• knows the format, no run-time parsing
• performs action on message event
• Must return free buffer to the system
• typically the incoming buffer if processing
  complete

28
Maintaining Scheduling Agility

• Need logical concurrency at many levels of the
  graph
• While meeting hard timing constraints
• sample the radio in every bit window
• Retain event-driven structure throughout
  application
• Tasks extend processing outside event window
• All operations are non-blocking

29
A Complete Application
SenseToRfm
generic comm
IntToRfm
AMStandard
RadioPacket
UARTPacket
packet
noCRCPacket
photo
Timer
MicaHighSpeedRadioM
phototemp
SecDedEncode
SW
byte
RandomLFSR
SPIByteFIFO
HW
ADC
UART
ClockC
bit
SlavePin
Mica2 stack Interpostioning ChipCon
30
Composition

• A component specifies a set of interfaces by
  which it is connected to other components
• provides a set of interfaces to others
• uses a set of interfaces provided by others
• Interfaces are bi-directional
• include commands and events
• Interface methods are the external namespace of
  the component

provides
StdControl
Timer
provides interface StdControl interface
Timer uses interface Clock
Timer Component
Clock
uses
31
Components

• Modules
• provide code that implements one or more
  interfaces and internal behavior
• Configurations
• link together components to yield new component
• Interface
• logically related set of commands and events

Clock.nc interface Clock command result_t
setRate(char interval, char scale) event
result_t fire()
StdControl.nc interface StdControl
command result_t init() command result_t
start() command result_t stop()
32
Example top level configuration
configuration SenseToRfm // this module does
not provide any interface implementation
components Main, SenseToInt, IntToRfm, ClockC,
Photo as Sensor Main.StdControl -gt
SenseToInt Main.StdControl -gt IntToRfm
SenseToInt.Clock -gt ClockC SenseToInt.ADC -gt
Sensor SenseToInt.ADCControl -gt Sensor
SenseToInt.IntOutput -gt IntToRfm
Main
StdControl
SenseToInt
ADCControl
IntOutput
ADC
Clock
33
Nested configuration
StdControl
IntOutput
includes IntMsg configuration IntToRfm
provides interface IntOutput interface
StdControl implementation components
IntToRfmM, GenericComm as Comm IntOutput
IntToRfmM StdControl IntToRfmM
IntToRfmM.Send -gt Comm.SendMsgAM_INTMSG
IntToRfmM.SubControl -gt Comm
IntToRfmM
SendMsgAM_INTMSG
SubControl
GenericComm
34
IntToRfm Module
command result_t StdControl.start() return
call SubControl.start() command result_t
StdControl.stop() return call
SubControl.stop() command result_t
IntOutput.output(uint16_t value) ...
if (call Send.send(TOS_BCAST_ADDR, sizeof(IntMsg),
data) return SUCCESS ... event
result_t Send.sendDone(TOS_MsgPtr msg, result_t
success) ...
includes IntMsg module IntToRfmM uses
interface StdControl as SubControl
interface SendMsg as Send provides
interface IntOutput interface StdControl
implementation bool pending struct
TOS_Msg data command result_t
StdControl.init() pending FALSE
return call SubControl.init()
35
A Multihop Routing Example
36
Given the framework, whats the system?

• Core Subsystems
• Simple Display (LEDS)
• Identity
• Timer
• Bus interfaces (i2c, SPI, UART, 1-wire)
• Data Acquisition
• Link-level Communication
• Power management
• Non-volatile storage
• Higher Level Subsystems
• Network-level communication
• Broadcast, Multihop Routing
• Time Synchronization, Ranging, Localization
• Network Programming
• Neighborhood Management
• Catalog, Config, Query Processing, Virtual Machine

37
Timer

• Clock abstraction over HW mechanism
• 3-4 registers of various sizes
• Operations to set scale, phase, limit
• One-shot, periodic
• Signals critical event
• One physical clock must operate in minimum energy
  state with rest of the device power off
• Timer provides collection of logical clocks
• Clean units
• One-shot or periodic
• Manage Underlyng Clock Resources
• Mapping to physical clock
• Minimum spacing and accuracy
• Signals non-critical event

38
Data Acquistion

• Many different kinds of sensor connections
• Analog sensor into ADC
• Digital Sensors (i2c)
• External ADCs
• Get Command to Logical Sensor
• Data-ready signal
• Power management
• Quality of sampling interval (jitter) depends on
  timer and sensor substack
• Sample rate limits?

39
Communication

• Parameterized Active Message Abstraction
• Multiple Media
• Radio
• RFM, Mica-RFM, ChipCon
• UART
• i2C
• Media routing based on destination address
• Side Attributes
• Link-level ack
• Timestamp
• Signal-strength
• Security can be interposed (TinySec)
• Inherently Bursty and Unpredictable
• Unless higher level protocols make it predictable

40
Composable Power Management

• Scheduler
• On idle drops into preset sleep state till
  interrupt
• Proc Active 5 mA, Idle 2 mA, Sleep 5 uA
• Radio, Sensor, Co-Processors
• All Components implement Std_Control Interface
• Power mgmt of non-processor resources
• Start bring subsystem to operational level,
  inform pwr_mgmt
• Stop bring subsystem to sleep, inform pwr_mgmt
• Power-management Component
• Establishes sleep state should scheduler idle
• Knows what is shut down, potential sleep duration
• Timer
• Pulls system out of deep sleep
• What happens after depends on what event handler
  does
• Applications compose policy

41
Example Pwr Mgmt Policies

• Great Duck Island
• Every 5 mins sample, send, low_power_route
• Low-power listen
• Transmit include wake preamble (x)
• Receive powers off radio (x) if sufficiently
  long
• Application just adjusts x
• Generic Sensor Kit epochs
• Periodic Active phase / Sleep Phase
• Active
• Sample, process query, route, aggregate
• Sleep
• Power down for set time

42
Dynamics

• All sleep till clock expires
• Interrupts proc to HW event
• Critical handling, signal clock event
• Timer Handles Clock Event
• If merely expired as part of longer sleep, system
  returns to deep sleep
• If events are associated with event, they are
  signalled
• Post tasks
• Call Start,

43
Typical Operational Mode

• Major External Events
• Trigger collection of small processing steps
  (tasks and events)
• May have interval of hard real time sampling
• Radio
• Sensor
• Interleaved with moderate amount of processing in
  small chunks at various levels
• Periods of sleep
• Interspersed with timer mgmt

44
Where to classical RTOS issues arise?

• Low latency events
• Two-level scheduling, short critical events
• Lowest layer of components complete critical
  section before signalling logical event
• Closely aligned low-latency events?
• Radio/uart/timer happen to hit at same time
• Periodic, pseudo-periodic
• Timer component design
• Static scheduling
• Rate monotonic?
• Scheduler?
• Networking is bursty. Network sheduling?
• Deadline?
• Data queuing
• Load shedding?
• At load duty cycle?
• Provability?

45
Supporting HW evolution

• Distribution broken into
• apps top-level applications
• lib shared application components
• system hardware independent system components
• platform hardware dependent system components
• includes HPLs and hardware.h
• Component design so HW and SW look the same
• example temp component
• may abstract particular channel of ADC on the
  microcontroller
• may be a SW i2C protocol to a sensor board with
  digital sensor or ADC
• HW/SW boundary can move up and down with minimal
  changes

46
Scalable Simulation Environment

• target platform TOSSIM
• whole application compiled for host native
  instruction set
• event-driven execution mapped into event-driven
  simulator machinery
• storage model mapped to thousands of virtual
  nodes
• radio model and environmental model plugged in
• bit-level fidelity
• Sockets basestation
• Complete application
• including GUI

47
Issues

• Timer Architecture far more fundamental than
  anticipated
• Simple applications built around clock
• One component set period, all took multiples of
  it.
• Poor composition.
• Application Oblivious Timer too imprecise
• More complex applications have 1 high precision
  timer
• maybe 2
• sometimes high rate
• Besides radio
• Must be serviced by hardware periodic counter
• Other general uses should have a clean way to
  obtain intervals in multiples of primary tick in
  clean manner
• Offset in phase to avoid collision
• Coordinate multiple low-jitter subsystems
• UART and Radio

48
Multiple levels of tasks

• Easy to implement multiple, static priorities
  upon single stack
• Lowest level events handler immediately
• System tasks to realize concurrency within
  components
• Must still be quick to keep flow throughout
  system smooth
• Grains of sand, not bricks
• Background tasks
• Run as long as they want, preempted by system
  tasks
• Build next to it, but not on top of it.

49
Co-processors for fidelity

• Several designs have subsystems with dedicated
  tinyOS processors
• Ranging
• MotoControl

50
Ranging Board

• Lessons from sounder, UCLA ultrasound, VU ranging
• Dedicated Atmega 8, like motor-control board
• Dual proc TinyOS with UART link
• TX
• Command Processor
• Generate 25 KHz 10v signal (range)
• RX
• Triggered by radio chirp
• Analog switch
• Starts Atmega 8 timer
• Analog compare amplified recv with digital pot
  threshold.
• Completes timer
• Signals time-stamped event

25 KHz US Transceiver
tone
Atmel 8
digital pot
UART AM channel
Main TinyOS Mote
51
Ultra-Sound Ranger

• Design Background
• UCLA Ultra-sound provides good directional
  ranging
• High frequency, high amplitude pulse
• PEG requires ranging in the plane
• Rob Szewczyk designed variant of CMU cone
• Vanderbilt showed good acoustic ranging with high
  sample rate and sophisticated processing
• Dual TinyOS approach off-loads sampling
• Relatively simple processing
• 5 cm accuracy in the plane
• 5 m range
• Demo display range of roving node

reflector
ultrasound
52
Managing Critical Section

• General philosophy
• Lowest level (hardware abstraction) components
  perform minimal processing to package interrupt
• Reenable before signally event
• Implies that events may stack
• Non-interference from separation of components at
  the leaves
• No multiple interrupts within component
• Event handling less than interval for particular
  type
• Where insufficient, build a queue
• Interrupt disable may be used to realized
  atomicity not provided in the hardware

53
Avoiding Races

• AC code reachable from a hardware interrupt
• SC code reachable only from a task
• AC code requires great care to avoid potential
  races
• SC code can be much sloppier (and cheaper) if
  there is no sharing with AC code
• EX managing a pending flag
• Most complex components are higher level and will
  never be connected to truly asynchronous event
  sources
• Compiler can help detect race conditions
  knowing context of components use

54
Other points of discussion

• No dynamic allocation of msg buffers may be too
  draconian
• Encapsulation in stack composition
• Component behavior dependent on composition
• Layered Headers
• System construction as synthesis from
  specification