Devices in RealTime Systems Mark Stanovich

1
Devices in Real-Time SystemsMark Stanovich
1
2
Outline

• Who am I?
• Area of Research
• Throttling On-Disk Schedulers
• Modeling Device Driver Effects
• What I have learned?

2
3
Who am I?

• Worked with Dr. Baker, Dr. Wang and Dr.
  Gopalan on NSF grant
• Focus is fitting device drivers into real-time
  systems
• Received MS from Florida State April 2008
• Currently working on PhD

3
4
Throttling On-Disk Scheduler to Meet
Soft-Real-Time Requirements
4
5
Overview

• Problem context real-time disk I/Os
• Meeting I/O timing constraints in the presence of
  interfering background requests
• Frequently misaligned goals between the OS and
  hard disk schedulers
• Solution
• Dynamic coordination with the hard disk scheduler

5
6
Background
I/O requests
user space
kernel space
OS
RT aware
hardware
disk
RT unaware
6
7
I/O Scheduling
I/O requests
user space
RT
kernel space
RT
hardware
19 msec
7
8
I/O Scheduling
I/O requests
BE
RT
RT aware
RT
BE
RT unaware
2 sec
8
9
Command Queuing

• Protocol to issue multiple, outstanding requests
  to the disk
• NCQ/TCQ

9
10
Autonomous disk characteristics

• Internal queue
• ability to issue multiple commands to the drive
• Internal scheduler
• reordering of command

I/O Scheduler
I/O Scheduler
OS
Hardware
10
without internal queue
with internal queue
11
Logical Block Ordering
200
400
700
900
logical block addresses to retrieve
11
12
Service Ordering
TCQ/NCQ
12
1
2
3
4
13
Pros of On-disk Scheduling

• Can better optimize performance using
• Zero-latency reads
• Radial track position
• Angular position of the data
• Serpentine ordering
• Flaw mapping
• Zoned bit recording
• ...
• Features exist for important reasons

13
14
Cons of on-disk Scheduling

• Less preemptible
• may have to wait until all previously queued
  requests complete
• Potentially undisclosed scheduling algorithm
• not chosen by OS or application
• once sent to the disk, requests are out of OS
  control
• Extremely hard to predict response times

14
15
Alternative 1 No Disk Scheduling

• OS sends at most 2 requests at a time
• OS orders requests based on disk layout, timing
  characteristics, current internal state
• potentially good performance
• imposes timing constraints on driver
• needed information costly to obtain
• not published
• depends not just on model but also on instance

15
16
Alternative 2 On-Disk Scheduling

• Instructs command queuing to service RT requests
  first
• Pros
• bounds response time
• Cons
• disks cannot optimize until RT requests are done
• most of the time this may not be needed
• does not utilize the full benefit of the disk's
  hardware

16
17
Our Approaches

• Allow the disk to handle response times
• Throttle other disk requests to allow queue to
  drain when real-time request present
• Limit on-disk queue depth to bound worst-case
  drain time

17
18
Scope

• Single RT request stream
• Multiple best-effort requests
• Independent disk drives

18
19
Experimental Configuration

• Pentium D 3GHz
• 2GB DDR2 533 RAM
• Linux 2.6.21-RT PREEMPT
• Adaptec SAS controller on PCI-Express
• 15K RPM Fujitsu disk drive

19
20
1. Built-in Starvation Control

• Disk internal scheduler prevents starvation
• Bounds worst-case response time
• Throughput is decent
• If this is good enough, we are done

20
21
How to Test Starvation Control
single real-time request stream
many best-effort streams
21
22
Response Time
22
23
2. Draining Disk Queue

• One source of extended delays
• BE requests can starve RT request
• Draining
• No requests to disk until RT request returns

23
24
RT response time
Fujitsu
IBM
Maxtor
24
25
Observations

• On-disk scheduling is complex
• Different characteristics for each vendor
• Draining reduces worst-case response time

25
26
Worst-Case Drain Time
Objective find worst-case drain time
20 request batches ?
all requests 256K reads, from random locations
26
27
Drain 20 Requests
27
28
Observations

• Drain time can be bounded empirically
• lt 20x worst-case individual read time

28
29
Verification of Draining
Objective verify that draining with queue size
20 guarantees 160 msec worst-case read time.
queue limit 20
background tasks (unlimited demand)?
real-time task (one request 160 msec deadline)?
all requests 256K reads, from random locations
29
30
Response Times with Draining
30
31
Observations

• Throughput remains 40 MB/sec
• No deadlines missed at 160 msec
• compare to 2 sec without draining
• Can shorten queue for shorter deadlines

31
32
Drain Time vs. Queue Length

• drain_time(n)? max time to service n disk
  requests

32
33
3. Dynamic Draining

• Objectives
• support short deadlines
• good bandwidth
• Idea drain queue only when deadline at risk
• send new request when
• now drain_time(X1) earliest deadline
• X current on-disk queue depth

33
34
Verification of Dynamic Draining
Objective verify that draining with variable
queue limit guarantees shorter worst-case read
times.
queue limit 7
background tasks (unlimited demand)?
real-time task (one request 75 msec deadline)?
all requests 256K reads, from random locations
34
35
Response Times with Draining
35
36
Observations

• No deadlines missed at 75 msec
• Bandwidth dropped from 40 to 34 MB/sec
• 70 drop in queue lengthwith only 15 bandwidth
  drop

36
37
Work in Progress

• Multiple real-time requests

37
38
Conclusion

• More intelligent, autonomous hard drives increase
  the complexity of scheduling requests
• External disk scheduling becomes more difficult
  as disks evolve
• Possible to use built-in disk schedulers while
  still maintaining timeliness constraints
• Our approaches are not vendor-specific and can be
  applied to other operating systems

38
39
39
40
Modeling Device Driver Effects in Real-Time
Schedulability Analysis Study of a Network Driver

• Mark Lewandowski, Mark J. Stanovich, Theodore P.
  Baker, Kartik Gopalan, An-I Andy Wang

41
Overview

• Device drivers a major challenge for real-time
  systems schedulability analysis in fixed priority
  systems
• Usually provided by third parties,not easily
  tailored to fit application
• Workloads
• Arrivals tend to be aperiodic, hard to
  characterize
• Difficult to account for execution time
• Fragmented control flow makes worst case
  execution time difficult to determine
• Scheduling
• Non-preemptive
• Adhoc low-level mechanisms
• Fragmented control

42
Overview
dA
dB
B
A
43
Overview
dA
dB
A
B
44
Overview

• Our approach
• Empirical measurements to characterize workloads
  and compute an upper bound on the systems load
• Modified device driver task scheduling to use a
  time budgeting algorithm

45
Key Terms

• ISR
• Interrupt Service Routine
• Soft IRQ
• Work triggered by an interrupt that the device
  driver can defer
• Load for a task
• The CPU utilization for a task over some time
  interval
• maximum amount of time task
  ?i executes in any interval of length ?

46
Background

• Move work from ISRs and ad hoc mechanisms to
  scheduled threads
• e.g., TimeSys Linuxdoes this

47
Roadmap

• Model device-driver workload
• Apply appropriate scheduling priority and policy
  to device-driver threads
• Analyze schedulability of system with device
  driver

48
System Profiles

• 3.0 GHz Pentium D (single core mode)
• 2.0 GB RAM
• Intel Pro/1000 Ethernet adapter on a dedicated
  gigabit switch
• Operations
• Input
• Output
• NAPI
• Reduces the number of interrupts
• Driver polls during periods of high traffic

49
B
A
10us
System receives and responds to packets from
A Task B Estimates the amount of CPU time
consumed by the device driver.
Task A Sends ICMP ping traffic to B
50
Observed interference with ping flooding
51
A Comparison of Load Bounds
52
Schedulability Test

• A task will always complete by its deadline if it
  satisfies the above equation

53
Individual load bound functions for real time
task device driver, and their sum
54
How well can we trust such an empirical bound?

• Probably as well as worst case execution time
  estimates
• If thats not good enough, we can enforce a
  bandwidth limit
• In that case, periodic (zig-zag) load bound may
  be used, for greater precision
• Also permits better tuning to balance CPU
  interference due to driver against I/O performance

55
Interference vs I/O Service Quality
B
A
C
Heartbeat (10ms)
Ping flood (10us)
Best case receive 100 of heartbeat messages
while missing no deadlines for the periodic tasks
on machine C
56
Expectations

• Vanilla Linux scheduling
• Has rough bandwidth-limit
• Softirqs served by low-priority thread past
  limit
• Foreground highest thread priority
• Missed deadlines ?, received heartbeat packets ?
• Background lowest thread priority
• Missed deadlines ?, received heartbeat packets ?
• Sporadic server highest thread priority
• Missed deadlines, received heartbeat packets
  bounded

57
Missed Deadlines
58
Heartbeats received
59
Conclusions

• Model the worst-case interference of the device
  driver by a hyperbolic load bound function
  derived from empirical data
• Schedule the device by an aperiodic scheduling
  algorithm and budget processor time to be
  consistent with the analytically derived load
  bound function

60
What I have learned?

• Do something you enjoy
• Talk with your advisor
• Make a list of goals and deadlines
• Focus your area of research

60
61
Device Interference with No Network Traffic
62
What is Load?

• Generalization of processor utilization
• Percent of processor time a task executes
• ratio of the maximum amount of
  time task ?i executes to any interval of length ?

?1
?1
?1
?1
?1
63
Related Work

• Scheduling hardware interrupts
• NAPI Mogul97
• Hardware Priorities
• Motorola 68xxx architecture
• Special hardware and software support for
  reducing interrupts Regehr05
• RTLinux resides between hardware and host OS to
  schedule interrupts Yodaiken99

64
Related Work

• Defer interrupt-triggered work to threads
• Executing Device Drivers in the context of
  threads Klieman95, del Foyo06, Thekkath93,
  Druschel96
• Facchinetti05 execute driver code in interrupt
  context and enforce a time budget.
• Executes all devices in the context of the same
  thread, at the same priority
• Zhang06 propose a method for minimizing the
  priorities of a drivers bottom halves
• Our research addresses the case where there is no
  clear process to charge for ISR execution
• Our research is not susceptible to backlog