2003 Conference for Computing in High Energy and Nuclear Physics

1
The DataFlow of the ATLAS Trigger and Data
Acquisition System

• Giovanna Lehmann
• On Behalf of the ATLAS
• Trigger/DAQ DataFlow Subsystem

2
Outline

• ATLAS
• Interaction rates and event sizes
• The Trigger/DAQ architecture
• The DataFlow
• ROS
• Design Performance
• LVL2 dataflow
• Design Performance
• Event Builder
• Design Performance
• Conclusions Outlook

3
CERN Accelerators Complex

• Colliding particles protons
• Center of mass Energy 14 TeV
• Bunch crossing rate 40 MHz
• Interaction rate 109 Hz
• Event size 1-2 Mbytes

4
ARCHITECTURE
Trigger
DAQ
Calo MuTrCh
Other detectors
40 MHz
D E T
L 1
FE Pipelines
2.5 ms
Lvl1 acc 75 kHz
ROD
RoI
120 GB/s
10 ms
D A T A F L O W
H L T
LVL2
ROS
RoI requests
L2SV
RoI data 2
L2N
L2P
EB req/clears
data
Lvl2 acc 2 kHz
EB
sec
Event Filter
3 GB/s
EFacc 0.2 kHz
5
The ROS

• Receive buffer event fragments from the 1628
  detector ROLs
• Up to 160 MB/s per ROL
• Send selected event fragments on request
• ROI requests high rate, low data volume
• Rate LVL1 rate (75 kHz), volume 2 of ROLs
• EB requests low rate, high data volume
• Rate 3 of LVL1 rate (2 kHz), volume complete
  event data
• Provide fragment sampling for data monitoring

6
ROS High Level Components
ROS subsystem
RODs
L2 EB
500x
Data requests
I/O Manager (SW process)
RobIn (custom module)
150x
Event Fragments
Local Controller (SW process)
150x
Control/Configuration
Online SW
Monitoring data
7
Test Setup ROS performance

• ROS implemented on a 2 GHz PC, with 4 PCI busses
  (64 bit/66 MHz)
• 3 RobIn emulators on PCI
• On-board local bus limited to 266MB/s
• Each simulates 4 input channels ? 12 ROLs per ROS
• I/O to/from L2 EB emulator
• Connected to the ROS through a GE switch
• Sends ROI/EB requests and clears to the ROS
• Receives data fragments back
• Uses TCP as communication protocol (maximum
  possible overhead for message passing)

8
ROS Performance
ATLAS baseline conditions (from paper model
contains safety factor 4 with respect to physics
simulation )
9
The LVL2 Dataflow

• Receive RoI information from LVL1
• 8 ROLs _at_ LVL1 rate (75 kHz)
• Form a LVL1 result record
• Build 1 record out of 8 _at_ LVL1 rate
• Retrieve RoI data from ROSs
• 2 of full Event (30 kB)
• Forward the LVL2 decision to the EB
• _at_ LVL2 accept rate (rejects are grouped)
• Forward the LVL2 decision record to the EB
• _at_ LVL2 accept rate (2 kHz)

10
LVL2 High Level Components
LVL2 subsystem
EB
LVL1
RoIBuilder (custom module)
L2SV (SW process)
Decisions
RoI information
1x
10x
pROS (SW process)
L2 record
ROS
1x
L2PU (SW process)
DC Controller (SW process)
RoI req./data
200-500x
Online SW
Control/Configuration
11
Performance of RoI Builder, L2SV and pROS

• Performance of RoI Builder
• Custom built 12U VME prototype has achieved
  required performance
• Performance of each LVL2 supervisor
• Measured to be 30 kHz on a 2.4 GHz dual CPU PC
• Is insensitive to the number of L2PUs
• Performance of pROS
• Not a demanding application
• Requirement to receive lt10 kB at LVL2 accept rate
  (3 kHz) and forward them to the EB is largely
  satisfied.

12
Test Setup Performance of L2PU

• ROS emulators used to send data over Gbit
  Ethernet.

• RoI data collection takes always a small fraction
  of the time requested by the LVL2 event
  processing (10 ms).
• From a dataflow point of view ltlt 100 L2PUs could
  sustain already the LVL1 rate.

13
The Event Builder

• Receive LVL2 decisions
• _at_ LVL2 accept rate (2 kHz rejects are grouped)
• Request data from ROS and pROS
• Build complete events
• Depending on ROS implementation merge 140-1600
  fragments into one.
• 70 MB/s at every SFI
• Distribute clears to ROS and pROS
• _at_ rate lt LvL2 accept rate
• Forward complete events to EF
• 70 MB/s at every SFI
• Provide fragment sampling for data monitoring

14
EB High Level Components
EB subsystem
L2
pROS
DFM (SW process)
LVL2 Decisions
Clears
1x
ROS
SFI (SW process)
DC Controller (SW process)
EB req./data
10x
50x
EF
Control/Configuration
Complete Event
Online SW
Monitoring
15
Test Setup EB Performance
Many ROSs to many SFIs
DFM
...
...
8x
16x
Switch

• ROS Emulators
• ALTEON programmable GE NICs
• Raw ethernet communication protocol
• Simulating n sources

• SFI applications were run on 2.4 GHz dual CPU PCs

16
EB Performance
8 ROLs/ROS Flow Control
Limit of 16 ROS emulators for single frame
messages
1 ROL/ROS No Flow Control
EB rate with 8 SFIs 350Hz (17 of ATLAS EB
rate)
17
Conclusions Outlook

• All elements of the DataFlow system have shown
  that they can satisfy the ATLAS requirements
  already with the present implementations and with
  todays technology.
• From now on emphasis will be put on the
  performance of the integrated DataFlow system.
• Testbeds are being setup to measure its behaviour
  and the first results are encouraging.

18
Spares
19
Results of Test 1 (no I/O to LVL2 EB)
ATLAS baseline conditions (from paper model)
20
Test Setup 2 Scaling of LVL2 Network
From a dataflow point of view a few L2PUs sustain
already a large fraction of the LVL1 rate.
21
Test Setup 1 Performance of the DFM
DFM
LVL2 decision (group)
Tester L2SV n SFIs
DFM_Decision
SFI EOE
Clears (group 300)

• Tester Application emulates L2SV and many SFIs
• DFM handling full I/O as for real ATLAS
• DFM exposed to full input message rate from
  tester
• DFM sending to non existing destinations
  (Connectionless protocol used)

22
DFM Performance
Rate function of CPU clock -gt
Test on a 2.2 GHz dual CPU PC
raw ethernet frames
udp
ATLAS event building rate
23
Test Setup 1 SFI Performance
Many ROSs to 1 SFI
1 Gbit/s Ethernet
DFM
...
16x
EF
Switch

• SFI application was run on a 2.4 GHz dual CPU PC

• ROS Emulators
• ALTEON programmable GE NICs
• Raw ethernet communication protocol
• Simulating n sources

24
SFI Performance
95 MB/s IO limited
EB only
With output to EF
CPU limited (2.4 GHz CPU)
ROLs/ROS

• Reaching I/O limit at 95 MB/s otherwise CPU
  limited
• 35 performance gain with at least 8 ROLs/ROS
• Will approach I/O limit for 1 ROL/ROS with
  faster CPU