L1 timing measurements and timing in track trigger

1

Pulsar Status and Plan

Ted Liu, Oct. 13, 2003, RunIIb DAQ review

• PeopleInstitutions involved
• Project status whats been done whats left
  to be done
• Roadmap to the final new system
• Man power issues

More details can be found at http//www-cdf.fnal.
gov/internal/WebTalks/0310/031010_2bl2.html Projec
t web page http//hep.uchicago.edu/thliu/project
s/Pulsar/ CDF L2 upgrade Pulsar
e-log http//www-cdfonline.fnal.gov/cgi-bin/det-s
ystem-elog.pl?nbl2upgrade
2
PeopleInstitutions involved in Pulsar project
? from few people last year to now

• Related to L2 decision crate upgrade
• ANL
• R. Blair, J. Dawson, B. Haberichter, J.
  Schlereth, J. Proudfoot
• FNAL
• R. Demaat, M. Hakala, R. Kivilahti, J. Lewis,
  C. Lin, T. Liu, T. Masikkala,
• F. Marjamaa, J. Patrick, S. Pitkanen, B.
  Reisert, P.Wilson
• UC
• M. Bogdan, Y. Kim, W. Fedorko, H. Frisch, S.
  Kwang, V. Rusu, H. Sanders,
• M. Shochet
• Upenn
• K. Hahn, P. Keener, J. Kroll, C. Neu, F.
  Stabenau, R. Van Berg, P. Wittich
• Related to SVT upgrade (using Pulsar)
• INFN
• F. Spinella, L. Ristori
• LBL
• A. Cerri

3
New people

• Two new posdocs joined the project recently
• Chris Neu (Upenn), and Vadim Rusu (UC)
• Both are damn good, and they will make a big
  difference.
• Y2K also has a new student (first year),
  Wojciech Fedorko
• YoungKees group is joining the project

4
Project status whats left to be done?

• Many aspects
• Hardware
• firmware
• VME standalone testing software
• DAQ code
• online monitoring code
• offline validation code
• PCI/CPU interface, L2 algorithm etc

Much easier to discuss this with
one baseline system configuration
5
Pulsar pre-processors

New L2 Baseline Configuration
L1 muon L1 XTRP L1 trigger SVT
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
L2 CAL
PC 1
cluster
PC 2
merger
TS
RECES
ShowMax
L2toTS
merger
6
Pulsar pre-processors
Pulsers (Tx)

Hardware involved (with Txs)
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
Pulsar mergers or GBE switch
L2toTS
merger
7
Roadmap (as shown in Jan 30th, 2003)

• Three phases/stages
• Phase A Teststand, core firmware, HW
  prod/testing
• Algorithm firmware
  specification
• PCI/CPU/OS/infrastructure
  software
• We are now entering phase B
• Phase B Rx algorithm firmware
  implementation/testing,
• system integration in
  standalone mode
• Phase C System integration in parasitic mode,
  test runs
• with cosmic and beam

8
Project Overview (as shown in Jan.30th, 2003)

• So far has been divided into four
  parallel efforts
• E1 Pulsar hardware/firmware/VME software
• E2 Design spec. for data format/algorithm/system
  interface
• E3 RoIB option effort (a backup option)
• E4 PCI/CPU architecture/OS/Infrastructure
  SF/L2D SF

E1 E2 E3 E4
new L2
Phase A Phase B Phase C
9
What has been done so far this year
The project has gone extremely well so far TONs
of work has been done, due to the hard work by
many We have achieved most of the goals (much
more in some areas) we set in Jan. 30th,
2003. Lots of work done by young students, This
is definitely a students friendly
project. Will summarize each effort first
10
E1What has been done so far

• ALL mezzanine cards productiontesting finished
• AUX card production/testing started
• Pulsar pre-production on-going
• All data paths have been initially tested in
  standalone mode
• VME standalone testing code automated for
  production testing
• fiber splitter has been tested in standalone
  mode
• initial RECES Rx firmware tested in standalone
  mode
• initial CLIST/Isolation Rx firmware testing
  on-going
• firmware for Pulsar ?? S32PCI64 (CPU) interface
  tested
• working closely with SVT folks for SVT II
  upgrade using Pulsar
• Muon/XTRP path fully established tested with
  beam
• Young people who made significant
  contribution to E1 effort
• FNAL Burkard/Sakari/Mikko/Tomi/Cheng-Ju/Fran
  , UC Shawn
• Muon special thanks to Eric James, DAQ folks
  (esp. Jane Nachtman),
• operation managers and friendly shift crews

11
E2What has been done so far

• Pulsar Data format spec for all data paths in
  place
• worked closely with DAQ folks for muon path
• offline software for muon/XTRP path working
  (five banks)
• emulation software for muon/XTRP internal
  algorithm working
• online monitoring software for XTRP/muon path
  working
• Alpha code to separate XTRP/muon data working
  with beam
• Alpha code for L2 muon algorithm on-going
• extensive offline data analysis for Pulsar
  muon/XTRP beam data
• worked very closelyeffectively with E1, esp.
  for commissioning
• the L2 muon/XTRP path with the beam

Young people who made significant contribution
for E2 effort FNAL Cheng-Ju/Frans/Burk
ard, UC Shawn
12
E4What has been done so far

• Algorithm timing performance done (alpha vs
  modern CPUs)
• S32PCI64 Tx as back channel established
• Initial testing software in place
• Pulsar ? S32PCI64?CPU ? S32PCI64?Pulsar test
  done
• Timing (w/o algorithm running) measured with
  Pulsar ?? CPU
• Gigabit Ethernet option testing on-going

people who made significant contribution for
E4 effort Upenn
Peter/Paul/Kristian/Fritz, ANL Bob
13
Pulsar is designed to be fully self-testable
at board level as well as system level
For every input/output, there can be an
output/input, All interfaces are
bi-directional (except VME)
TSI
SVT/XTRP
one for all and all for one
SRAMs
Gigabit Ethernet
RF clock
It is this feature which allows us to
developtune an upgrade
system in standalone mode
Overall planning is based on this feature
14
One slide from my talk on Jan 30th, 2003
Guideline for the project planning
Project planning follows
Pulsar design philosophy/methodlogy
(1) In God we trust, everything else we test (2)
One for all and all for one
Anyone who has a problem with this is in the
wrong project Success in muon path Pulsar
methodology at work ?
15
June 2003 Jimmy ordered us to jump, we asked
how high?
Pulsar RunIIA Muon path From concept to error
free path takes 3 months
Jimmys office discussion in June about L2
Pulsar muon Path for RunIIA
16
Muon path commissioning to kludge up something
is easy, to have something robust error free is
not

• The 2rd day we came up with a detailed plan for
  the
• first 6 weeks, since then
• each week the goals were met for all 6 weeks
• after that, another 6 weeks to FULLY debug the
  full path
• at the same time, we took care of taxi
  mezzanine cards
• production/testing initial Pulsar
  pre-production testing,
• training three new Finnish students
• during that period (three months time span),
• Sakari/Burkard/Cheng-Ju each took 2 weeks
• vacation (at different time)
• the team worked together in a highly coherent
  way!
• details see L2 upgrade Pulsar e-log
• This gives you some idea how much it takes to
  fully
• commission one data path. The experience gained
• is extremely useful for the rest of data paths

17
Muon path commissioning experience as a guide
Output Options
Pulsar Muon Board
Input Options
L2 TrkList Board in b0l2de00
L1 Muon (16 hotlink fibers) and/or L1 XTRP Cable
OR
Pulsar Receiver
OR
Pulsar muonxtrp data transmitter

• hardware firmware
• VME standalone test software
• DAQ code
• online monitoring (XFLD/XTRD/TL2D/TCMD/TP2D)
• offline validation (XFLD/XTRD/TL2D/TCMD/TP2D)
• commissioning teststand,cosmic/L2 torture/beam

We didnt waste one second of beam time (except
the reformatter errors), The full chain test with
beam works on the first try (error free).
18
Test Runs Summary
STORE 2985        2003/09/03 2149  2003/09/04
1451 Run         L3A  Lumi_live 
Max. rate     Avg. rate     Inst.Lumi
(E30)                     
(nb-1)  L1 / L2 / L3   L1 / L2 / L3  
start/end168766    335,930    196.220 
15.0k/268/58   13.3k/255/53  
35.0/30.2168767    155,974    85.949 
13.7k/240/53   13.0k/230/50 
29.5/28.0168775    596,981   258.197 
12.5k/190/45   10.6k/169/39 
19.7/15.8STORE 2988        2003/09/04 2126 
2003/09/05 1021Run         L3A 
Lumi_live  Max. rate       Avg. rate      
Inst.Lumi (E30)                     
(nb-1)  L1 / L2 / L3   L1 / L2 /
L3   start/end168819     90,076   
76.570  16.5k/148/30    13.8k/132/27  
24.7/23.1168820  1,101,959   471.469 
14.7k/235/50   12.0k/193/47 
22.7/14.8168821    166,146   114.954  
9.6k/ 94/22    8.6k/ 86/20 
14.5/13.2168822    121,685    50.591  
8.4k/134/34    7.3k/121/30 
13.2/12.6STORE 2988        2003/09/04 2126 
2003/09/05 1021Run         L3A 
Lumi_live  Max. rate       Avg. rate      
Inst.Lumi (E30)                     
(nb-1)  L1 / L2 / L3   L1 / L2 /
L3   start/end168889    2,816,155   
1,359.049  15.5k/300/60    12.2k/205/45  
40.0/13.7
ALL data collected for Pulsar beam test are good
for physics analysis. No single Pulsar
hardware/firmware problem was seen ( 5M evts.)
19
Muon path commissioning experience as a guide

• of course, doing it the right way doesnt mean
  success _at_ CDF
• I was in the mountains when I heard the rumor
  Pulsar is it
• you get much more publicity if you do it the
  wrong way

Jade Dragon Mountain, Yunnan, China, Sept. 24, 03
20
Hardware status (no revision to any board)
Hotlink Tx/Rx Prod./testing done Taxi
Tx/Rx Prod./testing done
MOAB (Mother of All Boards)
Pulsar
Expect two pre-production boards arrive this week
eight more in the same batch (next
month) Expect production to start next month
ANL SLINK-GBE (production)
SLINK LSC/LDC (have some, need to order
more)
AUX Card
Six S32PCI64 in hand
production started two weeks ago
21
Pulsar pre-processors
Pulsers (Tx)
All Tx firmware in placetested Fine tuning
on-going
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
L2toTS
merger
Firmware status
22
Pulsar pre-processors
Pulsers (Tx)
Muon/XTRP path firmware in placetested with
beam Need SLINK output formating
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
L2toTS
merger
Firmware status
23
Pulsar pre-processors
Pulsers (Tx)
Clist input DAQ firmware written, testing
on-going. Iso input firmware written, simulation
on-going Soon ready to test with real system.
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
L2toTS
merger
Firmware status
24
Pulsar pre-processors
Pulsers (Tx)
Reces input DAQ firmware written, tested ok in
standalone mode, ready to test with real RECES
inputs
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
L2toTS
merger
Firmware status
25
Pulsar ?? S32PCI64/CPU interface firmware in
placetested, timing measured with both LA and
Pulsar internal counters (via VME)
Pulsar pre-processors
Pulsers (Tx)
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
L2toTS
merger
Firmware status
26
Initial SLINK merger firmware existed long ago.
New version is straightforward. (to be
done)
Pulsar pre-processors
Pulsers (Tx)
Simple firmware for L2toTS, to ensure L2A in L1A
FIFO order (to be done)
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
L2toTS
merger
Firmware status
27
Online code for the Muon/XTRP path (with five
banks) running in TrigMon. Other paths need to be
written
Pulsar pre-processors
Pulsers (Tx)
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
L2toTS
merger
Online monitor code status
28
Offline code written and used extensively for the
Muon/XTRP path (with five banks) to convert
real data into test patterns. Other paths need to
be written
Pulsar pre-processors
Pulsers (Tx)
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
L2toTS
merger
Offline validation code status
29
Have tested CPU performance (alpha vs modern
CPUs). Initial software for S32PCI64 (Tx/Rx)
in place tested with Pulsar, timing measured.
Pulsar pre-processors
Pulsers (Tx)
SLINK
muon
SLINK
PC 0
SLINK/GBE
PC 3
merger
PC 1
cluster
PC 2
merger
TS
RECES
L2toTS
merger
PCI/CPU status
30
Round-Trip timing
31
Round-Trip timing measured the performance
looks good

• w/ algo Blue w/o algos Red

• PULSAR ? PC
• ? Round-trip timing with realistic
• hardware setup
• ?Zero suppressed,
• S-Link formatted data
• ?Pulsar Firmware support for
• timing L2 decision verification
• ?Pulsar On-board timing counter

32
Still lots of work ahead of us devil is in the
details
path
firmware
Spec/test
DAQ
monitor
comment
RunIIa muon mostly done (Shawn alpha code)
Sakari Tomi
Burkard Cheng-Ju
Cheng-Ju Frans
Muon/ XTRP
Jane
student help from BR/CJL
Sakari Tomi
Jane/ Vadim
Vadim
Cluster
Vadim
Jane/ Chris
Sakari Tomi
student help from BR/CJL
Reces
Chris
Chris
Merger/ SVT
Sakari Tomi
Burkard Cheng-Ju
Cheng-Ju Frans
Mikko
Jane
Sakari Tomi
Burkard Cheng-Ju
Cheng-Ju Frans
Mikko
L2toTS
Jane
PCI/ CPU
Risto others
Peter/Bob/Paul/Kristian/Fritz
33
Comments on firmware effort

• This is a big project from firmware point of
  view,
• well organized effort is the key to success
• so far, we havent got engineer support from lab
  on firmware,
• we have had to train our own engineers!
• every young person involved in this project has
  been trained
• with firmware tools (Peter/Sakari/Burkard/Cheng-
  Ju/Tomi/
• Mikko/Frans/Shawn/Kristian/Fritz ).
• Sakari now Tomi are the dedicated (100 of
  time) people
• writing firmware, they have done a wonderful
  job.
• Firmware implementation isnt for everyone, very
  time consuming.
• Typically takes half year FULL time to learn
  before one really
• knows how to write good firmware (more
  difficult than C).
• physicists are strongly encouraged to get
  involved to
• understand the firmware and learn how to do
  simple modifications
• Pat Lukens will find a lab engineer for long
  term support

34
Well organized firmware effort is one of the keys
to have a system clean/easy-to-understand/robust
with minimal maintenance effort
Common core firmware
muon
VME responses SLINK formatter DAQ
buffers CDFCtrl responses SRAM ctrl Mezzanine
interfaces Diagnostic RAMs Spy buffers
SVT XTRP
LVDS ext. FIFO
cluster
hotlink
L1
LVDS
Taxi
Isolation
TSI
Reces
Knowing one data path Pulsar ? knows all
Pulsars Data specific algorithm difference is
just minor details
have dedicated people responsible for
developing the core firmware
35
Coordinate the efforts

• Hardware/VME test software (coordinator
  Burkard)
• main tasks train new people/test all new
  Pulsars/system test software
• PCI/CPU/L2 algorithm (coordinator Peter)
• main tasks production version
  software/simulation/train new people
• Data format/spec/monitor (coordinator Cheng-Ju)
• main tasks algorithm spec/monitoring for all
  paths/train new people
• DAQ liason (coordinator Chris or Vadim?)
• main tasks DAQ spec/code for all paths/train
  new people
• Firmware (coordinator TL, will find a
  replacement later)
• main tasks firmware spec for all cases/train
  new people

36
Roadmap to final system

• plan to have each path fully tested next spring.
• this should be possible in teststand mode, and
  with
• cosmic/L2 torture with fiber splitters. We will
  need to
• request for beam time (in parasitic mode only).
• to make this easier, the RunIIa Pulsar muon
  board needs
• to be moved into L2 decision crate at some
  point.
• system performance simulation to guide system
  configuration.
• all SLINK hardware need to be purchased by next
  spring.
• system integration next summer, in teststand
  mode
• first, then with the system and with fiber
  splitters.
• will need beam time testing before next summer
  shutdown.
• will discuss the details of integration early
  next year.

37
System performance simulation study? need help
from Yale

• We need to perform the simulation study soon, to
  guide system
• configuration,to find out how much performance
  gain we can
• expect with the new L2 system. For example
• gain from modern technology (speed in
  FPGA/CPU/PCI device)
• gain from architecture flexibilities
• (1) having four CPUs each dedicated for a
  given buffer event,
• with L2A issued in L1A FIFO order
  events do not
• have to wait in the pipeline while
  previous event(s)
• is being loaded or processed
• (2) CPU doesnt need to wait for SVT data
  before processing
• a given event
• (3) gain from SVT II/Silicon DAQ performance
  improvements
• (4) more

38
To make sure we can deliver a solidnew system
next summer, we will need
(1) Fermilab hire Sakari as visiting engineer for
one year. (2) Extend the stay of the three
Finnish students for 6 months. We have
invested a lot in them they have done
wonderfully well Tomi (firmware),
Frans(Pulsar emulation, online/offline
code, data integrity checks),
Mikko(hardware testing, VME software, Pulsar
GUI). They havent learned how to BS, will
not on this project. They are young work
only for foodbeer (1500 per month per
person). (3) New posdocs have to spend
significant amount (if not full) time on the
project in the coming year (Chris Vadim) ?
Without any of the three above, next summer is
not possible. To make sure the new system
is maintainable, Fermilab has to provide
an engineer for long term firmware/hardware
support. Each group needs to provide new
student(s) and posdoc(s) sometime next year.
39
From the last slide of my talk on Jan. 30th, 2003
RunIIA and RunIIB 2B or not 2B?
This is no longer the question. The only people
who could stop, slow down, screw up or speed up
this project is us. ?Whatever we do, we
have to do our very best The only way we can
deliver a solid new L2 system next summer is to
work hard work as a coherent team There is
no need or motivation to bring up an unreliable
system