L1%20timing%20measurements%20%20and%20timing%20in%20track%20trigger

1

CDF Pulsar project
Design and Testing
Methodology for
Level 2 trigger RunIIb upgrade
Tiehui Ted Liu,
Fermilab
(for CDF Pulsar group)
Oct. 22, 2003, IEEE NSS,
Portland

Pulsar web page http//hep.uchicago.edu/thliu/p
rojects/Pulsar/
2
Common issues with HEP electronics

• Uniformity modularity flexibility
• CAMAC/NIM days long gone ? commercial vs
  custom hardware?
• Design verification methodology
• Prototyping/design verification ? blue wires
  ?
• (posdocs/students)
  life time vs revision times
• Testability commissioning strategy
• boardsystem tests ? waiting for
  upstreamdownstream?
• Maintenance firmware version control
• firmware organization ? which version did we
  use back then?..
• Documenting the work
• who has the time for that?

3
Blue wires Often grow on prototypes look
familiar ?
4
Sometimes even grow on production boards or
hot spares The lack of blue wire means the
lack of testing ? from an expert
5
The color of wires
Blue Patch wires added to circuit boards at the
factory to correct design or
fabrication problems. Blue wire is not
necessarily blue, the term describes function
rather than color. Yellow Repair wires used
when connectors got broken due to
some schlemiel pinching them, or to reconnect
traces after the field engineer mistakenly cut
one Red Patch wires installed by programmers
who have no business mucking with the
hardware. It is said that the only thing more
dangerous than a hardware guy with a code patch
is a softy with a soldering iron .... Pink As
blue wire, but used in military applications
6
CDF Pulsar project how we address the common
issues

• Uniformity modularity flexibility
• Lego-style, general purpose, open design,
  
• One for ALL and ALL for one interface with
  commercial stuff
• Design verification methodology
• simulation simulation single-board
  multi-board
• Testability commissioning strategy
• board system level fully self-testable
• In God we trust, everything else we test
• Maintenance firmware version control
• common firmware library (VHDL), CVS
• Documenting the work
• web page http//hep.uchicago.edu/thliu/proje
  cts/Pulsar/

Some background information about Pulsar project ?
7
CDF L2 legacy decision Crate

• Technical requirement need a FAST way to
  collect/process many inputs
• ?With the technology available back then (1990s),
  had to design custom (alpha) processor
  backplane (magicbus)
• had to deal with the fact that each data input
  was implemented in a different way
• 6 different types of custom interface boards
  custom processor backplane

electron
Input up to 50KHz
a CPU
L2 decision crate
Output 300Hz
L2 decision crate
8

• Can we design an universal board
• to replace ALL ?
• ? Pulsar project
• Convert/pre-process/merge
• different data inputs into
• a standard link (CERN S-LINK)
• Interface with commodity PC(s)
• via high bandwidth SLINK?PCI
• Much enhanced performance
• Much easier maintenance
• Pulsar is designed to be
• able to do more than these
• But will only discuss how we address

s
s
s
s
s
s
s
Commodity Processors (Linux box)
Concept
S
9
Pulsar is designed to be Modular, universal
flexible,
fully self-testable (board system level)

Has ALL interfaces L2 decision crate has
Personality cards
user defined interfaces
Spare lines
Standard link
one for all and all for one
SRAMs
another Pulsar or S-LINK to PCI
RF clock
Gigabit Ethernet
All interfaces are bi-directional (Tx Rx)
Lego-style, open design
10
General purpose, useful within outside CDF
Mezzanine slots
AUX card
Top view
Pulsar
Bottom view
S-LINK
11
Pulsar Design methodology
Pulsar

• A major fraction of the effort was dedicated to
  extensive design optimization verification
• Leonardo Spectrum VHDL synthesis
• Altera Quartus II place and route,
• FPGA level
  simulation
• Mentor Graphics QuickSim
• (multi)-board level simulation
• Interconnect Synthesis tool
• trace cross talk
  analysis
• IS_MultiBoard tool
• signal integrity
  between
• motherboard mezz cards
• Design work done mostly by Fermilab
  Physicist/student Univ. of Chicago engineer

12
Example Multi-board (9 boards) simulation
(from input connector pins to output connector
pins)
It took 1.5 GB memory on a 2GHz/2GB modern PC
to simulate 9 boards together at the same time
13
Results No blue wire on all prototypes ? All
became production boards
Is this luck?

MOAB (Mother of All Boards)
AUX Card
Pulsar

works up to 100 MHz

• Design
• Verifications
• 3 people in
• 9 months

• Initial checkout of
• ALL interfaces
• 2 people in
• 1 month

4 types of custom mezzanine cards (Tx/Rx)
14

• Another successful example in the past
• with similar design methodology
• BaBar L1 track trigger boards (built _at_LBL)
• ? Maybe it is not just luck

Binary Track Linker
PT Discriminator
Track Segment Finder

• No blue wire on
• ALL prototypes
• ALL became
• production boards

Interface cards
15

Modular, universal flexible ?
One possible Pulsar formation
L1 muon L1 track L1 trigger L2 SVT

• General purpose DAQ
• CDF L2 upgrade

SLINK
muon
PCI?SLINK
PC 0
SLINK/GBE?PCI
PC 3
merger
L2 CAL
SLINK
PC 1
cluster
Pulsar pre-processors
PC 2
merger
Trigger Supervisor
electron
Pulsar mergers or GBE switch
ShowerMax
SLINK
L2toTS
SLINK
merger
16
CDF L2 legacy decision Crate

• This June, CDF management
• requested on short notice
• that Pulsar be used as the
• muon board for the current
• (RunIIa) system
• Due to Pulsars flexibility,
• this is possible
• Pulsar is self-testable, it
• is possible to do it fast

electron
Input up to 50KHz
a CPU
L2 decision crate
Output 300Hz
L2 decision crate
17
RunIIa L2 Muon path commission methodology at
work ? from discussion to error free path lt 3
months June ? Sept.
2003
upstream
downstream
Pulsar Muon Board
L1 Muon (16 hotlink fibers) and L1 track input
Legacy L2 decision crate
Data sink
Data source
Pulsar Receiver
Pulsar muontrack data transmitter

• Fully self-tested before put in the running exp.
• up to 1 Billions events in self-test mode.

We didnt waste one second of beam
time The FULL chain test with collision beam
works on the first try (error free)
18
Pulsar pre-processors
Pulsars (Tx)

Pulsar system-level testability
?developtune an upgrade standalone
SLINK
muon
PCI?SLINK
PC 0
SLINK?PCI
PC 3
merger
SLINK
PC 1
cluster
Data sources
PC 2
merger
Trigger Supervisor
electron
well on its way to upgrade the L2
SLINK
SLINK
L2toTS
merger
19
Well organized firmware effort is one of the keys
to have a system clean/easy-to-understand/robust
with minimal maintenance effort
DAQ buffers
VME responses
SRAM interface
Mezzanine card interface
CDF Ctrl interface


Filter algorithm
SLINK formatter
Play/record
Play/record
downstream
upstream
RAM/ Spy buffers
RAM/ Spy buffers
Modularity also for firmware design
Blocks in green common to all FPGA types,
blocks in pink data path specific. All
firmware (VHDL) in CVS follow software practice

20
Well organized firmware effort is one of the keys
to have a system clean/easy-to-understand/robust
with minimal maintenance effort
Common core firmware (library)
muon
VME responses SLINK formatter DAQ
buffers CDFCtrl responses SRAM ctrl Mezzanine
interfaces Diagnostic RAMs Spy buffers
SVT track
LVDS ext. FIFO
cluster
hotlink
L1
LVDS
Taxi
photon
TS
electron
Knowing one data path Pulsar ? knows all
Pulsars Data specific algorithm difference is
just minor details
dedicated people developing the core firmware
21
The project has attracted many people after
prototype success ? from few people last year
to now (core posdocs students)

• Related to CDF L2 decision crate upgrade
• ANL
• R. Blair, J. Dawson, B. Haberichter, J.
  Schlereth, J. Proudfoot
• FNAL
• R. Demaat, M. Hakala, R. Kivilahti, J. Lewis,
  C. Lin, T. Liu, T. Masikkala,
• F. Marjamaa, J. Patrick, S. Pitkanen, B.
  Reisert, P.Wilson
• Univ. of Chicago
• M. Bogdan, Y. Kim, W. Fedorko, H. Frisch, S.
  Kwang, V. Rusu, H. Sanders,
• M. Shochet
• Upenn
• K. Hahn, P. Keener, J. Kroll, C. Neu, F.
  Stabenau, R. Van Berg, P. Wittich
• Related to CDF SVT II upgrade (using Pulsar)
• INFN
• F. Spinella, L. Ristori
• LBL
• A. Cerri

SVT
again due to Pulsars flexibilities (to improve
SVT performance)
Note the project started as a project to build a
test-stand tool
22
Summary how we addressed the issues

• Uniformity modularity flexibility
• Lego-style, general purpose design,
  backward forward compatible.
• Many applications within outside CDF
• Plan to replace/upgrade gt 10 different
  types of CDF trigger board
• Compatible with S-LINK standard ? commodity
  processors
• Knowledge gained transferable to and from
  LHC community
• Design verification methodology
• simulation simulation single/multi-board/tr
  ace cross talk analysis
• ? no single design or layout error
  (blue wire) on all prototypes
• Testability commissioning strategy
• Board system level self-testability fully
  integrated in the design,
• Suitable to develop and tune an upgrade
  system in stand-alone mode
• Minimize impact on running experiment
  during commissioning phase
• Maintenance firmware version control
• Built-in maintenance capability Common
  firmware library (VHDL), CVS
• Documenting the work
• http//hep.uchicago.edu/thliu/projects/P
  ulsar/

23
Some backup slides
24
CAMAC/NIM days long gone Question How should we
(HEP) address the issues in the future??
25
Round-Trip timing
26
Round-Trip timing measured the performance
looks good

• w/ algo Blue w/o algos Red

From CERN
27
1.8GHz Linux
500MHz No OS
Tested with real L2 trigger data
28
Parts Availability for Level 2 Processors Its
been Doom for every CPU Thats ere been used in
Level 2. Both MotDec were once employed Their
product lines are now destroyed. Will the axe
next fall on Pentium, II ?
-- Bill Foster, 1998
29
SLINK interface mezzanine card
SLINK ATLAS SLINK data format
30
New 32-bit SLINK to 64 bit PCI interface card
S32PCI64

• highly autonomous data reception
• 32-bit SLINK, 64-bit PCI bus
• 33MHz and 66 MHz PCI clock speed
• up to 520MByte/s raw bandwidth

High-speed follow up of the Simple SLINK to PCI
interface card
31
New 32-bit SLINK to 64 bit PCI interface card
S32PCI64
? developed at CERN

• highly autonomous data reception
• 32-bit SLINK, 64-bit PCI bus
• 33MHz and 66 MHz PCI clock speed
• up to 520MByte/s raw bandwidth

Initial test results from CERN
High-speed follow up of the Simple SLINK to PCI
interface card
http//hsi.web.cern.ch/HSI/s-link/devices/s32pci64
/
32
CDF Front-End /Trigger sub-systems
FE 66 9U VME Crates
L1 15 9U VME Crates
L2 15 9U VME Crates
Trigger Supervisor 3 9U VME Crates
33
Pulsar design
3 Altera APEX 20K400_652 (BGA) FPGAs
502 user IO pins each
9U VME (VME and CDF ctrl signals are visible to
all three FPGAs)


P1
Data IO
Mezz card connectors
T S
T R K


P2

Control/ Merger
SRAM

128K x 36 bits

SLINK signal lines
Data IO
P3
T R K
L 1

spare lines

SRAM
3 APEX20K400 FPGAs on board 3 Million system
gates/80KB RAM per board 2 128K x 36 pipelined
SRAMs with No Bus Latency 1 MB SRAM (5ns access
time)

34
PULSAR design
Front-panel (double width)
Each mezzanine card can have up to 4
(hotlink/Taxi) fiber channels
LVDS connectors
Mezzanine cards
T S
Mezzanine Card
VME
optional user defined signal connection from P2/P3
L 1
IO

Hotlink/ Taxi/ S-LINK
Optional SVT input
L 1
Ctrl
S-LINK Tx or Rx
S V T
IO
SLINK
S-LINK Tx or Rx
S V T
spares
To/from Pulsar or a PC
Three FPGAs Atlera APEX20K400
component side
The mezzanine card connectors can be used either
for user I/O or SLINK cards
35
Pulsar approach Goal only build one type of
custom interface board and the rest are all
commercial products. Use mezzanine cards to
take care optical data paths (Cluster,Isolation,
Muon and Reces)
Pulsar design
T S I


Optical IO
P1
T R K

Control/ Merger

P2

Mezz card connectors


SLINK
Extra features L1 trigger bits and trk/svt input
are visible to all three FPGAs for flexibility
Optical IO
P3
T R K
L 1
36
L1 trk svt clist Iso reces mu
SumEt,MEt
Tracks

• What does Level 2 really do?
• Combine/matches trigger objects into e,muon
• Count objects above thresholds, or,
• Cut on kinematics quantities

Jets
electron
photon
muon
Taus

Met

SumEt

P1

Data IO
Mezz card connectors
T R K
T S


Control/ Merger
P2
user ctrl
Most trigger objects need L1 and track/svt
trigger information, this is reflected in
Pulsar design for flexibility




Data IO
P3
SLINK I/O
T R K
L 1

spare lines


37
Pulsar as Muon Pre-processor (interface
board)
data from Matchbox (16 hotlink fibers)
Can create physics objects (i.e., muons) at this
stage
Data IO


0-120 degree
P1
Info available L1, XTRP, Muon matchbox data
(0-240 degree)


120-240 degree
Control/ Merger
P2

ALL muon data available with Track info
SRAM


DataIO
SLINK signals
muons
240-360 degree

Info available L1, XTRP, Muon matchbox, Pre-match
box
data from pre-match box (4 hotlink fibers)
P3
T R K
L 1

spare lines

SRAM
One example to show design is driven by physics
requirement

38
Pulsar as Processor Controller
Pulsar design is driven by physics needs
possible to combine trigger objects here
(optional)
Data IO


Reces/trk
P1
Info available L1, SVT, XTRP, Reces, Cluster, Iso
T S

Cluster/Iso

Tracks, Jets, e, photon,tau etc
Control/ Merger
P2

SRAM
ALL Info available for L2 decisions,


DataIO
SLINK signal lines
Muon/trk

Info available L1, SVT,XTRP, Muon,Met, SumEt
P3
Tracks, muon, Met,SumEt etc
S V T
L 1
Met/SumEt

spare lines

SRAM

Board Level Flexibility