The Front-End Driver Card FEDv1

1
The Front-End Driver Card FEDv1

• VME crate requirements definition
• FEDv1 status
• Hardware
• Firmware low-level software
• FEDv1 testing plans

http//www.te.rl.ac.uk/esdg/cms-fed/index.html
2
FEDv1 CratesRequirements Document

• Requirements needed for Tracker Crate ordering
• Aim to have common VME64x bus crates for FED and
  FEC
• List of special needs, connectors, poweretc.
• E.g. FED may require Transition cards (DAQ links)
• References to LHC Crate technical specification
  EP document
• Released for comments

3
FEDv1 CratesRequirements Document
4
FEDv1 Overview
96 Tracker Opto Fibres
CERN Opto- Rx
Modularity 9U VME64x Form Factor Modularity
matches Opto Links 8 x Front-End
modules OptoRx/Digitisation/Cluster
Finding Back-End module / Event Builder VME
module / Configuration Power module Other
Interfaces TTC Clk / L1 / BX DAQ Fast
Readout Link TCS Busy Throttle VME Control
Monitoring JTAG Test Configuration
9U VME64x
Analogue/Digital
JTAG
FPGA Configuration Compact Flash
FE-FPGA Cluster Finder
VME Interface
VME-FPGA
BE-FPGA Event Builder
TCS
TTC
TTCrx
DAQ Interface
Buffers
Power DC-DC
Temp Monitor
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
5
FEDv1 Project History
96 Tracker Opto Fibres
CERN Opto- Rx
Analogue Circuit for signal match from OptoRx to
ADCs (good working relationship with
FrancoisJan) Fit 96 ADC channels on board FPGAs
choice of Xilinx Virtex-II family. Fitting FE
design into target part. But progress (always)
take longer than you think S-LINK Transition
card / Direct? Power VME modules took more
time than anticipated FPGA de-coupling schemes.
many minor design decisions. 1Q behind schedule
May 2002
9U VME64x
Analogue/Digital
JTAG
FPGA Configuration Compact Flash
FE-FPGA Cluster Finder
VME Interface
VME-FPGA
BE-FPGA Event Builder
TCS
TTC
TTCrx
DAQ Interface
Buffers
Power DC-DC
Temp Monitor
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
6
FEDv1 Board Layout Primary Side

• Complex board
• Pushing density limits
• Analogue Digital
• signal integrity

7
FED v1 Board Layout Secondary Side
Double sided
View thru the board
8
FEDv1 Overview
96 Tracker Opto Fibres
CERN Opto- Rx
Modularity 8 x Front-End modules
9U VME64x
Analogue/Digital
FE-FPGA Cluster Finder
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
9
FEDv1 Front-End module
Dual ADC 10-bits 40 MHz
OpAmp
CERN Opto Rx
Digital Processing
1
1
1
1
5
10
ASIC
Data Control
N
3
2
2
3
CLK
Each individual ADC clock skew is adjustable in
steps 1nsec
Delay FPGA
4
LVDS
CLK40 from TTC
3
5
2
12 Fibre Ribbon
6
Double Data Rate I/O
PD Array
DATA OUT _at_ 160 MHz
4
4
7
CLK
Delay FPGA
8
Full
Partially Full
5
9
3
RESET
10
6
11
CLK
DCM
Cluster Finding FPGA
CLOCK
Delay FPGA
12
DATA
Temp Sensor LM82
6
CONTROL
12x trim DAC
XC2V1500
XC2V40
AD9218
EL2140
10
FEDv1 Front-End module Primary Routed
OptoRx
ADCs
OpAmps
40K FPGAs
1500K FPGA
Note FPGA de-coupling
Dense circuitry
12 optical channels
11
FEDv1 Front-End module Secondary Routed
Double-sided
12
FEDv1 Full-Scale 9U Layout
96 Tracker Opto Fibres
CERN Opto- Rx
Back-End module For each L1-Trigger Collects 8
x FE variable length data fragments Formats FED
event for DAQ Appends TTC synch
information Buffers in External QDR SRAM Sends
data via DAQ Front-end Readout Link FRL Signals
to TCS Busy/Throttle
9U VME64x
Analogue/Digital
FE-FPGA Cluster Finder
BE-FPGA Event Builder
TCS
TTC
TTCrx
DAQ Interface
Buffers
PinDiode /Amp
ASIC
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
13
FEDv1 FED-DAQ Interface
Due to mechanical constraints place DAQ link card
on Transition card
Transition Card
TCS
Busy Throttle
BE-FPGA Event Builder
DAQ Mezzanine Card
TTC
TTCrx
DAQ Front-end Readout Link FRL
S-LINK64
Buffers
FRL DAQ links use S-LINK64 standard Implementation
Channel Link 800 MBytes/sec max Average DAQ
rate 200 MBytes/sec
14
FEDv1 FED-DAQ Interface (alternative)
TCS
BE-FPGA Event Builder
TTC
Alternative scheme Channel Link in Virtex-II
BE-FPGA
TTCrx
Buffers
DAQ
15
FEDv1 VME module
VME Interface Minimum Features for 2003 A32/D32
Slave access only Possible Slave BLT32? No DMA
engine No Interrupts Possible 64x Dynamic
Addressing? FPGA Configuration Xilinx System
ACE Compact Flash MPU-VME interface for in-situ
re-programming via Crate Controller (Not in
2003?) JTAG for Configuration Test
9U VME64x
JTAG
System ACE CF
VME Interface
VME-FPGA
FPGA Configuration
Xilinx Virtex-II FPGA
16
FEDv1 Power module
CERN Opto- Rx
Standard LHC spec crate supplies 3.3V, 5V,
12V Derive -5V, 1.5V, 2.5V on board Board
Estimate 80 W Hot Swap Controllers Electronic
fuses Protection against out of range voltage
current Sequence power Protections Over
temperature shutdown
9U VME64x
Analogue/Digital
Power DC-DC
Temp Monitor
Warning
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
17
FEDv1 Hardware Status
96 Tracker Opto Fibres
CERN Opto- Rx
9U VME64x
Analogue/Digital
JTAG
Board Status
FPGA Configuration
FE-FPGA Cluster Finder
VME Interface
VME-FPGA
BE-FPGA Event Builder
TCS
TTC
TTCrx
DAQ Interface
Buffers
Power DC-DC
Temp Monitor
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
18
FED Schedule
FEDv1 Full scale Prototype FEDv2
Pre-production FEDv3 Final production
Schedule 2002/Q4 2 x FEDv1 _at_ for UK test Batch
1 2003/Q4 10? x FEDv1 _at_ CERN Batch
2 2004/Q4 10? x FEDv2 manufacture 2005/Q2
500 x FEDv3 manufacture (funds permitting)
OptoRx added post-assembly. Procurement
started for critical parts for Batch 2
19
FED TIB Test Schedule
Q. Deliver 2 x FEDv1 Batch 1.5 _at_ CERN for TIB
test starting July 2003 ??

• Very tough. Testing is a big job. Cant promise
  to deliver in June?
• Use Batch1 PCBs. ie No design iteration possible.
• Risk that design has major fault.
• Prioritise essential firmware and testing plan.
• Require explicit list of functionality essential
  for the TIB assembly centre.

20
FEDv1Firmware Low-Level Software
21
FEDv1 Firmware Software
96 Tracker Opto Fibres
CERN Opto- Rx
9U VME64x
Digital Processing Xilinx Virtex-II FPGAs 40K-gt3M
gates 4 Species of FPGA Delay FPGA 40K Clock
skew ... FE FPGA 1.5M Cluster Finding ... BE
FPGA 2M Event Building ... VME FPGA 1M VME
Interface ... in general one designer per
species FPGAs programmed in both VHDL VERILOG
Analogue/Digital
JTAG
FPGA Configuration
FE-FPGA Cluster Finder
VME Interface
VME-FPGA
BE-FPGA Event Builder
TCS
TTC
TTCrx
DAQ Interface
Buffers
Power DC-DC
Temp Monitor
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
22
FEDv1 Front-End module
Dual ADC 10-bits 40 MHz
OpAmp
CERN Opto Rx
Digital Processing
1
1
1
1
5
10
ASIC
Data Control
N
3
2
2
3
CLK
Each individual ADC clock skew is adjustable in
steps 1nsec
Delay FPGA
4
LVDS
CLK40 from TTC
3
5
2
12 Fibre Ribbon
6
Double Data Rate I/O
PD Array
DATA OUT _at_ 160 MHz
4
4
7
CLK
Delay FPGA
8
Full
Partially Full
5
9
3
RESET
10
6
11
CLK
DCM
Cluster Finding FPGA
CLOCK
Delay FPGA
12
DATA
Temp Sensor LM82
6
CONTROL
12x trim DAC
XC2V1500
XC2V40
AD9218
EL2140
23
CMS Silicon Strip Tracker FED Front-End FPGA
Logic
1x
per adc channel phase compensation required to
bring data into step
Cluster Finding FPGA VERILOG Firmware
Clock 40 MHz
2x
DLL
4x
Synch in
Synch out
2 x 256 cycles
256 cycles
nx256x16
Synch
emulator in
trig2
trig3
trig4
trig1
Synch error
10
10
11
11
Re-order cm sub
s-data
16
10
8
Hit finding
16
Phase Registers
sync
ADC 1
d
Ped sub
d
DPM
Global reset
s-addr
8
hit
Sub resets
8
8
Sequencer-mux
a
a
No hits
Control
Full flags
8
averages
header
status
control
mux
4x
data
4
Packetiser
256 cycles
256 cycles
nx256x16
160 MHz
trig1
trig2
trig3
10
10
11
11
Re-order cm sub
s-data
16
10
Phase Registers
8
16
sync
Hit finding
d
Ped sub
d
DPM
ADC 12
s-addr
8
Serial I/O
hit
Sequencer-mux
8
8
a
a
Serial Int
No hits
8
header
status
averages
Temp Sensor
Local IO
Opto Rx
Delay Line
Raw Data mode, Scope mode, Test modes...
BScan
Config
24
FEDv1 Full-Scale 9U Layout
96 Tracker Opto Fibres
CERN Opto- Rx
Back-End module For each L1-Trigger Collects 8
x FE variable length data fragments Formats FED
event for DAQ Appends TTC synch
information Buffers in External QDR SRAM Sends
data via DAQ Front-end Readout Link FRL Signals
to TCS Busy/Throttle
9U VME64x
Analogue/Digital
FE-FPGA Cluster Finder
BE-FPGA Event Builder
TCS
TTC
TTCrx
DAQ Interface
Buffers
PinDiode /Amp
ASIC
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
25
FEDv1 Back-End FPGA Logic
x2
VME copy path to local buffer
BSCAN
1
Clock40
x4
DCM
x1
diagnostics
Circular Buffers
1
VME
8
Control
Header
64
64
Data
FRL to DAQ SLINK64
Reset
1
80 MHz
x8
APV hdrs
Frame_Syncs
FIFO
x8
DECODE CONTROL MONITOR
Readout_Syncs
Lengths
x8
18
Data Out
Monitor_Syncs
FIFO
160 MHz
2
FF/PF Flags
Bx,Ex
2 x QDR SRAM x2 burst
Lengths
FIFO
40 Mhz
R/W Address Generator
TTC Rx
20
Address
Em Hdr
TCS
160 MHz
FIFO
2 MBytes
L1 100 kHz
4
Data_stream 0
64
18
Data In
80 Mhz
4
160 MHz
Data_stream 7
160 MHz
26
FEDv1 Status
96 Tracker Opto Fibres
CERN Opto- Rx
9U VME64x
Board Status
Analogue/Digital
JTAG
FPGA Configuration
FE-FPGA Cluster Finder
VME Interface
VME-FPGA
BE-FPGA Event Builder
TCS
TTC
TTCrx
DAQ Interface
Buffers

• Design effort now re-directed to completing
  Firmware. Team is sufficient.
• Design is progressing well.

Power DC-DC
Temp Monitor
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
27
FEDv1 Module Testing Firmware Functionality
Essential
Non Essential

• VME interface A32/D32 only
• Raw Data events only (unpacked)
• Scope mode for timing
• TTC interface ?
• Individual ADC clock skews
• Hardware trigger throttle ?

• S-LINK
• TCS interface
• Cluster Finding Firmware
• In-situ Firmware updates
• Auto-Calibration
• VME64x dynamic addressing
• Board hot swapping

Q. What event rate needed for testing? Q. What
features? Do we need individual channel DAC
offsets ? (or is common OptoRx offset
sufficient) Note Firmware is tested
(simulation/dev boards) but real world often
has surprises.
28
FEDv1 Low-Level Software
fed task
fed task

• XDAQ Hardware Abstraction Layer
• we need to learn more but...
• fedlib C class API lowest level access
  structures functions for user programs (object
  contained in fed class?)
• fedlib calls HAL read/write functions
• Hide details/sequences from user.
• int fedlib_load_fe_peds(fe_mod, struct peds,..)
• Maybe additional layer above fedlib? (see
  Costass talk)
• Assume PC Linux HAL supported VME Interface card

fedlib
user space
29
FEDv1 Address Map

• Hidden from outside world
• But need this before fedlib API
• Address Map under construction. Implementation
  tightly coupled to Firmware.
• Probably reserve lt 1 MByte / FED

Event Memory
FPGA confign
TTCrx
FED local CSR
FE FPGA 7
FE FPGA 6
FE FPGA 5
FE FPGA 4
Pedestals Skew values ...etc
FE FPGA 3
FE FPGA 2
FE FPGA 1
FE FPGA 0
30
FEDv1 Parameter Loading

• Mechanism of FED setup hidden from User
• e.g. Loading FED parameters not trivial
• Write constants e.g. Peds to small local FPGA
  buffer. Limited FPGA memory.
• Send command to load to FE FPGA
• FPGA engine handles serial commands
• Optional readback
• Repeat for next block reusing same parameter area
• Constraints
• Cant load individual channels
• NB cant access constants during Run!

under development
Parameter Readback block
Parameter Load block
Command Status
31
FEDv1 Firmware Software
96 Tracker Opto Fibres
CERN Opto- Rx
9U VME64x
Digital Processing Flexible Digital Logic Xilinx
Virtex-II FPGAs 40K-gt3M gates some in pin
compatible packages Features Dual Ported Block
Rams Digital Clock Managers DCM Double Data Rate
I/O DDR Digitally Controlled Impedance
I/O Various I/O signal standards Debugging Logic
Analyser cores FPGAs programmed in VHDL
VERILOG
Analogue/Digital
JTAG
FPGA Configuration
FE-FPGA Cluster Finder
VME Interface
VME-FPGA
BE-FPGA Event Builder
TCS
TTC
TTCrx
DAQ Interface
Buffers
Power DC-DC
Temp Monitor
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
32
FEDv1 Database Requirements

• Per FED 25 K strips
• Per Strip 256 x 96
• Pedestals _at_ 10 bits
• Cluster Thresholds _at_ 2 x 8 bits
• Enable bits _at_ 1 bits
• Per APV 2 x 96
• Enable _at_ 2 bits
• Per ADC 1 x 96
• Skew _at_ 8 bits
• Frame Thresholds _at_ 5 bits
• Control _at_ 2 bits
• DACs _at_ 8 bits
• Per OptoRx 1 x 8
• Control _at_ 8 bits

Total 100 KBytes

• ...
• TTCrx settings
• FPGA configuration
• FED run settings
• FED ID

33
FEDv1 Event Readout

• Readout (via VME) Formatted Events identical? to
  those sent to S-LINK
• Load Parameters e.g. Peds, Skews
• Set run mode e.g. Raw Data, TTC clock
• Start Run and wait for Triggers/Frames
• Poll on Event Counter
• Get length of event
• (Request to) Readout (in chunks)
• Clear Event Repeat
• Check local status registers/counters
• i.e. similar to PMC (sorry)

DAQ Header
Tracker Header?
Formatted FED Data
DAQ Trailer
Event Formats Raw Data unprocessed. Scope
Mode. Tracker header?
34
FEDv1 Software Schedule

• Work has just started in tandem with Firmware
  design...
• These are still guesstimates and depend on the
  ease of Firmware implementation and the demands
  of testing
• Draft Memory Map internal (end November)
• Release Event Format specs (end December)
• Release draft fedlib API spec (basic calls/no
  code) (end January)
• Release draft operating instructions (end
  January)
• Release first version of fedlib (basic
  calls/code) (end March)
• more needed... DataBase model?
• Caveat Emptor It is possible/likely that changes
  will be necessary to these specs.

35
FEDv1 Comments

• In order to achieve minimum firmware software
  on TIB timescales require...
• Confirmation of functionality needed for TIB
  tests.
• Close co-operation between UK and Tracker DAQ
  group.
• Arrange small FEDv1 online meeting.
• Propose contact from UK to visit CERN for
  technology transfer.
• Keep systems h/w and s/w as close as possible
  (within UK) and between UK and CERN.
• Note We now have additional effort in UK for
  test software, database, etc.
• See Testing Talk

36
FEDv1 Testing
37
FEDv1 CratesComments

• 2 FEDv1s to test in UK starting December 2002
• FEDv1 is a complex board.
• Board evaluation will require considerable time
  and effort
• Originally aiming for Batch 2 delivery 10? FEDs
  to CERN Q4 2003
• Testing Design Hardware Optical/Analogue/Digital
  Firmware
• i.e. Boards arent ready one month after
  receiving them
• Problems will happen
• Need for special Test Firmware Software
• Common set-ups within UK and with CERN (see
  Costass talk)

38
FEDv1 Test Features
96 Tracker Opto Fibres
CERN Opto- Rx
JTAG Boundary Scan Digital integrity/connections
Electrical Analogue Integrity Tests pre
OptoRx Xilinx Chip Scope Integrated Logic
Analyser Cores Capture raw ADC data (pre VME
interface) Opto-Tests with IC Test
Board Special FPGA loads e.g. Pattern Generators
Additional Test Features Internal/External
Clocks External Triggers
9U VME64x
Analogue/Digital
JTAG
FPGA Configuration
FE-FPGA Cluster Finder
VME Interface
VME-FPGA
BE-FPGA Event Builder
TCS
TTC
TTCrx
DAQ Interface
Buffers
Power DC-DC
Temp Monitor
Front-End Modules x 8 Double-sided board
Xilinx Virtex-II FPGA
TCS Trigger Control System
39
FEDv1Test Plan Draft

• Test effort in UK shared
• RAL Electrical Digital
• IC Optical
• UK test team is sufficient
• Design vs Production testing
• Integrity vs Performance testing
• Hardware vs Firmware tests
• Originally foresaw 9 months design testing
  before Batch 2
• firmware still under development

Test Plan Draft
40
FEDv1Testing Schedule

• Preparation Crates/PSUs/Test Kits/Tools licenses
  (pre Boards)
• Pre Post Assembly tests (1 week)
• Basic Power Supply tests (1 week)
• Digital Integrity Boundary Scan JTAG (1 week)
• FPGA configuration test JTAG (1 week)
• Analogue Integrity Signal Gen/Test card Chip
  Scope/JTAG (4 weeks)
• (Pre VME, board default settings but requires
  Firmware for Clock set-up in all FPGAs)
• gt Assemble OptoRx Parallel Tests _at_ IC
• Opto Integrity tests at IC with IC Opto-Test
  card Chip Scope (4 weeks)
• Basic Digital tests CFlash FPGA configuration,
  Clocks TTCrx, QDRs (6 weeks)
• (Pre VME, Test card ChipScope , requires
  special test Firmware)
• VME Interface test PC VME Link (4 weeks)
• (Peek Poke gt Test software)
• Intermediate Digital Tests (Scope mode, Raw Data
  mode, Event readout) (8 weeks)
• Intermediate Analogue Tests (Pedestals, X-Talk,
  Step Response) (6 weeks)

41
FEDv1Testing Schedule 2003

• Enough tests done to assemble 2 more FEDv1 for
  TIB tests
• VME64x Dynamic addressing, MBLT64
• TTCrx advanced
• Temp sensors
• DAC offsets
• Serial ID EPROM
• In-situ CFlash programming
• TCS interface
• S-LINK interface
• Advanced Analogue Bandwidth, FFT, Signal/Noise
• Advanced Digital High rates
• and lot more before...
• ...Batch 2 go ahead. Design Iteration needed
  before Batch 2?

42
FEDv1FEDs for TIB Tests

• Minimum of 4 months testing from 2003 Jan-April
  Assemble Test May-June
• Optimistic scenario TIB FEDs deliver to CERN
  start July.
• (exchange second pair with UK FEDv1s?)
• Precludes design iteration.
• Assumes no major test problems. And OptoRx
  availability.
• Software API can be provided much earlier.

43
FEDv1Test Setups Software

• Test rig being set up at IC Crate, PC/Linux
  VME Interface
• Clone system at RAL
• Analogue Test piggyback card at RAL (1 -gt 12
  chans)
• Opto Test VME Card at IC (12 -gt 96 chans)
• Plus TTC systems, FEDKit
• XDAQ/HAL based fedlib library
• Dedicated UK Test software also using same
  fedlib library.
• Needed in UK in March.
• Firmware development continuing in parallel.
• Q. Suitable GUI for test software Qt ? Used
  LabView in past.

44
CMS Silicon Strip Tracker FED Crate Layout
1
21
2
FE 1
FE 2
FE 3
FE 4
DAQ
FE 5
TTC
100 KHz
FE 6
Throttle
FE 7
NN Synch
F-Bus
FE 8
B-Scan

• Crate Input Data Rate 50 Gbyte/s
• Crate Output Data Rate 1 GByte/s per percent
  hit occupancy

45
CMS Silicon Strip Tracker FED Counting Room
Layout (illustration)
FED
DAQ

• 40 K ADC Channels 10 Bit_at_40MHz
• Max Trigger Rate 100 kHz
• Input Rate 1.5 T Byte/s
• Output rate 25 Gbyte/s/

• 440 Boards 96 ADC/Board
• 24 Crates
• 8 Racks

4 TTC Partitions
46
CMS Silicon Strip Tracker FED Control
Monitoring
1
24
50K ADC Channels
VME SBC RTOS
FED CRATES
E-Net Switch
Tracker Control WS
FEC CRATES
TTC CRATES
D-BASE
DCS
DAQ
R/C