L2 Status and Plan

1
L2 Status and Plan

• Matt Worcester, Heather Ray, Monica Tecchio,
• Myron Campbell, Jane Nachtman, David Saltzberg,
  Tom Wright, Steve Kuhlmann, Karen Byrum, Masa
  Tanaka, Fred Neill, Peter Wittich, Nathan
  Goldschmidt, Ted Liu, Alexei Varganov, John
  Dawson, Bill Haberichter, Greg Feild, Stephen
  Miller
• December 7, 2001
• L2 Review

2
L2 Status

• L2 WORKS!

3
L2 Status

• Current L2 Crate Configuration
• 1 Alpha
• L1
• Clist
• Isolist
• Tracklist
• Svtlist
• Reces(x4)

4
L2 Status

• Current L2 Operations
• 1 Alpha, L1, Clist, SVTlist, Tracklist, Isolist,
  Reces(w/o readout)
• Currently used for data taking with L2 cutting on
  jets
• Tested at 20KHz with cosmics test
• All the interface boards work without sync and
  data errors.

5
L2 Goals and Plans
Slide From L2 Workshop 9/7/01

• Stephen Miller
• September 7, 2001
• L2 Workshop

6
Goals Before Shutdown
Slide From L2 Workshop 9/7/01

• L2 Testing Milestones
• Run cutting on Jets at L2
• Using Clist and L1 board run at 2-20Khz for 10M
  L1A
• Problems to fix Clist sync error. L1 bits error
• Run cutting on SVT track information
• Using SVTlist and L1 board run at 2-20Khz for 10M
  L1A
• Problems to fix L1 bits error. SVTlist dstrobe
  error
• Other work Correct d0. Write code for simple
  trigger.
• Run cutting on Electrons
• Using Clist, Tracklist, L1
• Problems to fix L1 bits error. Clist sync error.
  Clist-Tracklist collisions.
• Other work Finish electron code.

DONE
DONE
ALMOST DONE
7
Goals Before Shutdown
Slide From L2 Workshop 9/7/01

• L2 Testing Milestones cont..
• Run with L1, Clist, SVTlist, Tracklist with high
  rate L1A
• Main goal is to check data integrity.
• Test of Reces data
• Low rate test with beam
• Useful for understanding SMXR trigger data
• Ideally Would have Alpha-VME errors fixed for
  high rate test
• Test with Clist and Isolist
• Need test with beam to compare Isolist and Clist
  data
• Test with all interface boards
• Only consider this if all other problems have
  been fixed
• Would first do the test with Cosmics.

DONE
DONE
DONE (after shutdown)
DONE (after shutdown)
8
Goals Before Shutdown
Slide From L2 Workshop 9/7/01

• L2 Trigger/Physics Milestones
• Cutting on Jets
• Just Clist and L1 boards in system
• Point is to improve data quality at higher
  Luminosity
• Cutting on SVT track information
• Without Clist? And no cutting on Jets?
• Is there a target sample size useful for Physics
  groups?
• Cutting on Jets, Electrons and SVT tracks
• Ideal goal before shutdown
• Work to do
• Establish stable L2 operations
• Have error handling code in L2 to quickly find
  errors

DONE
SVX errors Required tagging mode
DONE
9
Plans during shutdown
Slide From L2 Workshop 9/7/01

• Magic Bus
• Install new Magic Bus
• Change termination
• Study Magic Bus collisions with cosmics
• Need to remove firmware workarounds to create
  collisions
• Alpha Tests
• Add additional processors
• First test with subset of interface boards
• Move to full crate test using cosmics
• Other
• Finish debugging known problems

Work in Progress
DONE
DONE
DONE
DONE Major Improvements in Clustering Crates
10
L2 Status

• Fixed Problems
• Enormous improvement for all boards since the L2
  Review in Sept.
• List of solved problems too long to cover
• Only known problems are listed below
• Unresolved issues
• Errors from Clustering Crate
• Appearance of single bit energy errors fixed
  with jumpers on backplane
• Isolation boards have 1 eta errors from
  Clustering Crates
• Error reproduced in teststand caused by Clique
  board
• (Worst Case) Can find error in Alpha code by
  comparing to Clist
• Gives 1 over efficiency for isolation triggers
• Major crate issue is Magic Bus arbitration (work
  in progress)
• Important for achieving design rate, not for
  operation at 20KHz
• Need to get reliable Reces readout
• Really a problem with Alpha fpga firmware
• Muon board is only missing interface board

11
Software Status

• Current Software Status
• Alpha executable used for making L2 decision
• Has been running and sending L2A since beam
  started
• Jet cutting algorithm is used for data taking. L2
  autoaccepts used for other triggers
• SVT based trigger tested. Electron and SVT
  triggers are written, but need testing
• Monitoring software will create error condition
  if interface boards fail to send data or get out
  of sync
• Alpha code organization
• Alpha software stored in online CVS repository
• Executable is generated automatically using cuts
  from the TriggerDB

12
Software Status cont..

• Current Software Status
• People working on Alpha code
• Peter Wittich, Masa Tanaka, Heather Ray, Stephen
  Miller, Tom Wright
• Starting a L2 software meeting to focus on L2
  code writing.
• Monitoring software
• TrigMon software used online by shift crew.
  Offline for analyzing data
• Has code for all interface boards
• L3 filter to check for data corruption in TL2D
• Uses simulation of trigger for exact comparison
  of TL2D contents with expected
• (Code will become basis for offline L2 trigger
  simulation)

13
Short Term L2 Plan

• Maintain stable L2 operation with current
  configuration
• Proposed L2 Crate Configuration for mid-January
• 3 or 4 Alphas, L1, Clist, SVTlist, Tracklist,
  Isolist, Reces(with readout) running at 20KHz
  L1A rate
• Rate well matched to expected luminosity
• 3 Alphas have been tested in crate in cosmic test
  at 20 KHz
• Need software work to ensure robust running with
  multiple alphas
• Need to process triggers in multiple alphas
• Reces readout problem can likely be fixed with
  Alpha firmware change
• Will be able to work on problem in teststand at
  Michigan

14
Short Term L2 Plan cont..

• Software expectations for mid-January
• Jet, SVT and Electron (with Isolation) triggers
  written and tested with beam
• Reces based triggers ready by end of January
• Improved tools for code development and testing
• Masa wrote tool for testing triggers with TL2D
  data
• Most of CDF4718 will be in place by end of
  January
• Will start optimization of triggers with high L1A
  rates

15
L2 Software and CDF4718
L2 options written are highlighted in Blue
Many still need to be tested
16
L2 Software and CDF4718 cont..
17
L2 Tests until mid-January

• Multiple Alpha tests
• Reces Readout tests
• Work on Low level errors in clustering system
• Tests of new trigger algorithms
• Test of new alpha firmware
• Tests of spare boards
• Tests with SVT-SVX (not L2 specific)
• High Rate tests with beam (change L1 prescales)
  CDF test
• Tests initially done without beam, but may
  request beam time for final verification

18
Path to Higher Rate

• Defining the Problem
• Current L2 crate runs at only 20KHz L1A rate
• This rate should be sufficient for Luminosities lt
  5x1031
• Rate limit due to delays for interface boards to
  send data on the backplane
• SVT board has 34us delay. (Isolation has 50us
  delay but doesnt always send data)
• Some delay due to overhead in code for
  controlling crate, interfacing with TS
• For higher luminosities, expectation is 50KHz L1A
  rate
• Ideal is 10us for loading data, 10us for
  processing event with both happening at same
  time.
• Solution Fix Magic Bus Arbitration, Improve
  Alpha firmware, Optimize Trigger Code, Run with 4
  Alphas
• Magic Bus arbitration
• Noise on arbitration lines causes interface
  boards to take the bus at same time
• Currently use time delays for each board to send
  data. Gives limit of 20KHz
• Delays for boards chosen arbitrarily. Possibility
  of reducing delays.
• New MagicBus is being tested at Michigan and
  lower crate
• Will explore other arbitration options if needed

19
Path to Higher Rate

• Improve Alpha firmware
• All control currently done in software- gives
  5-10us overhead per event
• Some improvements possible with optimizing some
  code
• New firmware written to handle control signals.
  Will give 1-2 us of overhead
• Partially tested at Fermilab. Need to verify TS
  handshake test.
• New code to match firmware has been written.
  Needs debugging.
• Additional firmware changes could reduce overhead
  to lt 1 us
• Software would just check for data having been
  loaded and sending decision to TS.
• Trigger code optimization
• Plan is to first write reliable code and then
  improve performance
• Timing studies can be done in lower crate
• Assume all improvements will be studied/solved
  independent of beam
• Tests done in teststands, lower crate, or upper
  crate with cosmics
• Ideally test new code/configuration in last hour
  of a store.
• Only after extensive testing with cosmics.
• Would use lt 1 of luminosity for tests

20
Plan for Muon Board

• Muon board being designed by Fred Neill at
  Michigan
• Muon data fpga design done and mezzanine board
  built
• 9U board layout done
• MagicBus interface fpga under design (Tom Wright)
• Plans for testing
• Firsts tests at Michigan with testclock, Muon
  trigger test card, Alpha for testing board
• Cosmic ray tests at B0
• Test at high rate standalone
• Test with full L2 crate
• No beam time will be requested for debugging
  problems
• Testing to be done by Fred, Eric, Tom, Stephen

21
Plan for Operations

• Requirements for Reliable L2 operations
• L2 monitoring
• Check of TrigMon summary.
• Running validation code on all events from
  several runs a week (or validation as part of
  production).
• Keeping up L2 with Beam web page describes
  crate performance for each run.
• L2 pager coverage
• Response to errors found by shift crew.
  Instructing aces on any special L2 requirements.
  Reporting status at 8am meeting.
• Currently handled full time by Heather Ray.
  Assembling list of names for additional
  personnel. Other people will be in place for
  start of physics data taking.
• Spares
• At least 1 spare exists for each board. Require
  at least 2 spares per board. Need to finish
  assembly and testing of some spares.
• Each institution must have an expert for repair
  of its boards

22
Plan for Operations cont.

• Requirements for Reliable L2 operations cont
• Documentation
• Documentation being collected and organized on
  web pages
• Schematics and some written documentation exists
  for all boards
• General non-expert document has been written
• Expert documentation/web page under development
• Proposal
• Monitoring and pager coverage can be combined for
  L1 and L2 trigger
• Much of monitoring and validation code is already
  combined
• Helps in distinguishing L1 and L2 problems
• Adds additional manpower to both groups and
  extends expertise across systems

23
Summary

• L2 works!
• Able to run at 20KHz with a full crate
• Used for data taking with cutting on jets
• Major improvement since September workshop
• Many errors have been fixed. (Most not mentioned
  in this talk)
• Short Term Plan
• Maintain reliable L2 operation
• Work on remaining issues needed to improve
  performance
• Multiple Alphas
• Reading Reces
• Higher Rates Magic Bus Arbitration
• Writing and testing trigger algorithms for
  Physics
• Team of people and tools are in place and being
  used.
• Longer Term Plan
• L2 debugging and improvements will continue
  without using beam
• Assemble group of people and knowledge base for
  maintaining L2 trigger