Summary for Working Group of Operations, Reliability, Injection, and Instrumentation

1
Summary for Working Group of Operations,
Reliability, Injection, and Instrumentation

• Michael Billing

Roger Erickson Alan Fisher Shigeki Fukuda
Eiji Kikutani Mario Serio Jim Turner
Fuhua Wang Uli Wienands
Associates
2
Motivation
Expectation is when a collider achieves Factory
or Super-Factory status ( L 1033 -1036 cgs ),
it will be able to operate repeatedly at that
level.

• To achieve this goal we must concern ourselves
  with
• Efficient Operations
• Reliable Hardware and Accelerator Conditions
• Ability to Detect Faults and Recover
• A Toolbox of Instrumentation and Supporting
  Analysis Software
• (Since the Collider Must Have Beams to Operate),
  Rapid Injection

3
Modus Operandi for This Working Group

• Drafted a list of questions under the preceding 5
  areas
• Discussed the questions
• Shared experiences from different laboratories
• Series of Talks
• Michael Billing Operating experience with CESR
• Eiji Kikutani Fill pattern control system of KEKB
• Roger Erickson Operational reliability of PEP-II
• Alan Fisher Developing the tune tracker for
  PEP-II
• Uli Wienands Trickle (continuous) injection
  issues with PEP-II
• Shigeki Fukuda KEKB-Linac upgrade plan using
  C-Band system for Super KEKB
• Dmitry Teytelman Fault analysis for PEP-II RF
• Uli Wienands Online lattice models and beam
  Jim Turner measurements

With RF Fdbk Group
4
Operations Scheduling

• Optimum HEP Data-Taking Time
• Function of Filling Time, Luminosity Lifetime
• Determined for Max Integrated Luminosity
• With slight variations all labs arrive same
  answer
• Typical Weekly Machine Studies Periods
  (shifts/week)
• PEP-II 0.5 (desire for 2-3)
• KEKB 2/2 weeks
• CESR-C 4-6
• Regularly scheduled Access Periods (Downtimes)
• PEP-II scheduled only as needed
• CESR-C 1 shift/week
• KEKB 1-4 shifts/2 weeks

5
Operations Personnel
KEKB PEP-II CESR-C
Director of Operations 1 1 1
Program Deputy 1 1
Operations Deputy 1 1
Personnel in the Control Room Personnel in the Control Room Personnel in the Control Room Personnel in the Control Room
Accelerator Physicist 1-2 1
Shift Leader 1 1
Operators 2 2 1

• Communications Operations Meetings
• PEP-II 1 Program Meeting 1 Fault Reporting
  Meeting/day
• KEKB 1 Program/Fault Reporting Meeting / day
• CESR-C 1 Program/Fault Reporting Meeting / week

6
Fault Recovery

• Off-Hours Repairs
• Control Room Personnel initiate /or support
  repairs
• Technical Personnel (e.g. Magnet technician)
  on-site at SLAC around the clock but not at KEK
  or CESR-C
• Specialists are On-Call at all times
• Escalation Policy
• After accelerators down more than 2 hours,
  higher level person is notified to coordinate the
  activities
• Person notified is
• Hardware Group Coordinator - PEP-II
• Operations Deputy - KEKB
• Director of Operations - CESR-C

7
Fault Recovery

• Faults are Logged
• All Labs have some form of Electronic Logbook
• Some Labs use paper Logbooks also
• Electronic Logbooks (e.g _at_ CESR-C)
• Permit automatic mail distribution of logged
  fault entries
• Semi-automatic logging of Beam Loss information
• Includes x, y, z FFTs of position signals
• Recording of which element tripped First
• Fault record file designations
• Other comments
• Semi-automatic characterization of conditions
  (KEKB CESR-C) (lt1 hour/week)
• Tunes, orbits, dispersions, betatron phase
  coupling, chromaticities, hardware status, et al
• Incredibly useful when recovering from faults

8
Fault Recovery Miscellaneous

• Spares
• Need to have parts on hand
• May need to guess buy a lifetime supply of
  parts
• Buy extra spares for hard-to-find parts
• Need to do last 2 more often !
• Safety Protection System
• Need checking typically 2/year by regulation
• Typical system checkout times (wide variation)
• KEKB 4 hours
• CESR 4 hours CHESS 4 hours
• PEP-II 6-8 shifts

9
Fault Prevention Reliability

• Importance of Record Keeping (Roger Erickson)
• All labs account for time usage
• For failures causing unscheduled downtime
• Account system-by-system
• Track Integrated Lost Time
• Track Mean-Time-To-Failure
• Track Mean-Time-To-Repair
• Indicate systems needing attention
• Reliability
• Obvious problems dealt with
• Repair of low priority items
• Repairs not made -gt hurt reliability or
  diagnostics???
• When do we upgrade OLD systems? - Cause headaches
• Document MTTF, MTTR, Cost of Repairs, downtime to
  decide

10
Fault Detection

• Avoid Information OVERLOAD - Many Faults
  Always Displayed
• Make Prudent Choice
• Fix even the low priority problems OR
• At least temporarily veto false indicators OR
• Raise Trip/Warning level
• Operator Training to detect cause of faults is
  important
• Specialists still needed to analyze complex
  recorded failure records
• MGB Future, even more complex accelerators will
  find many subtle ways to fail, the accelerator
  community MUST find a method for automatic
  analysis of recorded waveforms.
• At least automatically find the common failure
  modes
• Could use techniques from HEP?

11
Instrumentation Transient Fault Detection

• Triggered counters (KEKB CESR-C)
• Multiple cards with multiple inputs
• Detect delays between fault indications from many
  systems
• Need to empirically determine hardware/cable
  delays
• Identify correct source of failure large fraction
  of the time (but not always)
• Transient Recorders (All labs)
• 10 MSamp/sec ADC
• 1-several K samples in memory
• Signals such as
• x, y positions currents for beams
• RF waveforms Fwd, Rfl Power, Phases, Tuning
  Angle
• Abort kicker or separator voltages
• Feedback system power

CESR-C calls these Chicken Egg cards
12
Instrumentation Transient Recorders

• Dimitry Teytelman presented a number of analyzed
  transient signals from PEP-II (unfortunately
  unable to retrieve some to show)
• He has run several training sessions for
  operations personnel - teaches how to interpret
  waveforms - important for accurate diagnosing of
  problems
• CESR - Transient data gt Horizontal separator trip

13
Instrumentation Transient Recorders
CESR Beam Loss record W2 RF Waveguide Arc Trip
N.B. Beam Induced voltage in field
reflected power
14
Instrumentation Slow Transient Recoding

• Experience at CESR-C At times need to record
  for longer time
• 100-200 signals at 10 Hz rate
• Write data in memory to file on trigger
• Can see vacuum burst propagation
• Useful for slow transients

15
Instrumentation Diagnostics for Conditions

• Standard Toolbox Measure
• Orbits, tunes, dispersions, beta-functions (via
  changes in quadrupoles or phase measurements),
  coupling (global local)
• Feedback Systems Also Provide Diagnostics( John
  Fox)
• Grow-dampmeasurements
• DSP data recorder
• Bunch-by-bunchtunes, currents,etc.

16
Instrumentation Diagnostics for Conditions

• Beta Measurement from Phase Advance Data (Uli
  Wienands, Jim Turner)
• Drive beam with shaker
• Measure phase between BPMs
• Infer Betas
• Method adaptable to local
• coupling measure-
• ments also
• Able to repair conditions

17
Instrumentqtion During Collision

• Tune tracker (Alan Fisher)
• Uses small excitation of non-colliding bunch
• Lockin amplifier
• Determine tune by phase characteristics -gt
• Can correct tune change during collisions

Test perturbations
Feedback on
18
Instrumentation Philosophical View

• Automating machine controls adding feedback
  systems
• Long term effort to improve operational behavior
• Observational Toolbox Analysis Toolbox
• These Tools allow
• Measuring correcting accelerator errors
• Discovering or confirming intensity dependent
  behavior
• Measuring improving luminosity
• Determining maintaining injection performance
• Like any other tool in the workshop, these tools
  need maintenance - cost is time effort e.g.
• Calibrate BPM offsets
• Test software group controls
• Generate test cases for analysis tools
• Test accelerator model

19
Injection

• Routine injection mode is to Top-Off the beams
  (all labs)
• Tuning to keep detector backgrounds low is
  critical
• Tune Injector to keep good beam
• Reduce any injection errors (x, y, E), use
  collimators
• Orbits tend to drift
• Could standardize magnets more often BUT
• Tends to take a long time
• PEP-II (45min), KEB (12 min), CESR-C (4 min)
• Could save separate orbit for injection
• HEP Detectors are protected
• Can reduce REP rate
• Can dump beam
• Over Filling protection
• Dump beam if bunch significantly overfilled
  (PEP-II)
• Stop filling if DCCT / bunch current monitors
  (CESR-C)

20
Injection KEK-B Fill Pattern Generation

• Method for filling a variety of bunch patterns
• Requires 3 processors to communicate
• Uniform filling top-off filling (BCE) modes
• Good human interface
• Similar to those at PEP-II CESR-C

Eiji Kikutani
21
Super-KEKB Linac Upgrade
Shigeki Fukuda

• Examined a number of upgrade paths
• Some major changes could be
• Raising e energy 3.5 -gt 8 GeV
• Adding damping ring(s) for smaller emittance
• Increasing charge
• Accelerate e e- simultaneously
• Major progress
• Designed 5.7 GHz RF structure
• Procured RF source modulator for tests
• Successfully accelerated beam - 41 MV/m

22
Continuous, Trickle Injection for PEP-II
Uli Wienands

• Top-Off bunches continuously
• Low REP Rate (3Hz)
• Small charge
• BABAR
• Counts triggers for EM Calorimeter
• Provides Beam Loss
• diagnostics
• Gates off /-300 nsec about injected bunch
• Tests quite successful already see 12 increase
  in int-Lum
• Also has been tried at KEKB
• Process works
• Present current limit is beam induced heating