Mark Krasberg

1
Low Level Commissioning 2006/2007NSF
Annual Review,MadisonMark KrasbergMay 23rd,
2006
2
Oct 2005 London (applies this year)Commissioning
a DOMat The Testing DAQ Level
Question Test
Does the DOM pass communications? MOAT / QuadTool (software failed in 2005/2006 because of firmware problems)
Does the DOM function properly? STF
Does the DOMs Local Coincidence connections work? LCCHAIN.PY
Does the DOM calibrate its HV? DOMCAL
Does the DOM have reasonable rates? Multimon/TestDAQ
Does the DOM take data properly? TestDAQ (had to modify domhub-app to obtain stable data-taking mode)

3
Oct 2005 LondonMinimal Commissioning in
Water (T12 hours)

• As soon as possible after the drop

Task Action Length of Test
Leak-test All DOMs Plug in good QUADs 60 minutes
Turn on good DOMs two at a time Verify good DOM list, DOM positions, Record Temp, Pressure 5 minutes
Turn on all DOMs Check LC connections Check Noise Rates Do short QuadTool run Short TestDAQ run? 120 minutes
Turn off all DOMs Unplug all DOMs
4
Oct 2005 LondonExpanded Commissioning in
Water (T 12 hours)

• As soon as possible after the drop, in addition
  to Minimal Commissioning in water

Task Action Length of Test
Perform repeated leak checks
Monitor Noise rates Multimon As often as possible
Take special runs (flasher runs) Requires firmware, DOMCal, TestDAQ
5
Oct 2005 LondonCommissioning in Ice part 2 (T
2 weeks)

Task Action Length of Test
Upload firmware 30 minutes
Calibrate DOMs Run DOMCal 60 minutes
Short TestDAQ checkout Verify mechanics 30 minutes
Check Communications MOAT / QuadTool 14 hours
Check DOM function STF 1 hour
Automated TestDAQ checkout Run desired steering files 48 hours
Give hub to real DAQ if they want it
6
Oct 2005 LondonOne idea

• The final commissioning stage for each string (ie
  the bottom 12 DOMs)
• Perhaps we want to do this part of the
  commissioning in March/April all at the same
  time.
• Minimizes handoffs back and forth from real DAQ
  to TestDAQ to real DAQ.
• TestDAQ is used one last time special
  multi-string runs can be taken steering files
  can be generated for all DOMs to be used by real
  DAQ.

7
What happened in 2005/2006?

• 2005/2006 Low Level Commissioning Plan was
  followed very closely, whenever possible

8
Why operate the DOMs in water?

• Identify software problems and fix as early as
  possible (every year there are unexpected
  problems to deal with)
• Track DOM problems as early as possible
• Study triboluminescence
• Study hole freeze-in time (holes are gradually
  getting smaller)
• Take hole-ice data (ie while DOMs are in water)
• Data can be cross-correlated later with other
  South Pole tests (eg SuperDarn)
• Continuous data-taking means we have as much data
  as possible if there is a supernova etc

9
Problem Quads

• Leaking Quads
• 12/25 String 29 all quads leaked not used in
  water
• 01/04 String 39 none (quad 7 SJB problem) (turned
  on Jan 5)
• 01/08 String 38 leak-checked late (oscillations)
  (turned on Jan 19)
• 01/14 String 30 quads 4, 7 (quad 14 SJB prob)
  (turned on Jan 14)
• 01/18 String 40 quads 13 and 14 (turned on Jan
  20)
• 01/22 String 50 leak-checked late (damaged)
  (turned on Jan 30)
• 01/25 String 59 leak-checked late (swamped)
  (turned on Jan 31)
• 01/29 String 49 quads 6, 9 and 10 (turned on Jan
  30)
• Anomolies
• Any DOM which exhibited anomolous behavior was
  unplugged
• Several wire pairs had high currents and their
  quads were unplugged and periodically tested
• Communications Issues
• Encountered a cold temperature (cold reboot)
  problem
• Issue was a moving target as DOMs cooled and
  changed temperature

10
High Current DOMs

• Six high current wire pairs identified at pole
• Four of the high currents mysteriously went away
  (seemingly not related to temperature or
  freeze-in). One took weeks to go away!
• Two remaining high current pairs permanently
  plugged in at station close
• DOMs on both of these pairs have suffered
  problems in the months since they were plugged in
• In the future, high current pairs should not be
  used in normal data-taking for as long as is
  feasible (one year?)

11
How did the London plan go?

• The London plan was followed VERY closely
• Almost all elements of the plan were followed
• More data was actually taken than the initial
  plan called for, mainly because of the
  flexibility of the TestingDAQ and the fact that
  we have used hundreds of DOMs in water, all
  successfully.
• How was the plan not followed?
• QuadTool/MOAT (comms tests) did not produce
  useful results (firmware issues stemming from the
  cold temperature problem)
• realDAQ was very eager to get strings so they
  were shared, prior to low-level commissioning
  tasks being completed.
• Not all strings were hooked up immediately after
  deployment, so commissioning tasks were delayed
• All strings which were hooked up were
  continuously monitored, where possible.
• We ended up not repeatedly leak-checking the DOMs
  after they passed leak-checks
• Too much work to do this
• Repeated leak-checks damaged MS connectors
• It is believed that no DOM has ever started
  leaking after it passed the leak-test

12
String 39 two-week freeze-in movie
13
Triboluminescence

• Had a longstanding plan on how to cope with the
  triboluminescence (gt40 kHz individual DOM
  rates!)
• Use LC up.AND.down (Triple Local Coincidence)
• Use Scaler Deadtime (reduces LC trigger rate)
• The plan worked!
• able to take over 1 terrabyte of data during
  season and hand-carry north

14
Data-taking during Triboluminescence
Approximate maximum trigger rate for normal
TestDAQ data-taking
frozen-in LC rate
15
Surface DAQ Hardware

• Jan 15 string 21 hub stopped working, probably
  overheated (two other hubs reported overtemp
  alarms only overtemp hubs were ones which had
  strings hooked up to them).
• Did not operate all deployed strings
  simultaneously after this event until the power
  distribution and cooling fan problems in the TCH
  were solved
• Jan 30 string 59 DSB card didnt work was
  replaced
• String 59 DOMHub suffers from domhub-app JVM
  crashes. This is the only hub which has had more
  than one JVM crash with TestDAQ. RealDAQ has had
  problems with other hubs. Speculation was that
  the memory on this hub was bad (turned out to be
  true).

16
Cold Temperature Problems

• DOMs found to have temperature-related problems
  (cold reboot problem)
• Problems occur primarily at top of string
  (colder)
• THIS MADE IT IMPOSSIBLE TO COMMISSION ANY DOMs at
  Pole
• communications problems were a moving target
• the list of DOMs which would not communicate
  properly would change daily, as the DOMs cooled
  down.
• Most Problemmatic temperature was -28C

17
Able to deal with the expected unexpected
problems fairly easily(part of the plan!)

• When we hook up DOMs in a new environment we
  often encounter problems
• Each DOMHub had a full TestDAQ development
  workspace (ie all the code needed for data-taking
  could be modified and recompiled at pole).
• Patches could be emailed through Iridium
  satellite
• Patches could then be installed, recompiled, and
  tested within minutes
• Feedback could be given to the north almost
  instantaneously using Iridium satellite we did
  not have to wait for TDRS satellite. If we had,
  then the resulting code iterations would have
  gone on for days!

18
Communications stabilized on March 8

• TestDAQ softboot problem (a cold temp problem)
  eliminated
• Data taking became stable with TestDAQ as of
  March 8th (600 DOMs at that time) except for
  occasional string59 domhub-app crashes.
• DOMs could be commissioned since the
  communications problems were eliminated and
  data-taking was stable.

19
DOM Date Noticed (redpost turnon) Dead Serious Prob Can Work Misc Comments
29-59 Jan 1 X Auroraphobia short (in water)
29-60 Jan 1 X Nix short (in water)
38-59 Jan 21 X Blackberry HV went bad soon after turn-on (in water)
38-60 SLC Needs SLC because of Blackberry
30-23 Jan 27 X Peugeot_505 dead, post-refreeze
50-36 Jan 29 X Ocelot dead, post-refreeze
50-58 Jan 29 SLC Universitet (high current) LC broken.
49-51 Feb 1 X Pressure readback fails for Orzo
59-51--59-52 Feb 5 X LC between Medborgarplatsen and T_Centralen broke
40-51 Mid-Feb X Juneberry (high current) COMM issues got worse!
40-52 Mid-Feb X Alfa_Romeo_Spider (high current) COMM issues got worse!
39-08 Mar 10 SLC Shamal (high current) developed LC problem
49-17 March 14 X Flash chip problem for Biometeorology
59-59 before March 14 X Flash chip problem for Cosmology
29-18 Mar 10 X Yurei suffers 40 gain shift between different DOMCal runs
39-22 Noticed recently X Liljeholmen has spurious ATWD waveforms
29-07 Noticed recently X Koinoniphobia fails an FADC STF test

20
40-51 Juneberry 40-52 Alfa_Romeo_Spider
poor comms
50-58 Universitet bad LC
39-08 Shamal bad LC
59-59 Cosmology poor flash
38-59 Blackberry no HV
29-18 Yurei PMT?
49-17 Biometeorology bad flash
30-23 Peugeot_505 no power
50-36 Ocelot no power
29-59 Auroraphobia 29-60 Nix no power
21
There are currently 16 strings (not 9)!
hole 21 29 30 38
39 40 49 50 59 strings
1 1 2 1 2 2
2 3 2 total16
we had 17 mini-strings in UW FAT 6!
broken LC connection between two DOMs
Problem DOMs
a very short string!
cannot be used for hard LC
22
String 49 drill stopped for one
hour at 2300 meters
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
Summary of Commissioned DOMs

• To date the 528/540 InIce DOMs commissioned
• DOMs with serious problems
• Three communicating DOMs not useful for hard LC
  data-taking (this affects 4th DOM)
• 2 DOMs (40-51 Juneberry and 40-52
  Alfa_Romeo_Spider) are high current, poorly
  communicating, have recently degraded and have
  been unplugged
• 4 DOMs do not power up
• 1 DOM failed FADC STF test (marginal failure?)
• DOMs with less serious problems
• 2 DOMs (49-17 Biometeorology and 59-59
  Cosmology) have flash problems
• DOMs which have problems but pass commissioning
• 29-18 Yurei suffered a 40 gain shift in March
• 21-30 Phenol has sustained high dark rate
  problem, operating at lower gain
• 21-13 Wickueler does not like to be turned off
  (discovered during this season)
• Additional LC problem between 59-51 and 59-52
  (broken connection during refreeze)

27
49-55 Fusilli
28
(No Transcript)
29
(No Transcript)
30
How will low-level commissioning change this year?

• Commissioning of old strings
• after they are moved from TCH to ICL
• Estimate 48 hours to do this
• Commissioning of new strings
• Estimate 3 weeks per string, from time of
  deployment
• Want to hand off entire strings this year instead
  of piecemeal
• Triboluminescence must have finished so rates are
  reasonable for realDAQ
• Will not hold up entire strings because of a
  small number of DOMs like 49-55 Fusilli
• Will have a late cleanup in April/May, like
  this year
• Will develop a string traveller to make it
  obvious what has and has not been done for each
  string, and what problems exist