Remote observing with the Keck Telescopes from multiple sites in California

1
Remote observing with the Keck Telescopes from
multiple sites in California

• Robert Kibrick, Brian Hayes, Steve Allen
• University of California Observatories / Lick
  Observatory
  
• Al Conrad
• W.M. Keck Observatory
• Advanced Global Communications Technologies for
  Astronomy II

2
Overview of Presentation

• Background
• The Keck Telescopes and telescope scheduling
• Remote observing with the Keck Telescopes
• From remote control room at summit (30 meters)
• From Keck Headquarters in Waimea, Hawaii (32 km)
• From Santa Cruz, California via Internet2 (3200
  km)
  
• Network Reliability Concerns
• Providing a backup data path
• Recent operational experience
• Extending the model to multiple sites

3
The Keck Telescopes
4
Keck Telescopes use Classical Scheduling

• Kecks not designed for queue scheduling
• Schedules cover a semester (6 months)
• Approved proposals get 1 or more runs
• Each run is between 0.5 to 5 nights long
• Gaps between runs vary from days to months
• Half of all runs are either 0.5 night or 1 night
  long

5
From 1993 to 1995, all Keck observing was done at
the summit

• Observers at the summit work from control rooms
  located adjacent to the telescope domes

6
Conducting observations involves coordinated
effort by 3 groups

• Telescope operator (observing assistant)
• Responsible for telescope safety operation
• Keck employee normally works at summit
• Instrument scientist
• Expert in operation of specific instruments
• Keck employee works at summit or Waimea
• Observers
• Select objects and conduct observations
• Employed by Caltech, UC, NASA, UH, or other

7
Keck 2 Control Room at the Mauna Kea Summit

• Telescope operator, instrument scientist, and
  observers work side by side, each at their own
  computer.

8
Observing at the Mauna Kea summit is both
difficult and risky

• Oxygen is only 60 of that at sea level
• Lack of oxygen reduces alertness
• Observing efficiency significantly impaired
• Altitude sickness afflicts some observers
• Some are not even permitted on summit
• Pregnant women
• Those with heart or lung problems

9
Initiative to support remote observing from Keck
Headquarters

• 1995 Remote control rooms built at Keck HQ
• Initial tests via 1.5 Mbps (T1) link to the
  summit
  
• 1996 Videoconferencing connects both sites
• Remote observing with Keck 1 begins
• 1997 50 of Keck 1 observing done remotely
• Link to the summit upgraded to 45 Mbps (DS3)
• 1999 remote observing 90 for Keck 1 and 2
• 2000 remote observing now the default mode

10
The Remote Observing Facility at Keck
Headquarters in Waimea

• Elevation of Waimea is 800 meters
• Adequate oxygen for alertness
• Waimea is 32 km NW of Mauna Kea
• 45 Mbps fiber optic link connects 2 sites
• A remote control room for each telescope
• Videoconferencing for each telescope
• On-site dormitories for daytime sleeping

11
Keck 2 Remote Control Room at the Keck
Headquarters in Waimea

• Observer and instrument scientist in Waimea use
  video conferencing system to interact with
  telescope operator at the summit
  

12
Keck 2 Remote Observing Room as seen from the
Keck 2 summit

• Telescope operators at the summit converse with
  astronomer at Keck HQ in Waimea via the
  videoconferencing system.

13
Videoconferencing has proved vital for remote
observing from Waimea

• Visual cues (body language) important!
• Improved audio quality extremely valuable
• A picture is often worth a thousand words
• Troubleshooting live oscilloscope images
• Cheap desktop sharing (LCD screens)
• Chose dedicated versus PC-based units
• Original (1996) system was PictureTel 2000
• Upgrading to Polycom Viewstations

14
Interaction between video-conferencing and type
of monitors

• Compression techniques motion sensitive
• Moving scene requires more bandwidth
• CRT monitors cause flicker in VC image
• Beating of frequencies camera .vs. CRT
• CRT phosphor intensity peaking, persistence
• CRT monitor flicker causes problems
• Wastes bandwidth and degrades resolution
• Visually annoying / nausea inducing
• Use LCD monitors to avoid this problem

15
The Keck Headquarters in Waimea

• Most Keck technical staff live and work in
  Waimea. Allows direct contact between observers
  and staff. Visiting Scientists Quarters (VSQ)
  located in same complex.

16
Limitations of Remote Observing from Keck HQ in
Waimea

• Most Keck observers live on the mainland.
• Mainland observers fly 3,200 km to get to
  Waimea
  
• Collective direct travel costs exceed 400,000
  U.S. / year

17
Remote Observing from Waimea is not cost
effective for short runs

• Round trip travel time is 2 days
• Travel costs 1,000 U.S. per observer
• About 50 of runs are for 1 night or less
• Cost / run is very high for such short runs
• Such costs limit student participation

18
Motivations for Remote Observing from the U.S.
Mainland

• Travel time and costs greatly reduced
• Travel restrictions accommodated
• Sinus infections and ruptured ear drums
• Late stages of pregnancy
• Increased options for
• Student participation in observing runs
• Large observing teams with small budgets
• Capability for remote engineering support

19
Mainland remote observing goals and
implementation strategy

• Goals
• Target mainland facility to short duration runs
• Avoid duplicating expensive Waimea resources
• Avoid overloading Waimea support staff
• Strategy
• No mainland dormitories observers sleep at home
• Access existing Waimea support staff remotely
• Restrict mainland facility to experienced
  observers
  
• Restrict to mature, fully-debugged instruments

20
Mainland remote observing facility is an
extension of Keck HQ facility

• Only modest hardware investment needed
• Workstations for mainland remote observers
• Network-based videoconferencing system
• Routers and firewalls
• Backup power (UPS) especially in California!!!
• Backup network path to Mauna Kea and Waimea
• Avoids expensive duplication of resources
• Share existing resources wherever possible
• Internet-2 link to the mainland
• Keck support staff and operational software

21
Keck software is accessed the same regardless of
observers location

• The control computers at the summit
• Each telescope and instrument has its own
  computer
  
• All operational software runs only on these
  computers
  
• All observing data written to directly-attached
  disks
  
• Users access data disks remotely via NFS or
  ssh/scp
  
• The display workstations
• Telescopes and instruments controlled via X GUIs
• All users access these X GUIs via remote X
  displays
  
• X Client software runs on summit control
  computers
  
• Displays to X server on remote display workstation

22
Overall Topology
Type your question here, and then click Search.
23
View Empty
24
Santa Cruz Remote Observing Facility
25
Santa Cruz Remote Observing Video Conferencing
26
The Weather in Waimea
27
Why did we choose this approach?

• Operational Simplicity
• Operational control software runs only at the
  summit
  
• All users run identical software on same
  computer
  
• Simplifies management between independent sites
• Allowed us to focus on commonality
• Different sites / teams developed instrument
  software
  
• Large variety of languages and protocols were
  used
  
• BUT all instruments used X-based GUIs

28
Focus effort on X standardization and
optimization over long links

• Maintain consistent X environment between sites
• Optimize X performance between sites
• Eliminates need to maintain
• Diverse instrument software at multiple sites
• Diverse telescope software at multiple sites
• Coordinate users accounts at multiple sites
• Fewer protocols for firewalls to manage

29
Remote observing differences Waimea versus the
mainland

• System Management
• Keck summit and HQ share a common domain
• Mainland sites are autonomous
• Remote File Access
• Observers at Keck HQ access summit data via NFS
• Observers on mainland access data via ssh/scp
• Propagation Delays
• Summit to Waimea round trip time is about 1 ms.
• Summit to mainland round trip time is about 100
  ms.

30
Increased propagation delay to mainland presents
challenges

• Initial painting of windows is much slower
• But once created, window updates fast enough
• All Keck applications display to Waimea OK
• A few applications display too slowly to
  mainland
  
• System and application tuning very important
• TCP window-size parameter (Web100 Initiative)
• X server memory and backing store
• Minimize operations requiring round trip
  transactions

31
Simulating long propagations delays in the lab

• Instruments are designed and built on mainland
• Software is debugged on local area network
• Testing on LAN does not reveal delay problems
• Must measure delay effects before deployment
• Simulate WAN delays using NIST simulator
• Requires Linux PC with dual Ethernet interfaces
• Can select specific packets delays, losses,
  jitters
  
• http//www.antd.nist.gov/itg/nistnet

32
Shared access and control of instruments

• Most software for Keck optical instruments
  provides native multi-user/multi-site control
  
• All users have consistent view of status and
  data
  
• Instrument control can be shared between sites
• Multipoint video conferencing key to
  coordination
  
• Some single-user applications can be shared via
  X-based application sharing environments
  
• XMX http//www.cs.brown.edu/software/xmx
• VNC http//www.uk.research.att.com/vnc

33
Tradeoffs from this approach to remote observing

• Disadvantages
• X protocol does not make optimal use of
  bandwidth
  
• Long propagation delays require considerable
  tuning
  
• Advantages
• Minimizes staffing requirements at mainland
  sites
  
• Only vanilla hardware and software needed
  there
  
• Simplifies sparing and swapping of equipment
• Simplifies system maintenance at mainland sites
• Simplifies authentication/access control

34
Fast and reliable network needed for mainland
remote observing

• 1997 1.5 Mbps Hawaii - Oahu - mainland
• 1998 10 Mbps from Oahu to mainland
• 1999 First phase of Internet-2 upgrades
• 45 Mbps commodity link Oahu - mainland
• 45 Mbps Internet-2 link Oahu - mainland
• 2000 Second phase upgrade
• 35 Mbps Internet-2 link from Hawaii - Oahu
• Now 35 Mbps peak from Mauna Kea to mainland
• 2002 155 Mbs from Oahu to mainland

35
End-to-end reliability is critical to successful
remote operation

• Keck Telescope time is valued at 1 per second
• Observers wont use facility if not reliable
• Each observer gets only a few nights each year
• What happens if network link to mainland fails?
• Path from Mauna Kea to mainland is long
  complex
  
• At least 14 hops crossing 6 different network
  domains
  
• While outages are rare, consequences are severe
• Even brief outages cause session collapse
  panic
  
• Observing time loss can extend beyond outage

36
Keck Observatory policy on mainland remote
observing

• If no backup data path is available from mainland
  site, at least one member of observing team must
  be in Waimea
  
• Backup data path must be proven to work before
  mainland remote observing is permitted without no
  team members in Waimea

37
Mitigation plan install end-to-end ISDN-based
fall-back path

• Install ISDN lines and routers at
• Each mainland remote observing site
• Keck 1 and Keck 2 control rooms
• Fail-over and fall-back are rapid and automatic
• Toll charges incurred only during network outage
• Lower ISDN bandwidth reduces efficiency, but
• Observer retains control of observations
• Sessions remain connected and restarts avoided
• Prevents observer panic

38
Summary of ISDN-based fallback path

• Install 3 ISDN lines (6 B channels) at each site
• Install Cisco 2600-series routers at each end
• Quad BRI interfaces
• Inverse multiplexing
• Caller ID (reject connections from unrecognized
  callers)
  
• Multilink PPP with CHAP authentication
• Dial-on-demand (bandwidth-on-demand)
• No manual intervention needed at either end
• Fail-over occurs automatically within 40 seconds
• Uses GRE tunnels, static routes, OSPF routing

39
Running OSPF routing over aGRE tunnel

• On each router, we configure 3 mechanisms
• A GRE tunnel to the other endpoint
• A floating static route that routes all traffic
  to the other endpoint via the ISDN dialer
  interface
  
• A private OSPF domain that runs over the tunnel
• OSPF maintains its route through the tunnel only
  if the tunnel is up
  
• OSPF dynamic routes take precedence over floating
  static route

40
Fail-over to ISDN backup data path

• If the Internet-2 path is up, OSPF hello
  packets flow across the tunnel between routers
  
• As long as hello packets flow, OSPF maintains
  the dynamic route, so traffic flows through
  tunnel
  
• If Internet-2 path is down, OSPF hello
  packets stop flowing, and OSPF deletes dynamic
  route
  
• With dynamic route gone, floating static route is
  enabled, so traffic flows through ISDN lines

41
Fall-back to the normal Internet-2 path

• OSPF keeps trying to send hello packets through
  the tunnel, even with Internet is down
  
• As long as Internet-2 path remains down the
  hello packets cant get through
  
• Once the Internet-2 path is restored, hello
  packets flow between routers
  
• OSPF re-instates dynamic route through tunnel
• All current traffic gets routed through the
  tunnel
  
• All ISDN calls are terminated

42
Operational costs of ISDN backup data path

• Fixed leased cost is 70 per line per month
• Three lines at each site - 2,500 per site/year
• Both sites - 5,000/year
• Long distance cost (incurrent only when active)
• 0.07 per B-channel per minute
• If all 3 lines in use
• 0.42 per minute
• 25.20 per hour

43
Recent operational experience

• Remote observing science from Santa Cruz
• Low Resolution Imaging Spectrograph (LRIS)
• Echellete Spectrograph and Imager (ESI)
• Remote engineering and instrument support
• ESI
• High Resolution Echelle Spectrometer (HIRES)
• Remote Commissioning Support
• ESI
• DEIMOS (see SPIE paper 4841-155 4841-186)

44
Unplanned use of the facility during week of
Sept. 11, 2001

• All U.S. commercial air traffic grounded
• Caltech astronomers have a 5-day LRIS run on
  Keck-I Telescope starting September 13
  
• No flights available
• Caltech team leaves Pasadena morning of 9/13
• Drives to Santa Cruz, arriving late afternoon
• Online with LRIS well before sunset in Hawaii

45
The hardest problem was the lodging!

• LRIS operated from Santa Cruz all 5 nights
• ISDN backup path activated several times
• Observing efficiency comparable to Waimea
• Lodging was the biggest problem
• Motel check-in/check-out times incompatible
• Required booking two motels for the same night
• Motels are not a quiet place for daytime sleep

46
Extending mainland remote observing to other sites

• Other sites motivated by Santa Cruz success
• Caltech remote facility is nearly operational
• Equipment acquired
• ISDN lines and router installed
• Will be operational once routers are configured
• U.C. San Diego facility being assembled
• Equipment specified and orders in progress
• Other U.C. campuses considering plans

47
Administrative challenges scheduling shared
facilities

• Currently only one ISDN router at Mauna Kea
• Limits mainland operation to one site per night
• Interim administrative solution
• Longer term solution may require
• Installation of additional ISDN lines at Mauna
  Kea
  
• Installation of an additional router at Mauna Kea

48
Remaining challenges

• TCP/IP tuning of end-point machines
• Needed to achieve optimal performance
• Conflicts with using off-the-shelf
  workstations
  
• Conflict between optimal TCP/IP parameters for
  the normal Internet-2 path .vs. the ISDN
  fall-back path
  
• Hoping for vendor-supplied auto-tuning
• Following research efforts of Web100 Project
• Administrative challenges
• Mainland sites are currently autonomous
• Need to develop coordination with Keck

49
Summary

• Internet-2 makes mainland operation feasible
• Backup data path protects against interruptions
• Keck HQ is the central hub for remote operation
• Mainland remote observing model is affordable
• Mainland sites operate as satellites of Keck HQ
• Leverage investment in existing facilities and
  staff
  
• Leverage investment in existing software
• Share existing resources wherever feasible
• Avoid expensive and inefficient travel for short
  runs
  
• Model is being extended to multiple sites

50
Acknowledgments

• U.S. National Science Foundation
• U.S. Department of Defense
• University of Hawaii
• Gemini Telescope Consortium
• University Corp. for Advanced Internet
  Development (UCAID)
  
• Corporation for Education Network Initiatives in
  California (CENIC)

51
Author Information

• Robert Kibrick, UCO/Lick Observatory
• University of California, Santa Cruz
• California 95064, U.S.A.
• E-mail kibrick_at_ucolick.org
• WWW http//www.ucolick.org/kibrick
• Phone [phone]
• FAX [phone]