UNIVERSITY OF CALIFORNIA OBSERVATORIES / LICK OBSERVATORY

Shane AO Startup Procedures

Cabling/Power | Ethernet | Servers | ShARCS Setup | Power GUI | Motor GUI | ShARCS GUIs | SAO UI | Peeko | TT Peeko | SAO Initialization | Laser Shutdown Monitor | Startup Checklist

Cabling and powering up the instrument.

Assuming that everything was properly cabled up in the lab and that it has been mounted properly to the Shane 3-m Telescope, there are only six cables that need to be connected for the Shane AO (SAO) instrument:
  1. Ethernet Cable - This cable must be connected to the gigabit network.
  2. Power Cable 1 - Plugged into the UPS protected outlet on the TUB labeled AO Only. SAO Pulizzi B and C are powered from this outlet.
  3. Power Cable 2 - Plugged into the non-UPS protected PG&E power outlet on the TUB. SAO Pulizzi A is powered from this outlet.
  4. Three high voltage BNC cables need to be connected from the PI uplink tip/tilt mirror power supply to the TUB bundle. If all is well, these will already be connected, but if not they are connected thusly:
    1. X Axis to TUB BNC #3
    2. Y Axis to TUB BNC #4
    3. 100V to TUB BNC #5
  5. Two regular BNC cables from the PI uplink tip/tilt mirror power supply to the ShaneAO instrument. X-axis is a black BNC cable, Y-axis is a green BNC cable.
When the Pulizzis are powered on they are set to turn on the following items immediately:
Pulizzi A: Pulizzi B: Pulizzi C:

Checking Ethernet Communications

You should be able to ping the following computers, Lantronixes, etc. that are on the SAO instrument: The following are not used in regular operations, but are in the AO bench: The following computers are not on the instrument but are used for SAO and ShARCS operation:

Starting Server Software

There are lots of pieces of software that do many different things. For the most part you don't need to start most of the below pieces individually, but can start them all by typing the following command:

/usr/local/lick/bin/ao start

on hosts shark and covert. The services on real (saocon) and musher (woofer) should start at boot time or when starting the realtime code. More details about starting individual components are below.

Power Server

The power dispatcher runs on host shadow and can be started with the command:

/usr/local/lick/bin/ao start saopower

Motor Servers

The motor dispatchers run on host shadow and can be started with the command:

/usr/local/lick/bin/ao start saomot

If you need to only start a particular motor dispatcher, the command is:

/usr/local/lick/bin/ao start saomot1

or saomot2, saomot3, saomot4 (or rio), saomot5, or saomot7.

Motors controlled from each dispatcher (in Galil channel order ABCDEFGH):

Instrument Monitoring Servers

There are two monitoring programs, saomon1 and saomon2. saomon1 is the Primary Monitor (everything except the heartbeat monitor) and saomon2 is the Heartbeat monitor for the MEMS DM. They are started on host shark with the command:

/usr/local/lick/bin/ao start saomon

or saomon1 or saomon2 if you wish to start them individually.

Keyword History Server

The AO keywords are written to accession so a history of state and changes can be saved. The status of the aohistory service can be checked by logging onto accession and executing the following command:

/usr/local/lick/etc/init.d/ao_keygrabber status

To start the service type:
/usr/local/lick/etc/init.d/ao_keygrabber start

ShARCS Services

There are a number of services that help with the operation of ShARCS. On host: covert, the relevant programs are sharcspump, ao_keyheader, and sharcs. All are started on host covert by the command:

/usr/local/lick/bin/ao start

or individually by typing:

/usr/local/lick/bin/ao start sharcspump

or ao_keyheader or sharcs.

sharcspump controls the ion pump for the ShARCS dewar. ao_keyheader facilitates adding information to ShARCS FITS headers, and sharcs governs the communications between stealthie and covert for ShARCS detector operation.

Additional ShARCS services run on host shark. mako, which governs communications between ShARCS and SAO, and shawcs, the ShARCS World Coordinate System service. mako is started with the command:

/usr/local/lick/bin/ao start mako

and shawcs started with the command:

/usr/local/lick/bin/ao start shawcs

Temperature monitoring, sharcstherm, runs on shanevnc and be started with the following command:

/usr/local/lick/bin/ao start sharcstherm

Starting ShARCS

Information on starting ShARCS's detector control software on stealthie is available in: HxRG_CustomManual_WeissMod_v3.pdf.

Note: Replace HxRG2_3 with Test in step 10 of these directions to select the Test.soln file.

Start temperature and pressure monitoring through saomon with the following commands:

modify -s saomon SH_BASICDIS=0

modify -s saomon SH_COMBODIS=0

Starting AO Software User Interfaces

Power GUI saopower_ui

The power control GUI is started with the command
/usr/local/lick/bin/saopower_ui
After starting the GUI, make sure that the items turned on by default when the Pulizzi are powered up are on and that the others are off.

Motor Control GUI saomot_gui

To start the motor control GUI, type
/usr/local/lick/bin/saomot_gui

Once the GUI is up, the stages will all need to be calibrated. Stages can be calibrated one at a time by clicking the Calibrate button in the motor's detail panel or all at once using the Actions - Calibrate - All Stages menu item. Calibrating all at one time is not recommended at this time because the TT Cam X, TT Cam Y, Sci filter, and Waveplate motors are currently not installed.

Problems can occur with homing and there are known ways to deal with them as follows:

Once stages are calibrated, they should be moved to the following nominal, on-axis positions:

Also be sure to set the Orientation to OnTelescope.

Starting ShARCS GUIs

There are a few user interfaces for ShARCS:

Detector Controller sharcs_fe

The main GUI for taking exposures, sharcs_fe is started on covert with the following command:
sharcs_gui
or if that fails for some reason java -jar /usr/local/lick/java/jar/sharcs_fe1.jar

Once the GUI is started, open Windows - Setup and do the following:

  1. Make sure AsicCfg is C:\startup\SHARCSColdControlConfig.xml
  2. Make sure UnixRootDir is /data
  3. If Connected is NO or if the upper right corner of sharcs_fe is red and says "Lost Server", press the Connect button.
  4. Initialize system by pressing the Init button. Initing status will change to Yes during the initialization and SideCarInit should say Success! when done (and Initing status will change to No).
Note, there is no harm in pushing the Connect button if you are already connected, but if you do, you must then press the Init button.

Image Display sharcsdisplay

The image display software is started on covert with the command
/usr/local/lick/bin/sharcsdisplay

Filter Wheel Controller saowheels_gui

The wheel and aperture wheel motor controller is started on covert or shanevnc with the command
/usr/local/lick/bin/saowheels_gui
Once the GUI is up, calibrate each of the three wheels by clicking Calibrate on its detail panel. Now the filters can be moved to the desired positions.

Script Tool scriptproc.tcl

The script running GUI is started on covert with the command
/usr/local/lick/bin/scriptproc.tcl

Sound GUI eventsounds

The eventsounds GUI can be started on any of our user computers (e.g. shanevnc) with the following command:
/usr/local/lick/bin/eventsounds -a sao

Ion Pump GUI sharcspump_gui

ShARCS ion pump control GUI is started with the command
/usr/local/lick/bin/sharcspump_gui
Once the GUI is started, the high voltage power supply, HV needs to be turned on.

Starting Shane AO UIs

At this time there are few UIs for SAO operation, so many things need to be started and run from command line interfaces. Short list of items is:

Message Log

Before starting the shaneao realtime control code, start monitoring real's message log so you can monitor the state of the system. To view the message log, log into real as user and type the following:
tail -f /var/log/messages
Pay close attention to this log as important errors about DPIO2 will be shown there.

Realtime Code shaneao

Now the realtime SAO control code can be started on real from the user account with the following command:
/usr/local/lick/bin/shaneao
Once the program starts you will see the following information if all is normal. 
DPIO2:  DPIO2 Device Driver - Release: 2.4.13
        Copyright (C) 2005 - VMETRO, Inc.  All rights reserved.
        Creation Date: Apr 21 2014, Time: 16:07:22
DPIO2:  Scan PCI for DPIO2 modules. Found 1
DPIO2:  Device #1 (bus 55, device 9) is a DPIO2-LB with FPGA code BO514
The software should automatically check to make sure things are normal and can start. If not or if you get a vastly different message, either there is already a shaneao running (only one can run at a time) or there may be a problem with communicating with the DPIO2 MEMS driver (in which case a reboot of real might be necessary). In either case you'll want to exit shaneao at the prompt.

Another error that shows up in the Message Log is 

Aug  1 11:05:59 real kernel: last message repeated 14476 times
Aug  1 11:07:00 real kernel: last message repeated 14475 times
Aug  1 11:08:01 real kernel: last message repeated 14476 times
which indicates a problem and will result in poor performance of the SAO system. The recommended solution is to exit shaneao at the prompt and restart it.

SAO realtime control GUI - saocon_gui

The SAO realtime loop control GUI, saocon_gui, runs on shanevnc from the user account. To start it type
/usr/local/lick/bin/saocon_gui

WFS Display peeko

From the gavel account on shade, the WFS and DM actuator displays are started with the following command:
/u/gavel/local/bin/peeko

TT Camera Display ttpeeko

From the gavel account on shade, the TT Camera display is started with the following command:
/u/gavel/local/bin/tpeeko

Field Steer software

From user@shanevnc, the field steering GUI is started with the following command:
/usr/local/lick/bin/fieldsteer.tcl

Initializing ShaneAO

Now that all the software is started, it is ready for use. Issue the following command to initialize shaneao and move the lenslet array:
init(16)
for the 16x lenslet, or
init(8)
for the 8x lenslet.

If the above init command fails (which is extremely unlikely), you need to do the following:

  1. Start the WFS camera:
    camstart()
  2. Start the TT camera:
    ttcamstart()
  3. Read frame rate of WFS camera:
    rate()
  4. Read frame rate of TT camera:
    ttrate()
  5. Examine the state to see what might need to be changed:
    state()
At this point the realtime code is ready to use and the woofer and tweeter DMs can be turned on. The woofer is turned on via the saopower_ui GUI outlet B5. Before turning on the tweeter, check the message log on real to make sure there are no SW-IOMMU error messages. The tweeter MEMS mirror is turned on via the saopower_ui outlet B6 (labeled Heartbeat failsafe).

At this point you can turn on the woofer and MEMS in the shaneao software with the following commands:

  1. Turn on the woofer communications:
    woof('on')
  2. Turn on the tweeter (MEMS) communications:
    mems('on')

Starting the Flea Camera

Coming to a theater near you sometime in the future...... Sorry, can't go the theater because of COVID....

Starting Laser Shutdown Monitor

The laser shutdown monitor runs on shanevnc and is started with the command
/usr/local/lick/bin/LSM.tcl
Once started, select the proper directory and laser target.

Starting the Laser Offload GUI

The laser offload monitoring GUI runs on shanevnc and is started with the command
/usr/local/lick/bin/laserOffload.tcl

Starting AO telescope position GUI

The AO version of the telescope position GUI run on shanevnc and is started with the command
/usr/local/lick/bin/aopococ

Startup Checklist

  1. Mount SAO on TUB.
  2. Connect 2 power cables.
  3. Connect 1 ethernet cable.
  4. Connect 2 laser uplink BNC cables.
  5. Ping real, stealthie, musher, covert, and shade computers. Assume other devices (lantronixes, routers, etc.) are also up if these few computers ping.
  6. Do ao start on covert and shark.
  7. Start ShARCS detector software on stealthie.
  8. Start saopower_ui on shanevnc.
  9. Check that appropriate items are turned on.
  10. Start saomot_gui on shanevnc.
  11. Calibrate motors.
  12. Set motors to nominal positions (choose the most recent default setup on shanevnc: /u/user/observers/lgs/saomot_setups).
  13. Start sharcs_fe on covert.
  14. In sharcs_fe Windows - Setup, check parameters, connect, and initialize.
  15. Start sharcsdisplay on covert.
  16. Start saowheels_gui on shanevnc or covert.
  17. Calibrate wheels if necessary and set to desired positions.
  18. Start scriptproc.tcl on covert.
  19. Start eventsounds on shanevnc.
  20. Start sharcspump_gui on shanevnc.
  21. Turn on ion pump HV supply.
  22. Start tail -f /var/log/messages on real.
  23. Start shaneao
  24. Start saocon_gui on shanevnc.
  25. Start peeko on shade.
  26. Start ttpeeko on shade.
  27. In shaneao start WFS and TT cameras: camstart() and ttcamstart()
  28. Turn on power to Woofer (B5) and Heartbeat failsafe (B6) if there are no SW-IOMMU errors in the message log.
  29. Initialize shaneao: Type init(16) (or 8) in shaneao, or click Actions - Init 16 (or 8) in saocon_gui.
  30. Start fieldsteer.tcl on shanevnc.
  31. Start aopococ on shanevnc.
  32. Start LSM.tcl on shanevnc if using laser.
  33. Start laserOffload.tcl on shanevnc if using laser.
  34. start saoWFSFocusTrack.tcl on shanevnc if using laser.
  35. Start samon_gui on shanevnc.
Last modified: Fri Mar 19 15:28:21 PDT 2021