After completion of Observing Plan and whilst running unattended the AstroMain program froze and stopping functioning immediately after the Session entered the 'Closing_State' at 05:48 in order to begin Observatory Shutdown.   It is unclear what caused the program to stop working.  Other times (eg session as recently as 2019-02-xx S0066x) the observatory shutdown has operated ok.  One one previous occasion (2019-02-xx S00065) the program crashed/stopped working at around this point.  

There is nothing in the Report File or Log File to indicate why the Program stopped working.  Upon remote login to the computer there was nothing but blank program window and a message saying to stop. (see below) . There was a message in PHD2 window saying "PHD2 is not able to make sufficient corrections in RA... " but this may related to an earlier event is judged to be irrelevant to the issue under investigation)

Athough the critical functioning of the AstroMain (ie the Obs.Manger) had failed at 05:48 some functions of the program continued.  Obs.Monitor for example continued receiving and output sky and equipment data to (monitor).dat file.

Although AstroMain program stopped working at 05:48 the critical momment after finishing the observaing session and just before observatory shutdown secondary and tertiary protection measures operated as per design.

AstroGuard program was running and monitoring the Telescope Position and was ready to intervene and move the scope to a safe position should it have tracked beyond the Az.Limit of 350 deg , designed to prevent the cables from wrapping around the mount if the telescope moved beyond the North position.

The program picked up that the AstroMain Obs.Manager had stopped and therefore wasn't in a position to look after the Observatory
(this is a new feature and at the moment AstroGuard doesn't do anything with this information). 

When Light Conditions had passed a critical threshold at 07:18 an alert was raised and the Shutter Status was checked and the shutter would have been closed if it had been open. As it turned out the Shutter had already closed at 06:34.  This was at a time when light sensor was reading around 5.1 - 5.3 and is consistent with a direct relay controlled closing of the shutter when the CloudSensor registered unsafe conditions with a Light Level exceeding 5.0 (relay would have also operated if Clarity has dropped to 22 or below or Rain had reached a value of 1 or more)

 

Examining things in more detail the weather flag are sent to AstroGuard from AstroAllSky on a delay of up to 60s. AstroAllSky sets the  Light_Flag is to 'Critical' when light reading reach a level of 28 (or Sun Altitude is > 0°, ie daytime). 

Flag is set to 'Warning'  when light readings reach a level of 15 (or Sun Altitude is > -6°, ie Civil Twilight).  
In the event of no available Light Sensor Readings  the Light_Flag is deemed Critical if Sun Altitude is > -2°, otherwise it is set to 'Warning".

Due to switch over from night to daytime operating mode there is gap in the issuing of updated Scorecard information by AstroAllSky program around this time in the morning.    Last scorecard from the night session was at 07:02 when light reading  was at 27.5 (Sun Alt -2°), whereas the first scorecard from the day session was not till 07:17 when light reading as at a value of 30.5 (Sun Alt +0.1°).

AstroMain's Obs.Manager is designed to stop the observing queue and close the shutter and commence observatory shutdown if Light readings reach 15 or Sun Altitude rises above -6°, ie Civil Twilight, (unless DaytimeTesting_Flag is set)

Actions

>> AstroGuard should act to close shutter and park the telescope and the dome when light flag reaches Warning State (ie at Civil Twilight in the event that AstroMain has either stopped or hasn't acted after a 3 minute delay.
(There may be occasions where Dawn Flats are being taken, but these will need to be done under supervision, AstroGuard would be closed for occasional operations of this nature))
 
>> Review issue of Scorecard Data between at night/day time interface and whether AstroGuard should have acted as soon as ScoreCard data went down for its 15 minute gap

I’ve recently begun a new project to try to detect the optical afterglow of a Gamma Ray Burster (GRB). 

[ Gamma-ray bursts were first observed in the late 1960s by the U.S. Vela satellites, which were built to detect gamma radiation pulses emitted by nuclear weapons tested in space. From 1991 onwards their distribution was shown to be isotropic— i.e. not biased towards any particular direction in space, and therefore not part of the milky way, but are located in distant galaxies. An optical afterglow was first detected for one in 1997. Recently in 2017/2018 scientists are finding direct temporal relationships between some short-lived GRB’s and gravitational wave events associated with the merger of two neutron stars. ]

The aim is to pick up on GRB alert messages sent by observing spacecraft like Swift and then have my telescope automatically slew across to the location of the GRB and start taking pictures all within 90s of the burst event. Swift BAT detects around 100 events / year, of which may be 2 per year will be at night in the part of the sky that my telescope can reach and at a times when it is not cloudy and my telescope system is running.

A new application (AstroVOE) has been built that has a TCP/IP client with makes a direct socket connection to the Nasa’s GCN Network (https://gcn.gsfc.nasa.gov/) , a parser to read the various notices, a filter to extract the GRB alerts, an evaluator to see if the GRB is in the part of the sky that my scope can currently see here in Scotland and finally a fully-automated  target of opportunity (ToO ) process to enable the GRB to inserted into the observatory's observing queue.

In response to a Priority 1 ToO the Observatory Control Program (AstroMain)  program is able to immediately stop observations on the current target and immediately slew to the GRB target coordinates and begin observations (AstroMain-FastReact). In order to minimise the time to first observation, some shortcuts are taken over normal centering procedure and focus runs etc are purposedly skipped .

The system is still waiting for it's first live GRB at a time when my Observatory &  Scope are active, however one of the regular test GRB Alerts came through last night which provided a first full live test.

2019-02-26 02:37:24 GCN/SWIFT NOTICE   Type 82   Test SWIFT_BAT_GRB_POS_TEST     99999     GRB_ALERT
                    Submitting GRB Swift 99999_Test (ID: 627) to job queue to Execute Immediately
                    ToO submitted at 2019-02-02 02:37:25 (Trigger Num: 99999)

-----------------------------------------------------------------
TITLE:          GCN/SWIFT NOTICE
NOTICE_DATE:    2019-02-26T02:37:24
ROLE: Test
NOTICE_TYPE:    Swift-BAT GRB Position Test
PACKET_TYPE:    82
TRIGGER_NUM:    99999, Ser Num: 3
GRB IDENTIFIED: YES
CONSTELLATION:  Auriga (Aur)
GRB RA:         04 47 60.000 [J2000]
GRB DEC:        +35 00 00.00 [J2000]
ERROR_RADIUS:   3.0 [arc min]
COORD_SYSTEM:   UTC-FK5-GEO
EVENT_INTENSITY: []
BURST_TJD/SOD:  18540 [days], 9436.00 [sec]
BURST_DATE:     2019-02-26 (Tue)
BURST_TIME:     02:37:16.00 UT
BURST_INTENSITY: 1000 [cts]
BURST_PEAK:     500 [cts]
BKG_INTENSITY:  100 [cts]
BKG_DURATION:   12 [sec]
RATE_SIGNIF:    16.78 [sigma]
IMAGE_SIGNIF:   12.98 [sigma]
TRIGGER_INDEX:  12
IMPORTANCE:     0.90
INFERENCE PROB: 0.98
CONCEPT:        process.variation.burst;em.gamma
COMMENT:        Swift Satellite, BAT Instrument
IVORN:          ivo://nasa.gsfc.gcn/SWIFT#BAT_GRB_Test_Pos_2019-02-26T02:37:16.00-452
PACKET_NAME:    SWIFT_BAT_GRB_POS_TEST

AZIMUTH:           312.99 [deg]
ALTITUDE:          15.08 [deg]
MOON DISTANCE:     159.23 [deg]
MOON ILLUMINATION: 53.95 [%]
NEAR BRIGHT STAR:  NO
CAN REACT?:        YES
-----------------------------------------------------------------

Analysing the results of the test, found that whilst some things worked well (GRB filter & evaluator, ToO submission, ToO importing, and aborting of the target in progress once the ToO was queued) there was a failure to immediately move GRB target from the ToO list into the active job queue, and it was only moved when the target in progress had finished  leading to  a 14  minute delay in the process on this occasion.    The  timeline deduced from logs and other sources is as follows:

02:35     Started GCVS Z And
02:35     Aborted GCVS Z And (Failed airmass limit check)

02:35     Started GCVS CI Cyg
02:35     Began an 8 minute wait until 02:43 (5 min before Planned Start Time)

02:37:16  GRB Test Burst Time   (T=0)
02:37:24  GRB Notice Time       (T+8s) normally T+20 for a real GRB)
02:37:25  ToO Submitted         (T+9s) (just 1s for notice to get to UK and process it)
02:37:28  ToO “GRB Swift 99999_Test” (priority 1) imported and added to ToO list

02:43     Finished 8 minute wait
02:43     Slewed to GCVS CI Cyg
02:45:40  Started a Focus Run
02:51:41  Finished imaging GCVS CI Cyg

02:51:43  Started 61 Cyg
02:51:44  Target (GRB Swift 99999_Test, priority 1) inserted in job queue
02:51:45  Aborted 61 Cyg

02:51:45  Started ‘GRB Swift 99999_Test’  (T +14.5 mins)
02:52     Slewed from Az 59 to Az 316 (257 degrees, so a fairly long slew)
02:53:01  Started 10s locate frame        (T +15.6 mins , not good enough)
          (unable to link due to cloud, 2 further locate frames failed due to cloud)
02:55:16  Started first ‘Science’ frame   (T +17.7 mins)

On this occasion the telescope had to slew all the way round from Az 59 to Az 316 (ie 257 degrees of turn) in order to reach the GRB target.  Whilst the dome is able to take a shortcut across the north position, the telescope can't.   The slew was a fairly long one and took 63s to complete.     The target was only 15deg above the horizon and haze/cloud prevented any useful imaging on this occasion.

Digging deeper it was found the delay in getting the GRB Target into the job queue was due to a bug in the code, where an incorrect logic test was applied to test for a target in the ToO list. This has now been fixed ready for the next session.