David's Astronomy Pages
Notes - Session 1014 (2022-05-31)

 
Bullet Session Aims & Highlights
 - Observing Result
 - Night Summary Plot
 - Session Event Log
Bullet Operational Issues
  - Critical Issues (0),  Major Issues (0),  Minor Issues (1),  Small Defects (0),  Continuous Improvement (0)
 
Bullet Images from 2022-05-31 >>         [ Local Files >> ]    
Bullet Investigation -  Issue Identifying End Of Dome Slew (SS Cyg, 1:08)

Session Aims & Highlights (2022-05-31)

Main aims

  1. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by time & conditions.

 Equipment & Software

Highlights

Notes:

  Summary Plots & Logs

Observing Plan
Image
  
Observing Result
Image
   
  
Dome & Scope Slewing Performance
Image
  
Slew/Centering Performance
Image
  
Guiding Performance
Image
Image
 
  
Sky Conditions (Locate Frames)
Image
 
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Image   
  
Actual Weather vs Pre-Session Weather Forecast
Image
Image   
  
Session Event Log
Time     Event Detail
23:08:57 Session Monitoring AutoStart monitoring for Live Session opportunity between 23:46 & 01:13
23:08:58   AutoStart Waiting AutoStart waiting till earliest start time at 23:46
23:46:18   AutoStart Delayed Session delayed due to threat of rain
00:40:34 Session AutoStarting Session autostarting (00:40)
00:41:14   Camera1 Connected SBIG Camera Connected (set point -20°C)
00:41:16 Session Created Live Session Created (2022-05-31 S01014, ImageSaveNum: 1014001)
00:41:18   Scope Switched On Telescope Power has been switched on via UPB Powerbox.
00:43:00   Services Started Observatory Services started
00:43:07 Observatory (Auto) Observatory placed in Fully-Automated Mode
00:43:10 Session Pending Session pending (2022-05-31)
00:43:12 Session Initiating Session initiating (2022-05-31)
00:43:14   Plan Requested Observing Plan requested from AstroPlan (1.30.7)
00:43:51   Plan Loaded Observing Plan loaded to queue (Plan ID: 774)
00:44:06   Camera1 Connected SBIG Camera Connected (set point -15°C)
00:44:11   Telescope Connected Telescope Connected (TheSky6)
00:44:36 Session Equilibration Session ready for dome & camera equilibration
00:46:16 Wait On Weather Waiting for acceptable conditions
00:46:18 Session Suspended Session suspended
00:46:20   Dome Closed Dome already closed (closure time is unknown)
00:46:22   Queue Started Observing Queue started (6 targets selected)
00:55:47 Session Resuming Session resuming
00:56:32   Dome Opened Dome opened (opening time 45s)
00:56:34 Session Running Session running
00:56:36   Queue Resumed Observing Queue resumed
00:56:38     Target Started (NrZen) Target started (Focus Field 16, HIP 78946)
00:59:05       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:02:04       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide)
01:04:01       Focusing Completed Foc1 AutoFocus Completed (Profile No 1)
01:04:26       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
01:06:35       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide)
01:08:22       Focusing Completed Foc2 AutoFocus Completed (Profile No 2)
01:08:40     Target Completed Target completed (Focus Field 16, HIP 78946)
01:08:44     Target Started (1/6) Target started (1/6, GCVS SS Cyg)
01:11:35       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:13:16       Focusing Completed Foc1 AutoFocus Completed (Profile No 3)
01:17:59     Target Completed Target completed (1/6, GCVS SS Cyg)
01:18:03     Target Started (2/6) Target started (2/6, GCVS BL Lac)
01:20:41       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:22:53       Focusing Completed Foc1 AutoFocus Completed (Profile No 4)
01:26:56     Target Completed Target completed (2/6, GCVS BL Lac)
01:27:00     Target Started (3/6) Target started (3/6, NGC 4088 w/SN2022jzc)
01:32:33       Focusing Skipped Foc1 focusing skipped - unable to find a star (TCF-S)
01:49:46     Target Completed Target completed (3/6, NGC 4088 w/SN2022jzc)
01:49:50     Target Started (4/6) Target started (4/6, AT2022jrp (CV))
01:52:14   SoftSuspend Called Soft Suspend is called due to Deteriorating Conditions (reducing clarity)
01:53:06       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:53:44       Focusing Failed Foc1 focusing failed - star lost
01:56:39 Critical Cloud Alert Critical Cloud Alert (Obs.Manager will close the Dome)
01:56:41   HardSuspend Called Hard Suspend is called due to Critical Conditions (cloud)
01:56:44     Target Aborted (4/6) Target aborted (4/6, AT2022jrp (CV)) due to cloud
01:56:46   Queue Paused Queue paused for Session Suspension
01:56:48 Session Suspended Session suspended
01:57:29   Dome Closed Dome closed (closing time 45s)
01:58:59     Target Cancelled Target cancelled (4/6, AT2022jrp (CV)) due to cloud
02:12:12     Target Cancelled Target cancelled (5/6, GCVS U Her) due to cloud
02:16:26 Session Resuming Session resuming
02:17:12   Dome Opened Dome opened (opening time 45s)
02:17:14 Session Running Session running
02:17:16   Queue Resumed Observing Queue resumed
02:17:19     Target Started (6/6) Target started (6/6, GCVS AM Her)
02:26:50     Target Completed Target completed (6/6, GCVS AM Her)
02:26:54   Queue Completed Job Queue completed
02:26:58 Session Closing Session closing
02:27:53   Dome Closed Dome closed (closing time 50s)
02:28:36   Dome Parked Dome parked (parking time 36s), Az: 90.0 deg
02:29:57   Telescope Parked Telescope parked (parking time 79s)
02:30:13   Telescope State Scope parked Turn scope off. (Handbox)
02:30:35   Telescope Switched Off Telescope Power has been switched off via UPB Switch.
02:31:30   Services Stopped Night Services stopped
02:31:32 Session Housekeeping Housekeeping Started (Cleanup FITS, Create Fits Summary, Transfer Files)
02:31:38 Session Finished Session Finished
 
Session Alerts
Time     Alert Detail
01:09:27 Dome Discrepency Difference between Telescope Az and Dome Az is > 8 deg (GCVS SS Cyg)
 

Back to Top


Operational Issues (2022-05-31, S1014)

[ Prev | Next ]

Critical Issues

Major Issues

Minor Issues

Small Defects

Continuous Improvement

[ Prev | Next ]

Back to Top


Investigation -  Issue Identifying End Of Dome Slew (SS Cyg, 1:08)

Issue 
Problem with Dome Slew to Target 1/6 (SS Cyg) - either an actual problem with the slew or a slew monitoring/reporting problem.

Description
A 'Dome Discrepency' alert was issued at 01:09:27 "Difference between Telescope Az and Dome Az is > 8 deg (GCVS SS Cyg)" and the ObsPic on Current Session page showed the dome apparently not aligned with the telescope and apparently still 'slewing'. The Report file showed the message "Az: 171.8° (Dome)   ALERT. 99.7° difference from Scope Az" , followed a few moments later by "Dome Az. 157.5° (vs. Scope Az. 72.1°, Delta Az. 85.4°)"

The session's slew table reports that the dome slew to SS Cyg at 01:08 took 12s to slew from Az 203.1° to Az 171.8°  (an angle of 31.3°) at an anomously high speed of 2.57°, whilst the scope slew took 32s to slew from Az 208.6°  to Az 72.1°.  (overall slew time 32s).  The following dome slew to BL Lac at 01:18 was reported to have started from Az. 76.0°.

Analysis
The changing Dome Az (from 203.1° to Az 171.8°, then to 157.5° and then to destination Az of ~ 76° suggests that scope was still slewing after the Scope/Dome Slew monitor had assessed that the slew had been completed, and was physically normal.

Centering operations, and in particular Locate Frame 1 that followed directly after the slew operations, were normal (dome had evidently arrived at the required slaved location for SS Cyg target).  This confirms that the dome slew was physically normal.   

It would seem that the problem lies in the slew monitoring routine which has somehow failed to identify the correct end of dome slew.  This would be due to a flaw in the routine's logic or a glitch in the data being sent to the client from the Pulsar Dome driver.

Example Time Line associated with a normal slew

01:27:05.28  Client commands TheSky6.Telescope.SlewToRaDec (Asynchronous) to slew to NGC 4088
             (Scope at Az: 73.1°, Scope needs to move to Az 292.9°, Dome at Az 75.0°, Dome needs to move to Az 289.8°)
01:27:05.32     Dome commanded to Slew To Azimuth  289.7° 
                (Here DH is clearly commanding the dome to slew to the appropriate destination Az)
01:27:05.39     Dome slewing started Az 75.0°   (Dome.Slewing = True)

01:27:08.34   Scope slewing started Az: 73.1° (Scope.Slewing = True)
01:27:52.75   Scope slew finished Az: 292.8°  (Scope.Slewing = False)
01:28:20.27      Dome slewing finished, Az 289.8°  (Dome.Slewing = False)

This timeline is consistent with that expected from a ‘Best Way’ configuration. 
The scope and dome are sent slew commands at the same time.  The Dome Slew is clearly given the destination Az. 289.7°,  and this occurs even before the scope is perceived to have actually started to move.

Time Line associated with anomalous slew (SS Cyg, 01:08)

01:08:48.65   Client commands TheSky6.Telescope.SlewToRaDec  (Asynchronous) to slew to SS Cyg
                     (Scope at Az: 208.6°,  Scope needs to move to Az 72.1°,  Dome at Az 203.1, Dome needs to move to Az 75.0°)
01:08:51.93   Scope starting to slew, Az 208.5° (Scope.Slewing = True)
01:08:52.99   Scope slewing at Az 195.1°  
01:08:53.74     Dome commanded to Slew To Azimuth 192.5°
                (DH is responding to the result of periodic slaving calculation,
                 there isn’t the expected command to slew dome to the destination Az)
01:08:53.84   Scope slewing at Az: 176.6°
01:08:54.16      Dome starting to slew Az 202.8°  (Dome.Slewing = True)
01:08:57.20      Dome slewing at Az 196.3°
01:09:00.20      Dome slew finished,  Az 192.5°   (Dome.Slewing = False)
01:09:05.45   Scope slewing at Az 86.9°
01:09:08.43   Scope slewing at Az 76.0°
01:09:10.53   Scope slewing at Az 72.3°
01:09:10.77      Dome commanded to Slew To Azimuth  75.0°          
                 (DH is either responding to the result of a periodic slaving calculation
                 or this is a significantly delayed command to slew to destination Az)
01:09:11.32      Dome slewing at Az 202.8  (Dome.Slewing = True)    
01:09:20.24   Scope finished slew Az 71.1°  (Scope.Slewing = False)
01:09:20.44      Dome slewing at Az 171.8°
01:09:25.45      Dome slewing at Az 160.0°
01:09:40.57      Dome slewing at Az 126.2°
01:09:45.77      Dome slewing at Az 114.5°
01:10:03.85      Dome slewing at Az 89.0° 
                 (dome movement pauses at Az 89.0° for 36-40s,
                  known issue - wheels spinning on same spot till traction re-gained)
01:10:43.45      Dome slewing at Az 88.8°              
01:10:50.33      Dome slewing at Az 78.2°
01:10:53.35      Dome slew finished, Az 75.0° (Dome.Slewing = False)

The Dome-Scope Configuration used in the Observatory is shown in the following diagram, along with the data flow paths associated with a slew to target

Image

This a "Best Way" configuration and would normaly see both the telescope and the slaved dome being sent a slew command froom DeviceHub at the same time when a slew is initiated by DH or by an Ascom client connected to DH.Telescope (and Dome.Slaved = True).  This should mean that both devices should begin to slew directly to their new target position within a few 10ths of a second of each other. Both devices should move directly to their new destination in a single move.

Whilst most of slews requests (initiated from AstroMain via TheSky6/TeleApI/DH.Telescope show this expected behaviour, in at least the one anomalous slew case documented above this behaviour obviously didn't occur. Instead the dome 'followed' the scope movement, didn't occur in a single move and caused the dome azimuth to not arrive at the appropriate destination azimuth until some time after the scope has finished its slew.

Conclusion

 It would seem that Scope-Dome system works as intended for most of the time, but at least in one case documented it didn't. The hypothesis to explain this anomaly is as follows :-

Actions

Image Document Slew Anomaly (SS Cyg) and forward description of the problem and hypothesis to the author of the DeviceHub program for review/comment.
Image Modify AstroMain to handle the slew behaviour that occured for the slew to SS Cyg.

Update 2022-06-03
Whilst reviewing the Dome Slew Monitoring code in the SlewTheSkyScope() routine it was found that a 20s cutoff had been added 2022-05-10 (in response to MI, S995 2022-03-25) in the code for reset SlewStopped Flag if dome is slewing again.   Because the resumed dome slew to Az 75° only happened after 23s the opportunity to lower the SlewStoppedFlag and bDomeCompleted flags was lost, with the consequence that slew monitoring was exited before dome was in the correct position for the target. This is fixed in AstroMain 3.51.13.

Update 2022-06-04
For further diagnosis of this issue it is critical that DeviceHub ActivityLog is running and captures Dome & Scope Commands at the time of a similar event.  DH will be setup at start and end of each new session to capture ActivityLog message from each session in order to have information on the commands & any exceptions relevant to a future instance of this issue.

Session S1016 and S1017 both had one instance of an anomalous dome slew where the dome slew stopped part way from the slew to Target., but didn't effect the follow-on operation as Scope/Dome slew monitoring ended at appropriate point.  In both cases the Anomalous Slew was associated with slew to Focus Field target as the beginning of the session. After turning-on Slaving the dome began still slewing towards the telescope which was sitting close to its startup position (Az ~180°).  This slew was still underway when Scope was commanded to slew to the Focus Field target.  In S017 an attempt was made to abort this already active dome slew before commanding the slew to Focus Field, but process didn't wait longer enough for the dome to be confirmed as no longer slewing.

Update 2022-06-05
AstroMain 3.51.15 version built with updated fixes for this issue.

Update 2022-06-09
New version of DeviceHub installed (DeviceHub 6.6.0.11) which contains a fix for an issue with DH's dome slewing interruption that may have caused the anomalous dome slew seen during slew to SS Cyg (01:08).

Back to Top