David's Astronomy Pages
Notes - Session 1391 (2026-01-15)

 
Bullet Session Aims & Highlights
 - Observing Result
 - Night Summary Plot
 - Session Event Log
Bullet Operational Issues
- Critical Issues (0),  Major Issues (0),  Minor Issues (3),  Small Defects (3),  Continuous Improvement (5)
- Figures (0)
   
Bullet Images from 2026-01-15  >>           
   
2026-01-16
Bullet AstroMain - Further Refactoring, Obs Manager's MainLoop
   

Session Aims & Highlights (2026-01-15)

Main aims

  1. Targets.  Acquire images of a selection of variable stars, nearby stars, comets & deep sky targets as allowed by sky conditions.
  2. AstroMain. Check overall stability and capability of AstroMain 3.80.2 following significant software changes made in during the last fortnight, including introrduction of a new StateMachine (based on restructure of existing SessionState code blocks
  3. Session StateMachine. Carry out first live Observatory session using AstroMain/ObsManager's new Session StateMachine.
  4. Active Session Pipeline. Carry out second live Observatory session using AstroMain's new Create/Reopen Session pipeline that has had modification/fixes since the first live session

 Equipment & Software

Highlights

NotesNotes

Summary Plots & Logs

Observing Plan
Image
  
Observing Result
Image
   
Image
 
Dome & Scope Slewing Performance
Image
  
Slew/Centering Performance
Image
  
Guiding Performance
Image
  
Sky Conditions (Locate Frames)
Image 
  
Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)
Note:  The NorthCam Star Count line is unreliable and should be ignored
(The AstroNorthCam 'Find' Star method is producing many false positives from clouds)
Image   
  
Actual Weather vs Pre-Session Weather Forecast
Image
Image
 
Session Event Log
Time     Event Detail
18:05:10 Session Monitoring AutoStart monitoring for Live Session opportunity between 18:05 & 06:37
18:05:13 Session AutoStarting Session autostarting (18:05)
18:05:23 Session Created Session Created (Live, 2026-01-15 S01391)
18:05:25   CCDSoft Restarting CCDSoft being restarted (to set AutoSave No.)
18:05:45   Scope Switched On Telescope Power has been switched on via UPB Switch
18:06:42 Observatory (Manual) Observatory placed in Manual Mode
18:07:32   Services Started Observatory Services started
18:07:42 Observatory (Auto) Observatory placed in Fully-Automated Mode
18:07:46 Session Pending Session pending (2026-01-15)
18:07:48 Session Initiating Session initiating (2026-01-15)
18:08:10   Camera1 Connected SBIG Camera connected (Set point -20°C)
18:08:13   Plan Requested Observing Plan requested from AstroPlan (1.46)
18:08:55   Plan Loaded Observing Plan loaded to queue (Plan ID: 1181)
18:09:00 Session Resumed Session Resumed (Live, 2026-01-15 S01391)
18:09:34   Telescope Connected Telescope connected (TheSky6)
18:09:59 Session Equilibration Session ready to Open Dome
18:10:44   Dome Opened Dome opened (Opening time 45s, Zigbee 41s)
18:11:14 Session Running Session running
18:11:17   Queue Started Observing Queue started (43 targets selected)
18:11:20     Target Started (NrZen) Target started (Focus Field 2, HIP 9508)
18:11:34       Dome Unparked Dome unparked
18:14:56       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
18:17:32       Focusing Completed Foc1 AutoFocus Completed (Profile No 1, wide)
18:19:44       Focusing Completed Foc1 AutoFocus Completed (Profile No 1)
18:19:47       Focusing Started-Foc2 Foc2 Focusing Started (Secondary Scope, using ShCap)
18:21:45       Focusing Completed Foc2 AutoFocus Completed (Profile No 2, wide)
18:23:36       Focusing Completed Foc2 AutoFocus Completed (Profile No 2)
18:23:57     Target Completed (NrZen) Target completed (Focus Field 2, HIP 9508)
18:24:41     Target Started (1/43) Target started (1/43, C/2022 E2 (ATLAS))
18:29:01       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
18:31:41       Focusing Completed Foc1 AutoFocus Completed (Profile No 3)
18:40:13     Target Completed Target completed (1/43, C/2022 E2 (ATLAS))
18:40:17     Target Started (2/43) Target started (2/43, GCVS BL Lac)
18:43:41       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
18:46:12       Focusing Completed Foc1 AutoFocus Completed (Profile No 4)
18:50:02     Target Completed Target completed (2/43, GCVS BL Lac)
18:50:06     Target Started (3/43) Target started (3/43, NGC 100 w/AT2025aigy)
18:54:22       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
19:10:27     Target Completed Target completed (3/43, NGC 100 w/AT2025aigy)
19:11:49     Target Started (4/43) Target started (4/43, LEDA 3092696 w/SN2025nzh)
19:15:49       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
19:18:31       Focusing Completed Foc1 AutoFocus Completed (Profile No 5)
19:34:32     Target Completed Target completed (4/43, LEDA 3092696 w/SN2025nzh)
19:34:37     Target Started (5/43) Target started (5/43, AT2025agqo)
19:38:19       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
19:40:42       Focusing Completed Foc1 AutoFocus Completed (Profile No 6)
19:57:32     Target Completed Target completed (5/43, AT2025agqo)
19:57:36     Target Started (6/43) Target started (6/43, GCVS HH And)
20:01:52       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
20:04:19       Focusing Completed Foc1 AutoFocus Completed (Profile No 7)
20:06:12     Target Completed Target completed (6/43, GCVS HH And)
20:06:16     Target Started (7/43) Target started (7/43, GCVS Z And)
20:09:24       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
20:11:47       Focusing Completed Foc1 AutoFocus Completed (Profile No 8)
20:15:11     Target Completed Target completed (7/43, GCVS Z And)
20:15:15     Target Started (8/43) Target started (8/43, M31 w/AT2025agmm)
20:18:32       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
20:34:39     Target Completed Target completed (8/43, M31 w/AT2025agmm)
20:36:35     Target Started (9/43) Target started (9/43, 240P/NEAT)
20:40:13       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
20:42:28       Focusing Completed Foc1 AutoFocus Completed (Profile No 9)
20:53:00     Target Completed Target completed (9/43, 240P/NEAT)
20:53:19     Target Started (10/43) Target started (10/43, UGCA 87)
20:57:04       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
20:59:38       Focusing Completed Foc1 AutoFocus Completed (Profile No 10)
21:15:40     Target Completed Target completed (10/43, UGCA 87)
21:15:45     Target Started (11/43) Target started (11/43, GCVS NV Ori)
21:19:14       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:21:31       Focusing Completed Foc1 AutoFocus Completed (Profile No 11)
21:24:40     Target Completed Target completed (11/43, GCVS NV Ori)
21:25:53     Target Started (12/43) Target started (12/43, Minkowski 1-7)
21:29:26       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
21:31:41       Focusing Completed Foc1 AutoFocus Completed (Profile No 12)
21:54:35     Target Completed Target completed (12/43, Minkowski 1-7)
21:54:39     Target Started (13/43) Target started (13/43, AT2025agqu)
21:57:59       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
22:00:12       Focusing Completed Foc1 AutoFocus Completed (Profile No 13)
22:16:29     Target Completed Target completed (13/43, AT2025agqu)
22:16:33     Target Started (14/43) Target started (14/43, GCVS YZ Cnc)
22:19:56       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
22:22:24       Focusing Completed Foc1 AutoFocus Completed (Profile No 14)
22:29:39     Target Completed Target completed (14/43, GCVS YZ Cnc)
22:29:43     Target Started (15/43) Target started (15/43, GCVS TV Cas)
22:34:15       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
22:36:25       Focusing Completed Foc1 AutoFocus Completed (Profile No 15)
22:38:23     Target Completed Target completed (15/43, GCVS TV Cas)
22:38:27     Target Started (16/43) Target started (16/43, AT2025adhw)
22:42:06       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-LSI)
22:58:14     Target Completed Target completed (16/43, AT2025adhw)
23:00:07     Target Started (17/43) Target started (17/43, AT2024sje)
23:03:52       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:06:27       Focusing Completed Foc1 AutoFocus Completed (Profile No 16)
23:22:45     Target Completed Target completed (17/43, AT2024sje)
23:22:50     Target Started (18/43) Target started (18/43, Minkowski 1-8)
23:52:19     Target Completed Target completed (18/43, Minkowski 1-8)
23:52:23     Target Started (19/43) Target started (19/43, GCVS RR Tau)
23:56:14       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
23:58:34       Focusing Completed Foc1 AutoFocus Completed (Profile No 17)
00:04:14     Target Completed Target completed (19/43, GCVS RR Tau)
00:04:18     Target Started (20/43) Target started (20/43, GCVS U Gem)
00:07:55       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
00:10:34       Focusing Completed Foc1 AutoFocus Completed (Profile No 18)
00:13:37     Target Completed Target completed (20/43, GCVS U Gem)
00:13:41     Target Started (21/43) Target started (21/43, AT2023cwd (CV))
00:18:41       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
00:24:51     Target Completed Target completed (21/43, AT2023cwd (CV))
00:24:56     Target Started (22/43) Target started (22/43, AT2025ajwu)
00:28:41       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
00:45:20     Target Completed Target completed (22/43, AT2025ajwu)
00:46:32     Target Started (23/43) Target started (23/43, UGCA 142)
00:50:56       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
01:07:39     Target Completed Target completed (23/43, UGCA 142)
01:08:52     Target Started (24/43) Target started (24/43, UGCA 200)
01:13:13       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
01:15:58       Focusing Completed Foc1 AutoFocus Completed (Profile No 19)
01:38:33     Target Completed Target completed (24/43, UGCA 200)
01:38:38     Target Started (25/43) Target started (25/43, NGC 4653 w/SN2025mqc)
01:42:24       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
01:59:20     Target Completed Target completed (25/43, NGC 4653 w/SN2025mqc)
01:59:24     Target Started (26/43) Target started (26/43, M99 w/AT2026X)
02:03:04       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
02:20:28     Target Completed Target completed (26/43, M99 w/AT2026X)
02:21:24     Target Started (27/43) Target started (27/43, NGC 3124)
02:25:39       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
02:41:48     Target Completed Target completed (27/43, NGC 3124)
02:43:44     Target Started (28/43) Target started (28/43, GCVS RR Leo)
02:46:37       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
02:48:58       Focusing Completed Foc1 AutoFocus Completed (Profile No 20)
02:54:59     Target Completed Target completed (28/43, GCVS RR Leo)
02:55:55     Target Started (29/43) Target started (29/43, GCVS RZ LMi)
02:59:23       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
03:01:41       Focusing Completed Foc1 AutoFocus Completed (Profile No 21)
03:07:01     Target Completed Target completed (29/43, GCVS RZ LMi)
03:07:33     Target Started (30/43) Target started (30/43, GCVS CN Leo)
03:12:13       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
03:14:37       Focusing Completed Foc1 AutoFocus Completed (Profile No 22)
03:22:23     Target Completed Target completed (30/43, GCVS CN Leo)
03:22:27     Target Started (31/43) Target started (31/43, IC 700 w/SN2025aico)
03:25:48       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
03:28:12       Focusing Completed Foc1 AutoFocus Completed (Profile No 23)
03:44:11     Target Completed Target completed (31/43, IC 700 w/SN2025aico)
03:44:16     Target Started (32/43) Target started (32/43, NGC 3887)
03:48:04       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
04:04:27     Target Completed Target completed (32/43, NGC 3887)
04:05:35     Target Started (33/43) Target started (33/43, NGC 3464)
04:09:55       Focusing Skipped Foc1 focusing skipped - star is too dim (TCF-LSI)
04:26:22     Target Completed Target completed (33/43, NGC 3464)
04:27:55     Target Started (34/43) Target started (34/43, 29P/Schwassmann-Wachmann)
04:32:08       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-LSI)
04:47:58     Target Completed Target completed (34/43, 29P/Schwassmann-Wachmann)
04:49:09     Target Started (35/43) Target started (35/43, NGC 5033 w/SN2025mvn)
04:52:44       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-LSI)
05:09:24     Target Completed Target completed (35/43, NGC 5033 w/SN2025mvn)
05:11:09     Target Started (36/43) Target started (36/43, MCG +7-28-15 w/SN2025adje)
05:14:25       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
05:17:05       Focusing Completed Foc1 AutoFocus Completed (Profile No 24)
05:33:37     Target Completed Target completed (36/43, MCG +7-28-15 w/SN2025adje)
05:33:42     Target Started (37/43) Target started (37/43, GCVS R CrB)
05:36:27       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
05:38:33       Focusing Completed Foc1 AutoFocus Completed (Profile No 25)
05:41:27     Target Completed Target completed (37/43, GCVS R CrB)
05:43:10     Target Started (38/43) Target started (38/43, CGCG 137-080 w/SN2025zpw)
05:45:55       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
05:48:55       Focusing Completed Foc1 AutoFocus Completed (Profile No 26)
06:05:13     Target Completed Target completed (38/43, CGCG 137-080 w/SN2025zpw)
06:05:17     Target Started (39/43) Target started (39/43, T CrB)
06:08:14       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
06:09:48       Focusing Failed Foc1 focusing failed - star lost
06:14:00     Target Completed Target completed (39/43, T CrB)
06:16:07     Target Started (40/43) Target started (40/43, AT2025sdf (CV))
06:20:45       Focusing Skipped Foc1 focusing skipped - star is lost (TCF-LSI)
06:26:52     Target Completed Target completed (40/43, AT2025sdf (CV))
06:28:07     Target Started (41/43) Target started (41/43, HIP 87937)
06:30:20       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
06:32:44       Focusing Completed Foc1 AutoFocus Completed (Profile No 28)
06:36:05     Target Completed Target completed (41/43, HIP 87937)
06:36:29     Target Started (42/43) Target started (42/43, GCVS CY Lyr)
06:39:53       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
06:42:21       Focusing Completed Foc1 AutoFocus Completed (Profile No 29)
06:46:10     Target Completed Target completed (42/43, GCVS CY Lyr)
06:46:47     Target Started (43/43) Target started (43/43, GCVS AM Her)
06:47:00       Dome Unparked Dome unparked
06:50:15       Focusing Started-Foc1 Foc1 Focusing Started (TCF-S)
06:52:54       Focusing Completed Foc1 AutoFocus Completed (Profile No 30)
06:58:18     Target Completed Target completed (43/43, GCVS AM Her)
06:58:20   Queue Completed Job Queue completed
06:58:24 Session Closing Session closing
06:59:19   Dome Closed Dome closed (Closing time 51s, Zigbee 44s)
06:59:36   Dome Parked Dome parked (parking time 10s), Az: 90.0 deg
07:00:25   Telescope Parked Telescope parked (parking time 43s total)
07:00:49   Telescope State Handbox reads 'Scope parked Turn scope off.'
07:01:51   Telescope Switched Off Telescope Power has been switched off via UPB Switch.
07:03:16   Services Stopped Night Services stopped
07:03:19 Session Finishing Session Finishing started (Create Fits Summary, Transfer Files)
07:05:21     Dome (Find Park) Find Park started (Search Az 88.0 to 92.0°, Step 0.2°, Narrow)
07:06:07     Dome Find Park found (Best Park Az 90.6°)
07:06:14     Dome Dome has been synced at park position (Az 90.0°, Adjustment: -0.6°)
07:06:16   Dome Parked Dome parked
07:09:15   Dome Charging Dome hasn't started re-charging after 3.0 minutes
07:09:17 Session Finished Session Finished
07:09:25 Obs.Manager Obs.Manager is frozen in 'Finished' (last cycle was at 07:09:25)
07:11:28   Make Observatory Safe Obs.Overseer intervened to Make Observatory Safe
08:48:14 Observatory (Manual) Observatory placed in Manual Mode
 
Session Alerts & Alarms
Time     Type       Name Detail
18:06:49 Red Alert Program Hung AstroMain UI appears to have stopped responding at 18:06:43
18:06:54 Green Clear Program Resumed AstroMain UI has resumed again after 11s
07:09:15 Yellow Alert Dome Charging Dome hasn't started re-charging after 3.0 minutes
07:11:28 Red Alert Obs.Manager Obs.Manager is frozen in 'Finished' (last cycle at 07:09:25)
 

Back to Top


Operational Issues (2026-01-15 S1391)

[ Prev | Next ]

Critical Issues

Major Issues

Minor Issues

Small Defects

Continuous Improvement

[ Prev | Next ]

Back to Top Issue,  Back to Top


2026-01-16


AstroMain - Refactoring Obs Manager's Main Loop

1. Introduction

The ObsManager MainLoop is the core automation engine of the AstroMain Observatory Control Program. Every five seconds, throughout each night, it evaluates the state of the telescope, dome, weather, power systems, job queue, and the automation state machine. Over many years of incremental development, this loop grew into a 7,750-line monolith: reliable, but increasingly difficult to maintain or extend.

A major architectural refactor of this MainLoop is required. The goal was not to change behaviour, but to transform the structure into something modular, readable, and future-proof. This report summarises the motivation, analysis, implementation, and future directions of this work.


2. Analysis

2.1 The Legacy Situation

The original MainLoop() routine had evolved organically over more than a decade. It contained tightly interwoven logic, nested conditionals, multiple implicit state machines, and a mixture of hardware polling, safety checks, and automation logic. Although the system was stable and battle-tested, the structure made it increasingly risky to modify and difficult to reason about.

2.2 Identifying Domains

A detailed review showed that the monolith actually consisted of 73 distinct functional blocks, each responsible for a specific aspect of observatory operation. These blocks naturally grouped into domains such as:

In the legacy MainLoop, these responsibilities were interleaved, which obscured intent and made the control flow hard to follow.

2.3 Ordering and Dependency Considerations

The MainLoop runs on a five-second cycle, with three to four seconds typically spent in sleep. Most changes in the observatory environment occur during this sleep period, so the ordering of domains determines which subsystem gets first sight of those changes.

By analysing dependencies and reaction-time requirements, an ordering emerged that reflects the observatory’s real-world priorities:

This ensures that time-critical and safety-critical domains run first, while low-priority domains run later in the cycle.


3. Implementation

3.1 Domain Extraction

Each of the 73 legacy blocks was extracted into a dedicated routine with a clear, intention-revealing name. Examples include:

The internal logic of these routines was lifted directly from the legacy MainLoop, preserving timing, sequencing, and side-effects.

3.2 Domain Orchestrator

The refactored MainLoop is now a simple, readable orchestrator:

DoStart()

DoTimeAndSession()
DoEnvironmentAndSafety()
DoTelescope()
DoDomeAndShutter()
DoSubsystemHealth()
DoOutageManagement()
DoAutomationPipeline()
DoStateMachineStep()
DoAuxillaryDevices()

DoSleep()

This structure makes the flow of each cycle immediately understandable and allows domains to be reordered or extended without deep surgery.

3.3 Preserving Multi-Stage State Machines

Several behaviours in the legacy system were implemented as multi-cycle state machines, including shutter recovery and mains outage management. These were preserved by extracting them into explicit stage handlers driven by a stage counter, for example:

If bUnderOutageManagement Then
    Select Case PowerOutageStageNo
        Case 0 : DoOutage4_Stage0()
        Case 1 : DoOutage4_Stage1()
        Case 2 : DoOutage4_Stage2()
        Case 3 : DoOutage4_Stage3()
        Case 4 : DoOutage4_Stage4()
        Case 5 : DoOutage4_Stage5()
        Case 6 : DoOutage4_Stage6()
    End Select
End If

This preserves the original multi-cycle behaviour: one stage per loop, no blocking, and no accidental cascading through multiple stages in a single cycle.

3.4 Behavioural Fidelity

A strict principle guided the refactor: no behavioural changes, only structural improvements. All safety logic, timing, and decision-making remain as before. The observatory continues to operate with the same behaviour, but on a much clearer and more maintainable foundation.


4. Further Work

4.1 Internal Cleanup of Extracted Routines

Many extracted routines still contain long internal blocks. These can now be refactored incrementally into smaller leaf routines without touching the MainLoop or changing external behaviour.

4.2 Enhanced Telemetry and Logging

With domains clearly separated, it is now easier to add structured logging and telemetry at the domain level, improving diagnostics and post-session analysis.

4.3 Improved Error Handling and Recovery

Subsystem Health and Outage Management can be extended with richer diagnostics, clearer classification of failures, and more nuanced recovery strategies, all within their own domains.

4.4 Optional Parallelisation

Some low-priority domains, such as auxiliary devices, could be moved to background threads in future, without affecting the deterministic core of the MainLoop.

4.5 Domain-Level Testing

The new structure makes it feasible to test domains independently, improving confidence in future changes.


5. Technical Appendix

5.1 Final Domain Ordering

The final domain ordering in the refactored MainLoop is:

5.2 Example Domain Wrapper

An example of a domain wrapper in the new structure:

Private Sub DoDomeAndShutter()
    DoDome1_DomeMonitoring()
    DoDome2_GetShutterState()
    DoDome3_CheckShutterState()
    DoDome4_ReviewDomeLock()
    DoDome5_ResolveCloseShutterIssue()
    DoDome6_ResolveCloseShutterIssue1_TakeObsCamImage()
    DoDome7_ResolveCloseShutterIssue2_WaitOnObsCamImage()
    DoDome8_ResolveCloseShutterIssue3_OpenShutter()
    DoDome9_ResolveCloseShutterIssue4_CloseShutter()
    DoDome10_ResolveCloseShutterIssue5_WaitOnShutter()
    DoDome11_EnsureShutterClosed()
    DoDome12_CheckShutterBattery()
    DoDome13_DomeRecordMonitoring()
End Sub

Each domain is now a self-contained module, making the overall system easier to understand, maintain, and extend.