David's Astronomy - Notes 2022-07-26 (S1022)

Session Aims & Highlights

- Observing Result
- Night Summary Plot
- Session Event Log

Operational Issues

- Critical Issues (0), Major Issues (0), Minor Issues (4), Small Defects (2), Continuous Improvement (10)

Images from 2022-07-26 >> [ Local Files >> ]

2022-07-27

Missing Observing Opportunity (2022-07-27)

2022-07-28

Investigation - PHD2 Client/Server Comms Issues

2022-07-27

An observing opportunity was missed on night 2022-07-27. Conditons around dusktime looked hopeful but whilst Weather Forecasts (both DarkSky.net and xcweather.co.uk) indicated dry conditions (no rain) the cloud cover was expected to be poor (90-100% cover), and with AllSky Images indicated moving cloud with only minor gaps at 23:00 (sun alt -8.5°) it was decided not to put the observatory into autostart mode. In the event skies cleared at around 00:00 (~ sun alt-12°) and stayed clear until at least 02:30 (~ sun alt -12°).

Missed Observing Opportunity 2022-07-27

Night Sky Summary Plot
Top axis: Sky Brightness at Zenith (in ADU/s)
Lefthand axis: Local Time (hh LT). Righthand axis: Sun Altitude (degs)

Actual Weather vs Pre-Session Weather Forecast

2022-07-28

Issues

1) Since session In order to prevent issues with client/server data communications with PHD2 (where multiple incomplete PHD2 message reads occurs) a workaround has been used in which the AstroMain program is always freshly restarted between Observing Sessions. Is there preventing issues without having to perform this pre-session restart ?

This is not normally an inconvenience since AstroMain is typically restarted anyway between sessions in order to install a new version. However it would be nice to close PHD2 client/server comms robustly in a way so that AstroMain doesn't necessarily have to be restarted prior to a new session.

2) Incomplete PHD2 data stream read. can indicate a problem with client/server data communications. Can there presence be used to automatically restart PHD2 program and restart client connection ?

Fixes where made to AstroMain so that incomplete PHD2 data stream reads were recognised and result/event messages built up by appending data from following data stream reads, and the presence of these incomplete data stream reads counted, and then flagged up to warn/alert the user, as they can be indicative of a problem with client/server data communications that can compromise the generation of guiding charts and the production of guiding stats. Whilst problem of incomplete data stream reads seems to be associated in the past with occassions when AstroMain wasn't freshly restarted ahead of a new sessions, the problem could potentially appear at any time. It would be handy if AstroMain could automatically resolve the problem itself. Whilst server response and event messages are recoverable it situations where there is an incomplete data stream read, communications seem to be slowed down considerably and this affects quality of guiding charts/guiding stats.

A continuous improvement ticket was raised at time of S961 session to investigate ways to mitigate/correct INCOMPLETE Phd2 Data Stream reads, but has remained to now as an outstanding/non-completed task.

3) In session S942 (2021-12-16) the PHD2 exposure looping/guiding routines hung half-way through the night due to an unhandled & unknown program exception in PHD2 and led to all the remaining targets being unguided. Could AstroMain have intervened and attempted to fix problem by killing/restarting PHD2 ?

The incident during S942 was investigated at the time but was unable to find a specific reason behind the exception in PHD2 ( See Investigation - PHD2 application failed during guiding (unhandled and unknown exception) ). A continuous improvement ticket was raised to add monitoring of PHD2 to AstroMain to provide the potential opportuntity to automatically kill/restart PHD2 in the event that the PHD2 application appears to have stalled, but has remained to now as an outstanding/non-completed task.

Analysis

Problems have occurred with PHD2 communications in sessions S951, S953, S958, S961, S965 & S972. Whilst code has been previously changed to deal with 'Incomplete PHD2 data stream reads', in session S972 it was reported that this is still an occasional issue, which is resolved by closing and restarting AstroMain. It is uncertain if this is still a neccessary/recommended action prior to a new session or not.

Reviewing the code associated with PHD2 Service / PHD2 Client, the principal routines are:

PHD2.StartClient
Client created and connected using
Client = New TcpClient
Client.Connect(ServerAddress, PortNumber)

' once connected

DataStream = Client.GetStream()

PHD2.StopService
At end of an observing session the PHD2.StopService() is called, this
- ensures that PHD2 guiding is stopped , by using the 'stop_capture' method
- disconnects equipment attached to the current PHD2 profile by using the 'set_connected' (params: [false] method
- stops the client listening loop, by setting the bStopListening false to True
- waits 500ms to allow the Listening Loop to exit (but no feedback)
- disconnects the PHD2 Client by calling PHD2.StopClient

PHD2.StopClient
The StopClient() routine does the following
- closes the datastream (NetworkStream) with DataStream.Close()
- closes the client (TCPClient) with Client.Close()
- sets datastream to Nothing
- sets client to Nothing

DataStream.Close() closes the current stream and releases any resources (such as sockets and file handles) associated with the current stream. Instead of calling this method, ensure that the stream is properly disposed (using Datastream.Dispose() )

Client.Close() disposes theTcpClient instance and requests that the underlying TCP connection be closed. Calling this method will eventually result in the close of the associated Socket and will also close the associated NetworkStream that is used to send and receive data if one was created.

Requirement
- A test routine that can replicate the issue with PHD2 client/server data communications.
- A revised method of closing PHD2 Client/Server that fixes the issue with client/server data communications when starting a new session without restarting AstroMain
- Rerunning of test routine to demonstrate that issue has been fixed.

Test Routine
Using On-The-Fly Observing without Slew&Centre, but with 'AutoGuide' should provide an appropriate test of PHD2 client/server comms.

Select Telescope Service
Select PHD2 Service
Start Services

Run 'test routine' on-the-fly imaging of M67 with guiding

Stop Night Services

Select Telescope Service
Select PHD2 Service
Start Services

Run 'test routine' on-the-fly imaging of M67 with guiding

Stop Night Services (including PHD2.StopService / PHD2.StopClient )

Change Session

Select Telescope Service
Select PHD2 Service
Start Services

Run 'test routine' on-the-fly imaging of M67 with guiding

Unable to replicate issue with PHD2 client/server data communications where there are incomplete PHD2 reads.

Actions

1) Code Changes to Start/Stop PHD2 Client

Code changes made routine for starting/stopping PHD2 Client on 2022-07-28 (AstroMain 3.53.1).

StartClient() routine
DataStream = Client.GetStream() moved ahead of StartListening(), and delay added to ensure that Listener thread has fully initialised :

DataStream = Client.GetStream() StartListening() Thread.Sleep(500)

StopClient() routine
The following lines removed, based on .Net description of TcpClient.Close() method, which closes the datastream if there is one and disposes of the client.

' DataStream.Close() ' DataStream = Nothing ' Client = Nothing

2) Automatical restart of PHD2
Implemented 2022-07-30 (AstroMain 3.53.1) by looking for nIncompletePHD2reads > 5 and invoking RestartPhd2ProgramAndService() routine from OB.ExecuteTarget() before initiating guiding.

Previously it was was found that restarting PHD2 without restarting AstroMain in session S961 failed to resolve the problem of Incomplete PHD2 data stream reads, but it is hoped that new 'RestartPhd2ProgramAndService()' routine works better than past manual invoked methods.

3) Monitoring for Stalled PHD2 Program
Eventually implemented 2022-07-29 (AstroMain 3.53.1) by looking for the absence of any "LoopingExposures" events after calling PHD2' s "guide" method and then invoking RestartPhd2ProgramAndService() routine from OB.ExecuteTarget() routine before making a second attempt to initiate guiding.

4) Merging Incomplete PHD2 reads.
Investigate 3s delay and optimise. (from S961)

This Web Page:

Notes - Session 1022 (2022-07-26)

Last Updated :

2024-03-22

Site Owner :

David Richards

Home Page :

David's Astronomy Web Site

David's Astronomy Pages
Notes - Session 1022 (2022-07-26)

Session Aims & Highlights (2022-07-26)

Operational Issues (2022-07-26, S1022)

2022-07-27

Missed Observing Opportunity (2022-07-27)

2022-07-28

Investigation - PHD2 Client/Server Communication Issues

David's Astronomy Pages Notes - Session 1022 (2022-07-26)

Session Aims & Highlights (2022-07-26)

Operational Issues (2022-07-26, S1022)

2022-07-27

Missed Observing Opportunity (2022-07-27)

2022-07-28

Investigation - PHD2 Client/Server Communication Issues

David's Astronomy Pages
Notes - Session 1022 (2022-07-26)