guorbit / obc-firmware Goto Github PK

#include <msp430.h>

As shown in 3cb6756 on bug/szy-enforce-false-assumptions-mode-change, trying to implement #13 exclusively through enforcement only moves the assumption to before the initialisation sequence, but doesn't get rid of it.
As cleverly suggested by @ZonaDashti, this could be achieved by simply requesting a *-Mode-Status-Report-T from a subsystem without commanding a change to a *-Mode-T first.

Off the top of my head, this would require adding a new request * mode status interface from the OBC's system_mode_manager to each subsystem's *_mode_tracker to just request the current mode status, which could be simply delivered through the existing report * mode status interface.
Internally, the system_mode_manager should be fine just going into the Wait_* state after the request, but some extra logic should be added at the "output stage" because we probably wouldn't want to try downlinking a whole report every time.

These requests would be triggered at the beginning of the initialisation procedure as elicited by #13; additional interesting possibilities could also be upon request from the ground station, and cyclically to generate and store telemetry data.
#15 follows up about the first addition.

The telecommanded, mission-specific mode change features contained in modelling MR #41 shall be implemented in TASTE in accordance with the generated documentation:

• Mission idling
• Mission capturing
• Mission processing

Data View updates currently need to be manually triggered by executing:

taste-update-data-view obc-firmware.asn

This behaviour is non-standard; by default, a make call should automatically invoke that too.

This appeared after renaming the project (from obc-model to obc-firmware), so it's likely to be caused by a mismatch in Data View file name in some project configuration file -- possibly Makefile?

The mission activities contained in modelling MR #41 shall be implemented in TASTE in accordance with the generated documentation:

• image capturing (PWM signal)
• image processing (PWM signal)

The features contained in modelling MR #36 shall be implemented in TASTE in accordance with the generated documentation:

• OBC watchdog kicking
• OBC Watchdog timeout handling: hardware reset

The features for the OBC contained in modelling PR #54 shall be implemented in TASTE:

• add the serial communication library for msp430 (courtesy of @chgio)
• create a cmd queue, provide appropriate API
• send configuration to FlexTrack
• parse the FlexTrack UART messages

The Bug

MSC testing introduced by commit b44bbf3 reveals a sporadic bug in the subsystem-specific mode-changing mechanism initiated by a telecommanded mission mode change (#4). This bug manifests as a mismatch between the new mode to be adopted by a subsystem as part of a telecommanded mission mode change and the contents of the report -- however, this is likely to only appear upon repeated execution of the tests, as it seems to occur about 5-10 % of the time.

By inspection of internal MSCs generated by running taste-run-and-trace ./work/binaries/demo and ./work/tests/run_tests.sh in erroring cases, the root cause of this bug seems to lie in the asynchronous nature of the report_*_mode_status (sporadic interface) calls as opposed to the "forced-synchronous" behaviour of the system_mode_manager. The error is suspected to happen when, upon change of the mode of a subsystem S:

• S actually changes mode from M1 to M2, and
• at the mode_change_report_collector, the arrival of report_S_mode_status is followed by the arrival of complete_system_mode_change with too short of an interval ($\Delta_t \leq 2 , \textrm{ticks}$)

...Because such a short interval does not allow the mode_change_report_collector to assign the reported value of S's new mode M2 in the collated report; instead, mode_change_report_collector calls relay_system_mode_change_report with a report stating S is in its previous mode M1 -- despite S actually having just changed mode to M2. This would explain why, in the vast majority of the cases, the error involves the pipeline_mode_tracker subsystem (as it is the last one in the ordering of mode change calls), and only one of the two fields (*_mode and *_last_op) is a mismatch.

Additionally, Maxime has shown that the inflation of sporadic interfaces (at least with 1KB stacks) already exceeds the memory budget of the MSP430 (at least with the ada runtime) for this relatively simple feature.

The Refactor

Refactoring the majority of asynchronous calls (sporadic interfaces or combinations of sporadic interfaces) to synchronous calls should address this bug -- and significantly cut down the feature's impact on the memory budget.

To do so, follow the steps below.

Interface View Changes

• in comms_mode_tracker:
  • set change_comms_mode as an unprotected instead of sporadic interface
  • create the following mode-explicit, parameterless, unprotected interfaces:
    • change_comms_mode_idle
    • change_comms_mode_uplink
    • change_comms_mode_downlink
  • add T_Boolean success as an output parameter to all change_comms_mode_* interfaces
  • remove return_comms_mode_status interface
• in eps_mode_tracker:
  • set change_eps_mode as an unprotected instead of sporadic interface
  • create the following mode-explicit, parameterless, unprotected interfaces:
    • change_eps_mode_idle
    • change_eps_mode_high_power
  • add T_Boolean success as an output parameter to all change_eps_mode_* interfaces
  • remove return_eps_mode_status interface
• in pipeline_mode_tracker:
  • set change_pipeline_mode as an unprotected instead of sporadic interface
  • create the following mode-explicit, parameterless, unprotected interfaces:
    • change_pipeline_mode_idle
    • change_pipeline_mode_processing_fp
    • change_pipeline_mode_processing_lp
  • add T_Boolean success as an output parameter to all change_pipeline_mode_* interfaces
  • remove return_pipeline_mode_status interface
• in payload_mode_tracker:
  • set change_payload_mode as an unprotected instead of sporadic interface
  • create the following mode-explicit, parameterless, unprotected interfaces:
    • change_payload_mode_idle
    • change_payload_mode_imaging
  • add T_Boolean success as an output parameter to all change_payload_mode_* interfaces
  • remove return_payload_mode_status interface
• in adcs_mode_tracker:
  • set change_adcs_mode as an unprotected instead of sporadic interface
  • create the following mode-explicit, parameterless, unprotected interfaces:
    • change_adcs_mode_idle
    • change_adcs_mode_detumbling
  • add T_Boolean success as an output parameter to all change_adcs_mode_* interfaces
  • remove return_adcs_mode_status interface
• in deployer_mode_tracker:
  • set change_deployer_mode as an unprotected instead of sporadic interface
  • create the following mode-explicit, parameterless, unprotected interfaces:
    • change_deployer_mode_idle
    • change_deployer_mode_deployment
  • add T_Boolean success as an output parameter to all change_deployer_mode_* interfaces
  • remove return_deployer_mode_status interface

Implementation Changes

• in comms_mode_tracker:
  • remove (comms_mode) from input symbols
  • declare change_comms_mode_* procedures
  • copy and paste procedure parameters from the SDL top-level declaration (body of external procedure, from fpar)
  • transfer transition behaviour into change_comms_mode_* procedures (only leave blank transition behaviour, setting success and comms_mode under the assumption of a successful transition)
  • transfer transition policy enforcement logic to system_mode_manager
• in eps_mode_tracker:
  • remove (eps_mode) from input symbols
  • declare change_eps_mode_* procedure
  • copy and paste procedure parameters from the SDL top-level declaration (body of external procedure, from fpar)
  • transfer transition behaviour into change_eps_mode_* procedures (only leave blank transition behaviour, setting success and eps_mode under the assumption of a successful transition)
  • transfer transition policy enforcement logic to system_mode_manager
• in pipeline_mode_tracker:
  • remove (pipeline_mode) from input symbols
  • declare change_pipeline_mode_* procedure
  • copy and paste procedure parameters from the SDL top-level declaration (body of external procedure, from fpar)
  • transfer transition behaviour into change_pipeline_mode_* procedures (only leave blank transition behaviour, setting success and pipeline_mode under the assumption of a successful transition)
  • transfer transition policy enforcement logic to system_mode_manager
• in payload_mode_tracker:
  • remove (payload_mode) from input symbols
  • declare change_payload_mode_* procedure
  • copy and paste procedure parameters from the SDL top-level declaration (body of external procedure, from fpar)
  • transfer transition behaviour into change_payload_mode_* procedures (only leave blank transition behaviour, setting success and payload_mode under the assumption of a successful transition)
  • transfer transition policy enforcement logic to system_mode_manager
• in adcs_mode_tracker:
  • remove (adcs_mode) from input symbols
  • declare change_adcs_mode_* procedure
  • copy and paste procedure parameters from the SDL top-level declaration (body of external procedure, from fpar)
  • transfer transition behaviour into change_adcs_mode_* procedures (only leave blank transition behaviour, setting success and adcs_mode under the assumption of a successful transition)
  • transfer transition policy enforcement logic to system_mode_manager (leave uncalled as adcs is not involved in this feature)
• in deployer_mode_tracker:
  • remove (deployer_mode) from input symbols
  • declare change_deployer_mode_* procedure
  • copy and paste procedure parameters from the SDL top-level declaration (body of external procedure, from fpar)
  • transfer transition behaviour into change_deployer_mode_* procedures (only leave blank transition behaviour, setting success and deployer_mode under the assumption of a successful transition)
  • transfer transition policy enforcement logic to system_mode_manager (leave uncalled as deployer is not involved in this feature)
• in system_mode_manager:
  • replace intermediate "Wait for S" states with "S done" states (no longer required for blocking, but helpful for validation by inspection)
  • simply traverse subsystems calling change_*_mode_* in the current order
  • for every subsystem every subsystem involved in the feature, implement transition policy enforcement logic

Note that, in order to transfer behaviours where the target state is decided on the basis of an input parameter, it might be necessary to "exfiltrate" that decision from within the procedure to the outside process -- however, this can be simply done by declaring a variable in the process, assigning it within the procedure (as the outcome of a decision on a parameter), and then deciding on it in the process.
This is because, as TASTE's synchronous calls ((un)protected interfaces) are implemented as SDL exported procedures, all their parameters are exclusively accessible from within the procedure itself.
More details are documented in the Detailed SDL Tutorial.

The following is an example on dummy blocks:

dummy block architecture

top-level SDL declaration of the dummy_caller process

SDL behaviour of the dummy_caller process

top-level SDL declaration of the `dummy_function_ process

SDL behaviour of the dummy_function process

definition of the call_dummy exported procedure

Following up on #14, *-Mode-Status-Report-Ts could be requested and collated into a System-Mode-Status-Report-T upon reception of an appropriate TC from the GS.

After completion of #14, this shouldn't need anything more than a couple of new interfaces from and to the gs_gui and a new system_mode_status_request_tc_handler component within the OBC.

On the other hand, it would offer the very convenient option for a ground operator to just query the system about its actual mode without commanding any changes.

TASTE Docker installation instructions (from taste.tools):

You can build a complete Docker image (which is very similar to the Virtual Machine) by using this Dockerfile as follows:

$ mkdir TASTE 
$ cd TASTE
$ wget https://gitrepos.estec.esa.int/taste/taste-setup/raw/feature_buster/Dockerfile
$ docker build -t taste .  # don't forget the dot at the end

...and then run a fresh TASTE container with proper X11 redirection, with...

$ wget https://gitrepos.estec.esa.int/taste/taste-setup/raw/master/Docker-run.sh
$ ./Docker-run.sh

This will launch a Docker container able to run all TASTE applications, including the X11-based ones (e.g. the Ellidiss GUI). Note that your $HOME folder is mapped under /root/work inside the container - so save your work ONLY in there, because everything else will be deleted when the container dies. You can of course still commit the container and thus obtain your own custom TASTE images.

#4 and #11 introduced a relatively simple bug in the System Mode Manager: initial subsystem states are only assumed, not enforced.

As this bug became apparent through a discrepancy with the payload camera, for cloudview-1 (Release) this was hot-fixed by adding manual mode override through Serial in guorbit/camera-relay (cloudview-1 Release): src/main.cpp, lines 57-80 @ 3731638 (comment).

This "hack" adopts the approach of ensuring the assumptions are correct, which did work but is suboptimal in terms of robustness, especially in unmanned initialisation scenarios. Instead -- or, better yet, in addition -- it would be auspicable to programmatically enforce the assumptions in the initial transition of the System Mode Manager.
My idea is to substitute the assignments in the initialisation line with calls to the same code that requests the TC-based "arbitrary" subsystem mode changes, but requesting the currently assumed initial mode instead.

It should be possible to achieve this elegantly enough through SDL Joins or SDL Procedures: internal refactorings should be strongly preferred, to minimise the architectural impact and memory footprint of these changes.
(TASTE SDL Tutorial)

We should also think about how to handle failures: the most basic approach would be to wrap this in a loop to try again until success, but Rule 2 of NASA/JPL's The Power of 10 prescribes that indefinite loops should be avoided. This could potentially be an exception, as operational functionalities cannot be safely reached unless initialisation is successful; otherwise, a definite loop could leave an increasing time between successive tries to hopefully allow for completion of whatever process is causing the failure.
A perhaps better approach could be to get rid of the assumptions entirely, assigning the actual state contained in the subsystem's response instead of the requested assumption.

The automatic, generic mode change features contained in modelling MR #41 shall be implemented in TASTE in accordance with the generated documentation:

• Link State Loop traversal
• Sensing State Loop traversal

The system_mode_manager component currently has two implementations:

• default, where the requested mode change transition for each subsystem is validated (based on its current and target modes, and the relevant model FSM), and only requested if valid;
• test_subsystem_mode_change_validation, where the requested mode change transition is not validated for any subsystem.

The second implementation was created to test that all copies of the subsystem mode change transition validation logic, distributed into each subsystem mode tracker for redundancy, are correct.
However, this is a smelly solution: as revealed in @xszymonzur's work towards #13, any change to any other logic in the component must be duplicated manually in both implementations.

A better approach would be to use the SDL Alternative (compilation option) symbol to include both options (validating and non-validating) and select which one to compile through a simple flag -- probably in the form of a Context Parameter to the component.
Then, separate test runs can be done with the alternative flag set and unset to check both options.