Manage health message

Overview

SENSR outputs the health status of each component:

• Master node health
• Algo node health
• LiDAR sensor health
• SENSR log stream

Location of the health status

The Health status is shared through 2 methods: APIs and the Websocket output.

Websocket

The health status is shared through the output data every 1 second with the followig elements

• Master node status
  • None,
  • OK: No issue,
  • Storage shortage: Less than 85% storage left,
  • Internal error: Error in the master node.
• Algo node status
  • None,
  • OK: No issue,
  • ROS Error: Error with the ROS node,
  • Lost Connection: Connection lost with the Algo node,
  • Invalid GPU Config: Error returned by Cuda on the Algo node,
• Sensor status
  • Sensor Dead: Sensor not responding,
  • Sensor Alive: Sensor OK,
  • Sensor Erroneous: Sensor detected obstructed,
  • Sensor Tilted: Sensor detected tilted.

Sensr logs are also published as a stream on port 5057.

API

The health status is shared through the health API. The Health message is designed to provide Statuses as follows:

├── Master
    └──Algo
        └──Sensor
        └──Sensor
    └──Algo
        └──Sensor

Recommendation on health message integration

For continuous monitoring, it is recommended parsing the websocket output. Here is a sample code in python that parses the output message to extract the health status:

def _on_get_output_message(self, message):
    assert isinstance(message, sensr_output.OutputMessage), "message should be of type OutputMessage"

    if message.HasField('stream') and message.stream.HasField('health'):
        
        system_health = message.stream.health
        print('System health: {0}'.format(system_health.master))

        if len(system_health.nodes) > 0:
            for node_key in system_health.nodes:
                node_health = system_health.nodes[node_key]
                print('  Node ({0}) health: {1}'.format(node_key, node_health.status))

                if len(node_health.sensors) > 0:
                    for sensor_key in node_health.sensors:
                        sensor_health = node_health.sensors[sensor_key]
                        print('    Sensor ({0}) health: {1}'.format(sensor_key, sensor_health))
                else:
                    print('    No sensors are connected')
        else:
            print('  No nodes are connected')