Version: 3.2.1

Output Specification

Output Data Specification

The Protobuf data specification for all output messages can be found and downloaded from these links:

Streaming Data (TCP and Websocket)

This is a live data stream that can be parsed and processed by external applications. The Websocket Endpoints are detailed below:

Communication Protocol	TCP (Websocket)
Data Encoding	Protobuf 3
Output Port	5050
Protobuf Message	OutputMessage

Communication Protocol	TCP (Websocket)
Data Encoding	Protobuf 3
Output Port	5051
Protobuf Message	PointResult

Output Message

This is the main message that carries all the output results sent by SENSR in runtime mode. To reduce bandwidth usage, SENSR does not fill all the fields at all times. Each message has a defined output frequency described below.

Type: Continuous Stream (every frame)

Field	Unit	Type	Note
timestamp	-	Timestamp	Timestamp when streaming message is sent.
stream	-	StreamMessage	SENSR perception result stream message.
event	-	EventMessage	SENSR event message (Zone, Losing, SystemHealth)
custom	-	CustomMessage	SENSR additional result message (Field of Regard)

Stream Message

This message includes output results of SENSR. Each field will be filled based on the rates below:

objects : every frame
zones : every 10 sec.
health : every 1 sec.

The fields has_objects for objects and has_zones for zones act as a flag to indicate if a specific output result has been published or not. For the health message, its presence describes if the message has been published or not.

Type: Continuous Stream

Field	Unit	Type	Note
objects	-	Object	Object list in the latest frame.
has_objects	-	bool	A flag to check `objects` field is valid or not
zones	-	ZoneConfig	Event zone configuration.
has_zones	-	bool	A flag to check `zones` field is valid or not
health	-	SystemHealth	System health in the latest frame.
static_object	-	Object	Static object list in the latest frame.

Object

Object is the standard object tracking output. Note that points and history are optional fields. You can receive these optional fields by setting a specific configuration(“common.output.publish_point_cloud”).

Type: Continuous Stream (every frame)

Field	Unit	Type	Note
id	-	int32	ID of the object.
label	-	enum	0 = None 1 = Car 2 = Pedestrian 3 = Cyclist, 4 = Misc
confidence	-	float	0 to 1 probability of the object classification.
bbox	-	BoundingBox	Bounding box of the object.
velocity	m/s	Vector3	XYZ velocity of the object.
tracking_status	-	enum	0 = NONE, 1 = VALIDATING, 2 = INVALIDATING, 3 = TRACKING, 4 = DRIFTING, 5 = EXPIRED (VALIDATING : Check validity in early stage of tracking INVALIDATING : Short term prediction when tracking is lost in VALIDATING status TRACKING : Stable tracking (RECOMMENDED VALUE TO USE FOR TRACKING, IGNORE THE REST) DRIFTING: Short term prediction when tracking is lost in TRACKING status EXPIRED: Expired tracking)
yaw_rate	-	float	The yaw value difference of the object between the previous frame and the current frame is divided by the time delta. (radian)
last_observed_timestamp	-	Duration	Last updated time offset relative to `first_cloud_arrival_timestamp`
retro_reflective	-	bool	A flag to show if the object is classified as retro reflective.
points	meters	bytes	List of sequential 3 float value set (XYZ).
history	-	History	History of the object.
prediction	-	Prediction	Prediction of the object.
zone_ids	-	int32	List of zone occupied by the object.
intensities	-	bytes	Sequential 1 float value set

Python Sample Code using SENSR SDK

Get point, bbox, prediction

if message.HasField('stream') and message.stream.has_objects:
    for obj in message.stream.objects:
        float_size = ctypes.sizeof(ctypes.c_float)
        object_point_num = len(obj.points) // (float_size * 3) # Each point is 3 floats (x,y,z)

        intensity_np = np.frombuffer(obj.intensities, np.float32)
        
        if len(intensity_np) != 0:
            min_intensity = np.min(intensity_np)
            median_intensity = np.median(intensity_np)
            max_intensity = np.max(intensity_np)
        else:
            min_intensity = 0.0
            median_intensity = 0.0
            max_intensity = 0.0

        print('Obj ({0}): point no. {1}'.format(obj.id, object_point_num))
        print('Obj ({0}): point intensity [min, median, max] is [{1}, {2}, {3}]'.format(obj.id, min_intensity, median_intensity, max_intensity))
        print('Obj ({0}): velocity {1}'.format(obj.id, obj.velocity))
        print('Obj ({0}): bbox {1}'.format(obj.id, obj.bbox))
        print('Obj ({0}): tracking status {1}'.format(obj.id, sensr_type.TrackingStatus.Name(int(obj.tracking_status))))
        print('Obj ({0}): Object type {1}'.format(obj.id, sensr_type.LabelType.Name(int(obj.label))))
        print('Obj ({0}): prediction {1}'.format(obj.id, obj.prediction))

Get the velocity value and position angle of the objects

import math
for obj in message.stream.objects:

    Velocity = math.sqrt(math.pow(obj.velocity.x,2)+math.pow(obj.velocity.y,2))
    Angle = math.atan2(obj.velocity.y,obj.velocity.x)

Zone Config

This is a low frequency message to communicate the event zone shapes and locations for you who do not want to use the REST API.

Type: Continuous Stream (low frequency, 1 per 10 seconds or slower)

Field	Unut	Data type	Note
id	-	int32	Zone id
name	-	string	Zone name
pbox	-	PolygonBox	Array of X,Y,Z
type	-	Enumeration	0 = None 1 = Event

Event Message

This message will be filled whenever SENSR detects events. There are 3 types of events. zone, losing, system. Before parsing each field, you need to check the size of each field. The size 0 means no event related to that field.

Type: Trigger on event

Field	Unit	Type	Note
zone	-	ZoneEvent	Event list related to zone.
losing	-	LosingEvent	Event when SENSR lose tracking.
health	-	SystemHealth	Event when SENSR has trouble inside.

ZoneEvent

This is an event triggered message. This message gets sent when an object interacts with a zone. Four interactions are: Enter Zone, Exit Zone, Loitering(more than X seconds in a zone) and Velocity(speed above X m/s in a zone).

Type: Trigger on event

Field	Unit	Data type	Note
timestamp	-	TimeStamp	Event trigger time.
id	-	int32	Zone id
type	-	Enumeration	0 = None, 1 = Entry, 2 = Exit, 3 = Loitering, 4 = Exceed speed
object	-	ZoneEvent_Object	Object information

Python Sample Code using SENSR SDK

Get trigger message of the zone

for zone in message.event.zone:
    if zone.type == sensr_output.ZoneEvent.Type.ENTRY:
            print('Entering zone ({0}): obj {1}'.format(zone.id, zone.object.id))

    elif zone.type == sensr_output.ZoneEvent.Type.EXIT:
            print('Exiting zone ({0}): obj {1}'.format(zone.id, zone.object.id))

Object in ZoneEvent

ZoneEvent saves an object of this type.

Field	Unit	Data type	Note
id	-	int32
position	meters	Vector3 (float)	Object location at time of trigger
heading	-	Float	Yaw (0.0-2.0Pi)
velocity	m/s	Vector3 (float)	m/s + direction (optional) Filled in case of overspeed events.

LosingEvent

SENSR will send this message when an object is no longer detected.

Type: Trigger on event

Field	Unit	Data type	Note
timestamp	-	TimeStamp
id	-	int32	Object id
position	meters	Vector3 (float)	Last known location of object
heading	-	float	Yaw (0.0-2.0Pi)

Custom Message

This message includes additional information of SENSR.

Type: Continuous Stream

Field	Unit	Data type	Note
bg_learning_progress	-	float	Background learning progress (0.0 - 1.0)
profiling	-	ProfilingResultSet	Profiling result of current frame

System Health

This is part of the Health Monitoring System. This is both a continuous low frequency stream (acting as a heartbeat) and an event trigger stream.

Type: Continuous Stream + Trigger on event

Field	Unit	Data type	Note
master	-	Enumeration	0 = None 1 = OK 2 = Storage Shortage 3 = SlowDown Error 4 = Internal Error
nodes	-	map<string, NodeHealth>	Each algo node health

Python Sample Code using SENSR SDK

Get system health informantion

    if message.stream.HasField('health'):
        system_health = message.stream.health
        print(‘System health: {0}’.format(system_health.master))

        if len(system_health.nodes) > 0:
            for node_key in syste_health.nodes:
                    node_health = system_health.nodes[node_key]
                    print('Node ({0}) health: {1}'.format(node_key, health.status))

                    if len(node_health.sensors) > 0:
                        for sensor_key in node_health.sensors:
                                Sensor_health = node_health.sensors[sensor_key]
                                print('Sensor ({0}) health {1}'.format(sensor_key, sensor_health))
                    else:
                        print(‘ No sensors are connected’)
        else:
            print(‘No nodes are connected’)

Node Health

This is part of the System Health message.

Field	Unit	Data type	Note
status	-	Enumeration	0 = None, 1 = OK, 2 = ROS Error, 3 = Lost Connection, 4 = Invalid GPU Config
sensors	-	map<string, Enumeration>	0 = sensor dead 1 = sensor alive 2 = sensor erroneous(Sensor is obstructed.) 3 = sensor tilted 4 = sensor suspended(Sensor points are not transmitted temporarily.)

Shared Data Types

Bounding Box

This is part of the Object message.

Field	Unit	Type	Note
position	meters	Vector3	XYZ position of the bounding box. Position is defined as (Center X, Center Y, Bottom Z).
size	meters	Vector3	XYZ size of the bounding box.
yaw	radians	float	Bounding box rotation angle along the Z axis.

Polygon box

This is part of the Zone Config message.

Field	Unit	Data type	Note
points	-	Vector2 (float)	Shape of zone in 2D
min_z	meters	float	Base point in z axis
max_z	meters	float	Height of zone

History

This is part of the Object message.

Field	Unit	Data type	Note
states	-	History.State	List of history state

History.State

This is part of the Object message.

Field	Unit	Data type	Note
positions	meters	Vector3 (float)	object’s tracked XYZ position.
timestamp	-	Timestamp	timestamp of the tracked XYZ position.

ReplayInfo

Replay detail information.

Field	Unit	Data type	Note
current_index	-	int32	Current playing frame index in replay

RecordingInfo

Recording detail information.

Field	Unit	Data type	Note
saving_progress	-	float	Progress of saving file(0.0 - 1.0)

Point Result

This is an optional message. If a user wants to get raw point clouds, the user can use this. To get this, the user needs to enable a specific configuration(“common.output.publish_point_cloud” = 2) in advance. This message increases lots of burden in transfer bandwidth.

Type: Continuous Stream (low frequency, 1 per 10 seconds or slower)

Field	Unit	Data type	Note
points	-	PointCloud
uid	-	string	deprecated

PointCloud

Field	Unit	Type	Note
points	-	enum	0 = None, 1 = Ground, 2 = Background, 3 = Raw
uri	-	string	Address of point cloud.

Python Sample Code using SENSR SDK

Get Point Cloud

for point_cloud in message.points:
    float_size = ctypes.sizeof(ctypes.c_float)
    num_points = len(point_cloud.points) // (float_size * 3) # Each point is 3 floats (x,y,z)
    
    intensity_np = np.frombuffer(point_cloud.intensities, np.float32)
    
    if len(intensity_np) != 0:
        min_intensity = np.min(intensity_np)
        median_intensity = np.median(intensity_np)
        max_intensity = np.max(intensity_np)
    else:
        min_intensity = 0.0
        median_intensity = 0.0
        max_intensity = 0.0

    if point_cloud.type == sensr_pcloud.PointResult.PointCloud.Type.RAW:
        print('Topic ({0}) no. of points - {1}. Min and max intensity is [{2}, {3}]'.format(point_cloud.id, num_points, min_intensity, max_intensity))
        print('Intensity [min, median, max] is [{0}, {1}, {2}]'.format(min_intensity, median_intensity, max_intensity))
    elif point_cloud.type == sensr_pcloud.PointResult.PointCloud.Type.GROUND:
        print('Ground points no. of points - {0}.'.format(num_points))
        print('Intensity [min, median, max] is [{0}, {1}, {2}]'.format(min_intensity, median_intensity, max_intensity))
    elif point_cloud.type == sensr_pcloud.PointResult.PointCloud.Type.BACKGROUND:
        print('Environment points no. of points - {0}'.format(num_points))
        print('Point intensity [min, median, max] is [{0}, {1}, {2}]'.format(min_intensity, median_intensity, max_intensity))

Output Data Specification​

Streaming Data (TCP and Websocket)​

Output Message​

Stream Message​

Object​

Zone Config​

Event Message​

ZoneEvent​

Object in ZoneEvent​

LosingEvent​

Custom Message​

System Health​

Node Health​

Shared Data Types​

Bounding Box​

Polygon box​

History​

History.State​

ReplayInfo​

RecordingInfo​

Point Result​

Point Result​

PointCloud​

Output Data Specification

Streaming Data (TCP and Websocket)

Output Message

Stream Message

Object

Zone Config

Event Message

ZoneEvent

Object in ZoneEvent

LosingEvent

Custom Message

System Health

Node Health

Shared Data Types

Bounding Box

Polygon box

History

History.State

ReplayInfo

RecordingInfo

Point Result

Point Result

PointCloud