Multi-node Simulation using HLA

../../_images/certi.png

HLA and the Run-Time Infrastructure

High Level Architecture

HLA (High-Level Architecture) is a specification for software architectures that defines the management and deployment of a global simulation made of distributed simulators. Each simulator (called a federate) is connected to other simulators through the Run-Time Infrastructure (RTI).

Run-Time Infrastructure

RTI is the fundamental component of HLA. It provides a set of software services that are necessary to support federates to coordinate their operations and data exchange during a runtime execution. In other words, it is the implementation of the HLA interface specification but is not itself part of specification. Modern RTI implementations conform to the IEEE 1516 and/or HLA 1.3 API specifications. These specifications do not include a network protocol for RTI. It is up to the implementors of an RTI to create a specification.

The CERTI

MORSE-HLA uses the CERTI, an open-source RTI, distrubuted under the GPL/LGPL licence, that fully supports HLA 1.3, and partly IEEE 1516. The CERTI provides, among others, HLA implementations for C++ and Python.

The CERTI implementation of HLA is based on a process, the RTIG (RTI Gateway), that manages the simulation and transfers messages between federates. An RTIA process is automatically launched close to each federate, and manages the interaction between the federates and the RTIG.

../../_images/certi-architecture.png

The CERTI architecture.

HLA in MORSE

Federation Object Model

Data exchange in an HLA simulation (called a federation) is specified in a set of tables, that compose a FOM (Federation Object Model). Below are the two tables that are relevant for the MORSE federation.

Object Class Sructure
objectRoot (N) RTIPrivate (N)
Robot (PS)
  • PS stands for publishable/subscribable
  • N stands for neither P nor S

The Object Class Structure describes the objects available in the federation, their attributes, and their hierarchy. The only exchangable object of the MORSE federation is Robot.

Attributes
Object Attribute Datatype Card. Units Update type Update rate U/R
Robot id integer 1 NA never NA UR
position sequence 3 BU periodic BGE rate UR
orientation sequence 3 BU periodic BGE rate UR
  • NA stands for Not Applicable
  • BU stands for Blender Unit
  • BGE stands for Blender Game Engine
  • U stands for Update
  • R stands for Reflect

The attributes table indicates that the Robot object is described by an id, a position (a 3D vector) and an orientation (a 3D vector). Position and orientation are given in the Blender frame (hence in Blender Units). They are updated at each execution of the Game Engine, and they are available for MORSE nodes (i.e. federates) in update (i.e., writable) and reflect (i.e., readable) modes.

The FOM is owned by the RTIG as the morse.fed file, and allows any federate that complies with this specification to join the federation without depending on any MORSE data structure. Such a use-case is described in the Hybrid Simulation tutorial. The morse.fed file is installed along with MORSE-HLA, and is also available in the source tree.

Node synchronization

The HLA multi-node simulation uses the Best effort synchronization mode. Best effort is the simplest synchronization mode. Each node provides its robots’ states as fast as it can (once per Blender frame), and updates the external robots’ states as it receives them. The nodes are not really synchronized: nothing guarantees that the node worlds are the same at any moment. Moreover, the Blender engines are not constrained: each Blender node runs as fast as it can, resulting in different simulation rates in each node. However, if your network is fast enough, and if you do not use Blender logical time for your simulation, this multi-node simulation mode should be sufficient for most cases.

Time synchronization ensures that all the nodes will have the same time at each moment. Here, understand logical time, or Blender time: the number of frames executed each second on each Blender node will be identical. This simulation mode guarantees that all the nodes will have the same simulation state at each time step. The time synchronization integration in MORSE is still a work-in-progress.

HLA Multi-node simulation

Environment variables

In order to execute an HLA simulation, you have to configure your system environment. The CERTI documentation gives some information about the configurable variables. The environment variables that may be relevant to MORSE are described below:

  • CERTI_HTTP_PROXY: if you have to use a proxy in order to join the RTIG, you must use the CERTI_HTTP_PROXY variable; if your proxy is already defined in the http_proxy variable, please, reset it using:

    $ export CERTI_HTTP_PROXY=$http_proxy

    The http_proxy variable is not directly used in MORSE/HLA as, most of the time, it causes some simulation mistakes.

  • CERTI_FOM_PATH: this variable is only relevant for the RTIG. It says where to find the FOM file that lists all the messages that can be exchanged in the HLA simulation. This FOM file is installed with MORSE, so if you have installed MORSE on the computer where you launch the RTIG, you must define this variable as:

    $ export CERTI_FOM_PATH=$MORSE_ROOT/share/federations:$CERTI_FOM_PATH

    Otherwise, you will have to copy the morse.fed file from the src/morse/middleware/hla source directory to your computer and define the CERTI_FOM_PATH variable accordingly. Default directories where the RTIG is looking for FOM files are the working directory (from where you launched the RTIG) and the $CERTI_HOME/share/federations path.

HLA in the Builder API

The Builer API provides an Environment object that must also be configured for HLA multi-node simulation:

env.configure_multinode(protocol="hla", server_address="140.93.0.93", server_port="60400",
                        distribution={
                            "nodeA": ["MyFirstRobot"],
                            "nodeB": ["MySecondRobot"],
                        })
  • The server_address is the IP address of the computer hosting the RTIG.
  • The server_port is the TCP port used to connect to the RTIG. By default, the TCP port number is 60400.

If you want to use another port, you must specify it in the Builder configuration (as described above), and set the CERTI_TCP_PORT environment variable before launching the RTIG.