Multi-Platform Multi-Bus

This guide describes the setup process for a multi-platform connection that has a combination of ZeroMQ and RabbitMQ instances. For this example, we want to use the Forwarder to pass device data from two VOLTTRON instance to a single “central” instance for storage. It will also have a Volttron Central agent running on the “central” instance and Volttron Central Platform agents on all 3 instances and connected to “central” instance to provide operational status of it’s instance to the “central” instance. For this document “node” will be used interchangeably with VOLTTRON instance. The authentication of remote connections can be performed either using admin web interface or using command line interface. We will demonstrate both the approaches.

Node Setup

For this example we will have two types of nodes; a data collector and a central node. Each of the data collectors will have different message buses (VOLTTRON supports both RabbitMQ and ZeroMQ). The nodes will be configured as in the following table.

Node Configuration

Central

Node-ZMQ

Node-RMQ

Node Type

Central

Data Collector

Data Collector

Platform Driver

yes

yes

Forwarder

yes

yes

SQL Historian

yes

Volttron Central

yes

Volttron Central Platform

yes

yes

yes

Exposes RMQ Port

yes

Exposes ZMQ Port

yes

Exposes HTTPS Port

yes

The goal of this is to be able to see the data from Node-ZMQ and Node-RMQ in the Central SQL Historian and on the trending charts of Volttron Central.

Virtual Machine Setup

The first step in creating a VOLTTRON instance is to make sure the machine is ready for VOLTTRON. Each machine should have its hostname setup. For this walk-through, the hostnames “central”, “node-zmq” and “node-rmq” will be used.

For Central and Node-RMQ follow the instructions platform installation steps for RMQ. For Node-ZMQ use Platform Installation steps for ZeroMQ.

Instance Setup

The following conventions/assumptions are made for the rest of this document:

  • Commands should be run from the VOLTTRON root

  • Default values are used for VOLTTRON_HOME($HOME/.volttron), VIP port (22916), HTTPS port (8443), RabbitMQ ports (5671 for AMQPs and 15671 for RabbitMQ management interface). If using different VOLTTRON_HOME or ports, please replace accordingly.

  • Replace central, node-zmq and node-rmq with your own hostnames.

  • user will represent your current user.

The following will use vcfg (volttron-cfg) to configure the individual platforms.

Central Instance Setup

Note

This instance must have been bootstrapped using --rabbitmq see RabbitMq installation instructions.

Next step would be to configure the instance to have a web interface to accept/deny incoming certificate signing requests from other instances. Additionally, we will need to install a Volttron Central agent, Volttron Central Platform agent, SQL historian agent and a Listener agent. The following shows an example command output for this setup.

(volttron)user@central:~/volttron$ vcfg

Your VOLTTRON_HOME currently set to: /home/user/.volttron

Is this the volttron you are attempting to setup? [Y]:
What type of message bus (rmq/zmq)? [zmq]: rmq
Name of this volttron instance: [volttron1]: central
RabbitMQ server home: [/home/user/rabbitmq_server/rabbitmq_server-3.9.7]:
Fully qualified domain name of the system: [central]:
Would you like to create a new self signed root CAcertificate for this instance: [Y]:

Please enter the following details for root CA certificate
    Country: [US]:
    State: WA
    Location: Richland
    Organization: PNNL
    Organization Unit: volttron
Do you want to use default values for RabbitMQ home, ports, and virtual host: [Y]:
2020-04-13 13:29:36,347 rmq_setup.py INFO: Starting RabbitMQ server
2020-04-13 13:29:46,528 rmq_setup.py INFO: Rmq server at /home/user/rabbitmq_server/rabbitmq_server-3.9.7 is running at
2020-04-13 13:29:46,554 volttron.utils.rmq_mgmt DEBUG: Creating new VIRTUAL HOST: volttron
2020-04-13 13:29:46,582 volttron.utils.rmq_mgmt DEBUG: Create READ, WRITE and CONFIGURE permissions for the user: central-admin
Create new exchange: volttron, {'durable': True, 'type': 'topic', 'arguments': {'alternate-exchange': 'undeliverable'}}
Create new exchange: undeliverable, {'durable': True, 'type': 'fanout'}
2020-04-13 13:29:46,600 rmq_setup.py INFO:
Checking for CA certificate

2020-04-13 13:29:46,601 rmq_setup.py INFO:
 Creating root ca for volttron instance: /home/user/.volttron/certificates/certs/central-root-ca.crt
2020-04-13 13:29:46,601 rmq_setup.py INFO: Creating root ca with the following info: {'C': 'US', 'ST': 'WA', 'L': 'Richland', 'O': 'PNNL', 'OU': 'VOLTTRON', 'CN': 'central-root-ca'}
Created CA cert
2020-04-13 13:29:49,668 rmq_setup.py INFO: **Stopped rmq server
2020-04-13 13:30:00,556 rmq_setup.py INFO: Rmq server at /home/user/rabbitmq_server/rabbitmq_server-3.9.7 is running at
2020-04-13 13:30:00,557 rmq_setup.py INFO:

#######################

Setup complete for volttron home /home/user/.volttron with instance name=central
Notes:
 - On production environments, restrict write access to /home/user/.volttron/certificates/certs/central-root-ca.crt to only admin user. For example: sudo chown root /home/user/.volttron/certificates/certs/central-root-ca.crt and /home/user/.volttron/certificates/certs/central-trusted-cas.crt
 - A new admin user was created with user name: central-admin and password=default_passwd.
   You could change this user's password by logging into https://central:15671/ Please update /home/user/.volttron/rabbitmq_config.yml if you change password

#######################

The rmq message bus has a backward compatibility
layer with current zmq instances. What is the
zmq bus's vip address? [tcp://127.0.0.1]: tcp://192.168.56.101
What is the port for the vip address? [22916]:
Is this instance web enabled? [N]: y
Web address set to: https://central
What is the port for this instance? [8443]:
Is this an instance of volttron central? [N]: y
Configuring /home/user/volttron/services/core/VolttronCentral.
Installing volttron central.
['volttron', '-vv', '-l', '/home/user/.volttron/volttron.cfg.log']
Should the agent autostart? [N]: y
VC admin and password are set up using the admin web interface.
After starting VOLTTRON, please go to https://central:8443/admin/login.html to complete the setup.
Will this instance be controlled by volttron central? [Y]:
Configuring /home/user/volttron/services/core/VolttronCentralPlatform.
What is the name of this instance? [central]:
Volttron central address set to https://central:8443
['volttron', '-vv', '-l', '/home/user/.volttron/volttron.cfg.log']
Should the agent autostart? [N]:
Would you like to install a platform historian? [N]: y
Configuring /home/user/volttron/services/core/SQLHistorian.
['volttron', '-vv', '-l', '/home/user/.volttron/volttron.cfg.log']
Should the agent autostart? [N]: y
Would you like to install a platform driver? [N]:
Would you like to install a listener agent? [N]: y
Configuring examples/ListenerAgent.
['volttron', '-vv', '-l', '/home/user/.volttron/volttron.cfg.log']
Should the agent autostart? [N]: y
Finished configuration!

You can now start the volttron instance.

If you need to change the instance configuration you can edit
the config file is at /home/user/.volttron/config

Start VOLTTRON instance and check if the agents are installed.

./start-volttron
vctl status

Using the web interface:

Open browser and go to the platform web admin authentication page https://central:8443/index.html to accept/reject incoming certificate signing request (CSR) from other platforms.

Note

Replace “central” with the proper hostname of VC instance in the admin page URL. If opening the admin page from a different system, then please make that the hostname is resolvable in that machine.

Click on “Login To Administration Area”.

../../_images/csr-initial-state.png

Set the platform web admin username and password. This can be later used to login into the web admin authentication page. This username and password will also be used to log in to Volttron Central.

../../_images/csr-set-admin.png

Login into the platform web admin page.

../../_images/csr-login-page.png

After logging in, you will see no CSR requests initially.

../../_images/central_no_pending.png

Go back to the terminal and start Volttron Central Platform agent on the “central” instance. The agent will send a CSR request to the web interface.

vctl start --tag vcp

Now go to the platform web admin page to check if there is a new pending CSR request. You will see a “PENDING” request from “central.central.platform.agent”

../../_images/central_pending.png

Approve the CSR request to allow authenticated SSL based connection to the “central” instance.

Go back to the terminal and check the status of Volttron Central Platform agent. It should be set to “GOOD”.

Using command line:

Alternatively, you can also check the status of pending CSRs via the command line.

After starting the Volttron Central Platform agent, use the auth remote sub-command’s list to display the current pending certs.

vctl auth remote list

You will see the pending CSR appear in the list.

USER_ID                              ADDRESS        STATUS
central.central.platform.agent       192.168.56.101 PENDING

Approve the pending CSR using the approve command.

vctl auth remote approve central.central.platform.agent

Run the list command again to verify that the CSR has been approved.

USER_ID                              ADDRESS        STATUS
central.central.platform.agent       192.168.56.101 APPROVED

Node-ZMQ Instance Setup

On the “node-zmq” VM, setup a ZeroMQ based VOLTTRON instance. Using “vcfg” command, install Volttron Central Platform agent, a platform driver agent with a fake driver.

Note

This instance will use old ZeroMQ based authentication mechanism using CURVE keys.

(volttron)user@node-zmq:~/volttron$ vcfg

Your VOLTTRON_HOME currently set to: /home/user/.volttron

Is this the volttron you are attempting to setup? [Y]:
What type of message bus (rmq/zmq)? [zmq]:
What is the vip address? [tcp://127.0.0.1]:
What is the port for the vip address? [22916]:
Is this instance web enabled? [N]:
Will this instance be controlled by volttron central? [Y]:
Configuring /home/user/volttron/services/core/VolttronCentralPlatform.
What is the name of this instance? [volttron1]: collector1
What is the hostname for volttron central? [http://node-zmq]: https://central
What is the port for volttron central? [8080]: 8443
['volttron', '-vv', '-l', '/home/user/.volttron/volttron.cfg.log']
Should the agent autostart? [N]:
Would you like to install a platform historian? [N]:
Would you like to install a platform driver? [N]: y
Configuring /home/user/volttron/services/core/PlatformDriverAgent.
['volttron', '-vv', '-l', '/home/user/.volttron/volttron.cfg.log']
Would you like to install a fake device on the platform driver? [N]: y
Should the agent autostart? [N]: y
Would you like to install a listener agent? [N]:
Finished configuration!

You can now start the volttron instance.

If you need to change the instance configuration you can edit
the config file is at /home/user/.volttron/config

Please note the Volttron Central web-address should point to that of the “central” instance.

Start VOLTTRON instance and check if the agents are installed.

./start-volttron
vctl status

Start Volttron Central Platform on this platform manually.

vctl start --tag vcp

Check the VOLTTRON log in the “central” instance, you will see “authentication failure” entry from the incoming connection. You will need to add the public key of VCP agent on the “central” instance.

../../_images/vc-auth-failure.png

At this point, you can either accept the connection through the admin page or the command line.

Using the admin page:

Navigate back to the platform web admin authentication page. You should see a pending request under the ZMQ Keys Pending Authorization header.

../../_images/zmq_pending_credential_1.png

Accept the credential in the same method as a CSR.

Using the command line:

As with the pending CSR, list the current pending certs and credentials.

vctl auth remote list

You will see the pending ZMQ credential has been added to the list.

USER_ID                              ADDRESS        STATUS
central.central.platform.agent       192.168.56.101 APPROVED
68ef33c4-97bc-4e1b-b5f6-2a6049993b65 127.0.0.1      PENDING

Approve the pending ZMQ credential using the approve command.

vctl auth remote approve 68ef33c4-97bc-4e1b-b5f6-2a6049993b65

Run the list command again to verify that the credential has been approved.

USER_ID                              ADDRESS        STATUS
central.central.platform.agent       192.168.56.101 APPROVED
68ef33c4-97bc-4e1b-b5f6-2a6049993b65 127.0.0.1      APPROVED

Complete similar steps to start a forwarder agent that connects to “central” instance. Modify the configuration in services/core/ForwardHistorian/rmq_config.yml to have a destination VIP address pointing to VIP address of the “central” instance and server key of the “central” instance.

---
destination-vip: tcp://<ip>:22916
destination-serverkey: <serverkey>

Note

Replace <ip> with public facing IP-address of “central” instance and <serverkey> with serverkey of “central” instance. Use the command vctl auth serverkey on the “central” instance to get the server key of the instance

Install and start forwarder agent.

python scripts/install-agent.py -s services/core/ForwardHistorian -c services/core/ForwardHistorian/rmq_config.yml --start

Using the admin page:

To accept the credential, navigate back to the platform web admin authentication page. You should see another pending request under the ZMQ Keys Pending Authorization header.

../../_images/zmq_pending_credential_2.png

Accept this credential in the same method as before.

Using the command line:

To accept the credential via the command line,

vctl auth remote list

You will see the pending ZMQ credential has been added to the list.

USER_ID                              ADDRESS        STATUS
central.central.platform.agent       192.168.56.101 APPROVED
68ef33c4-97bc-4e1b-b5f6-2a6049993b65 127.0.0.1      APPROVED
fb30249d-b267-4bdd-b29a-d9112e6a6082 127.0.0.1      PENDING

Approve the pending ZMQ credential using the approve command.

vctl auth remote approve fb30249d-b267-4bdd-b29a-d9112e6a6082

Run the list command again to verify that the credential has been approved.

USER_ID                              ADDRESS        STATUS
central.central.platform.agent       192.168.56.101 APPROVED
68ef33c4-97bc-4e1b-b5f6-2a6049993b65 127.0.0.1      APPROVED
fb30249d-b267-4bdd-b29a-d9112e6a6082 127.0.0.1      APPROVED

In either case, you should start seeing messages from “collector1” instance on the “central” instance’s VOLTTRON log now.

../../_images/vc-collector1-forwarder.png

Node-RMQ Instance Setup

Note

This instance must have been bootstrapped using –rabbitmq see RabbitMq installation instructions.

Using “vcfg” command, install Volttron Central Platform agent, a platform driver agent with fake driver. The instance name is set to “collector2”.

(volttron)user@node-rmq:~/volttron$ vcfg

Your VOLTTRON_HOME currently set to: /home/user/.volttron

Is this the volttron you are attempting to setup? [Y]:
What type of message bus (rmq/zmq)? [zmq]: rmq
Name of this volttron instance: [volttron1]: collector2
RabbitMQ server home: [/home/user/rabbitmq_server/rabbitmq_server-3.9.7]:
Fully qualified domain name of the system: [node-rmq]:
Would you like to create a new self signed root CA certificate for this instance: [Y]:

Please enter the following details for root CA certificate
    Country: [US]:
    State: WA
    Location: Richland
    Organization: PNNL
    Organization Unit: volttron
Do you want to use default values for RabbitMQ home, ports, and virtual host: [Y]:
2020-04-13 13:29:36,347 rmq_setup.py INFO: Starting RabbitMQ server
2020-04-13 13:29:46,528 rmq_setup.py INFO: Rmq server at /home/user/rabbitmq_server/rabbitmq_server-3.9.7 is running at
2020-04-13 13:29:46,554 volttron.utils.rmq_mgmt DEBUG: Creating new VIRTUAL HOST: volttron
2020-04-13 13:29:46,582 volttron.utils.rmq_mgmt DEBUG: Create READ, WRITE and CONFIGURE permissions for the user: collector2-admin
Create new exchange: volttron, {'durable': True, 'type': 'topic', 'arguments': {'alternate-exchange': 'undeliverable'}}
Create new exchange: undeliverable, {'durable': True, 'type': 'fanout'}
2020-04-13 13:29:46,600 rmq_setup.py INFO:
Checking for CA certificate

2020-04-13 13:29:46,601 rmq_setup.py INFO:
 Creating root ca for volttron instance: /home/user/.volttron/certificates/certs/collector2-root-ca.crt
2020-04-13 13:29:46,601 rmq_setup.py INFO: Creating root ca with the following info: {'C': 'US', 'ST': 'WA', 'L': 'Richland', 'O': 'PNNL', 'OU': 'VOLTTRON', 'CN': 'collector2-root-ca'}
Created CA cert
2020-04-13 13:29:49,668 rmq_setup.py INFO: **Stopped rmq server
2020-04-13 13:30:00,556 rmq_setup.py INFO: Rmq server at /home/user/rabbitmq_server/rabbitmq_server-3.9.7 is running at
2020-04-13 13:30:00,557 rmq_setup.py INFO:

#######################

Setup complete for volttron home /home/user/.volttron with instance name=collector2
Notes:
 - On production environments, restrict write access to /home/user/.volttron/certificates/certs/collector2-root-ca.crt to only admin user. For example: sudo chown root /home/user/.volttron/certificates/certs/collector2-root-ca.crt and /home/user/.volttron/certificates/certs/collector2-trusted-cas.crt
 - A new admin user was created with user name: collector2-admin and password=default_passwd.
   You could change this user's password by logging into https://node-rmq:15671/ Please update /home/user/.volttron/rabbitmq_config.yml if you change password

#######################

The rmq message bus has a backward compatibility
layer with current zmq instances. What is the
zmq bus's vip address? [tcp://127.0.0.1]:
What is the port for the vip address? [22916]:
Is this instance web enabled? [N]:
Will this instance be controlled by volttron central? [Y]:
Configuring /home/user/volttron/services/core/VolttronCentralPlatform.
What is the name of this instance? [collector2]:
What is the hostname for volttron central? [http://node-rmq]: https://central
What is the port for volttron central? [8443]:
['volttron', '-vv', '-l', '/home/user/.volttron/volttron.cfg.log']
Should the agent autostart? [N]:
Would you like to install a platform historian? [N]:
Would you like to install a platform driver? [N]: y
Configuring /home/user/volttron/services/core/PlatformDriverAgent.
['volttron', '-vv', '-l', '/home/user/.volttron/volttron.cfg.log']
Would you like to install a fake device on the platform driver? [N]: y
Should the agent autostart? [N]: y
Would you like to install a listener agent? [N]:
Finished configuration!

You can now start the volttron instance.

If you need to change the instance configuration you can edit
the config file is at /home/user/.volttron/config

Note

The Volttron Central web-address should point to that of the “central” instance.

Start VOLTTRON instance and check if the agents are installed.

./start-volttron
vctl status

Start Volttron Central Platform on this platform manually.

vctl start --tag vcp

Accept the pending CSR request.

Using the admin page:

Go the master admin authentication page and check if there is a new pending CSR request from VCP agent of “collector2” instance.

../../_images/remote_rmq_pending.png

Approve the CSR request to allow authenticated SSL based connection to the “central” instance.

Using the command line:

As before, this can be done via the command line as follows:

vctl auth remote list
USER_ID                                 ADDRESS        STATUS
central.central.platform.agent          192.168.56.101 APPROVED
central.collector2.forwarderagent-5.1_1 192.168.56.101 PENDING
68ef33c4-97bc-4e1b-b5f6-2a6049993b65    127.0.0.1      APPROVED
fb30249d-b267-4bdd-b29a-d9112e6a6082    127.0.0.1      APPROVED

Approve the pending CSR using the approve command.

vctl auth remote approve central.collector2.forwarderagent-5.1_1

Run the list command again to verify that the CSR has been approved.

USER_ID                                 ADDRESS        STATUS
central.central.platform.agent          192.168.56.101 APPROVED
central.collector2.forwarderagent-5.1_1 192.168.56.101 APPROVED
68ef33c4-97bc-4e1b-b5f6-2a6049993b65    127.0.0.1      APPROVED
fb30249d-b267-4bdd-b29a-d9112e6a6082    127.0.0.1      APPROVED

Now go back to the terminal and check the status of Volttron Central Platform agent. It should be set to “GOOD”.

Let’s now install a forwarder agent on this instance to forward local messages matching “devices” topic to external “central” instance. Modify the configuration in services/core/ForwardHistorian/rmq_config.yml to have a destination address pointing to web address of the “central” instance.

---
destination-address: https://central:8443

Start forwarder agent.

python scripts/install-agent.py -s services/core/ForwardHistorian -c services/core/ForwardHistorian/rmq_config.yml --start

Using the admin page:

Go the master admin authentication page and check if there is a new pending CSR request from forwarder agent of “collector2” instance.

../../_images/rmq_remote_forwarder_pending.png

Approve the CSR request to allow authenticated SSL based connection to the “central” instance.

../../_images/rmq_remote_forwarder_accepted.png

Using the command line:

If using command line for this process:

vctl auth remote list
USER_ID                                 ADDRESS        STATUS
central.central.platform.agent          192.168.56.101 APPROVED
central.collector2.platform.agent       192.168.56.103 APPROVED
central.collector2.forwarderagent-5.1_1 192.168.56.103 PENDING
68ef33c4-97bc-4e1b-b5f6-2a6049993b65    127.0.0.1      APPROVED
fb30249d-b267-4bdd-b29a-d9112e6a6082    127.0.0.1      APPROVED

Approve the pending CSR using the approve command.

vctl auth remote approve central.collector2.forwarderagent-5.1_1

Run the list command again to verify that the CSR has been approved.

USER_ID                                 ADDRESS        STATUS
central.central.platform.agent          192.168.56.101 APPROVED
central.collector2.platform.agent       192.168.56.103 APPROVED
central.collector2.forwarderagent-5.1_1 192.168.56.103 APPROVED
68ef33c4-97bc-4e1b-b5f6-2a6049993b65    127.0.0.1      APPROVED
fb30249d-b267-4bdd-b29a-d9112e6a6082    127.0.0.1      APPROVED

Now go back to the terminal and check the status of forwarder agent. It should be set to “GOOD”.

Check the VOLTTRON log of “central” instance. You should see messages with “devices” topic coming from “collector2” instance.

../../_images/vc-collector2-forwarder.png

To confirm that VolttronCentral is monitoring the status of all the 3 platforms, open a browser and type this URL https://central:8443/vc/index.html. Login using credentials (username and password) earlier set during the VC configuration step (using vcfg command in “central” instance). Click on “platforms” tab in the far right corner. You should see all three platforms listed in that page. Click on each of the platforms and check the status of the agents.

../../_images/vc_platforms.png