Pundat

Search…
Pundat
​Pundat subscribes to data published over BOSSWAVE and saves it in BTrDB. Pundat should be run on the same cluster or node as your BTrDB installation.
Configuration
BOSSWAVE Entity
Pundat operates by listening on a set of BOSSWAVE namespaces for archive request messages (documented below) which contain instructions on what streams of data to archive. For this to work, Pundat needs to be configured with a BOSSWAVE entity ($PUNDAT_ENTITY) with subsscription access to each stream it wants to archive. Create the entity and place it in a known location that you don't mind being mounted into a docker container (such as /etc/pundat).
The easiest way to do this is just to grant the following DoT
1
bw2 mkdot -f $PRIVILEGED_ENTITY         \
2
          -t $PUNDAT_ENTITY             \
3
          -u $NAMESPACE/*               \
4
          -x C*                         \
5
          -m "Pundat access to namespace"
Copied!
If you want more fine-grained control, then you can grant the following DoT to ($PUNDAT_ENTITY) to have it discover archive requests
1
bw2 mkdot -f $PRIVILEGED_ENTITY                     \
2
          -t $PUNDAT_ENTITY                         \
3
          -u "$NAMESPACE/*/!meta/archiverequest"    \
4
          -x 'C*'                                   \
5
          -m "Pundat discover archive requests"
Copied!
and then grant a direct subscription DoT for each individual stream
1
bw2 mkdot -f $PRIVILEGED_ENTITY             \
2
          -t $PUNDAT_ENTITY                 \
3
          -u "$NAMESPACE/sensors/s.sensor/123/i.xbos.temperature_sensor/signal/info"  \
4
          -x 'C'                            \
5
          -m 'archiver access to sensor stream'
Copied!
Keep in mind this is very tedious and error-prone.
$PUNDAT_ENTITY also needs permission to operate on a namespace ($GILES_BW_NAMESPACE). The DoT to grant is
1
bw2 mkdot -f $PRIVILEGED_ENTITY             \
2
          -t $PUNDAT_ENTITY                 \
3
          -u $NAMESPACE/s.giles/*           \
4
          -x PC*                            \
5
          -m 'Pundat archiver endpoint'
Copied!
Environment Variables
The environment variables Pundat expects are:
BTRDB_SERVER: this is the local address and port of your BTrDB endpoint
MONGO_SERVER: this is the local address and port of a MongoDB server (does not need persistent storage)
GILES_BW_ENTITY: this is the container-local path to the entity Pundat should use
GILES_BW_NAMESPACE: this is the namespace Pundat operates on
GILES_BW_ADDRESS: this is the address of the BOSSWAVE agent Pundat should use. If Pundat is running in a Docker container (either through Kubernetes or otherwise), then we recommend configuring the BOSSWAVE agent to listen on the default Docker interface. In the BOSSWAVE config file /etc/bw2/bw2.ini, make sure you have the following lines:
1
  [oob]
2
  ListenOn=172.17.0.1:28589
Copied!
GILES_BW_LSTEN: this is a space-separated list of all of the namespaces Pundat wants to subscribe to
COLLECTION_PREFIX: this is the prefix Pundat adds to its collection names for MongoDB, which helps avoid collisions.
Installation
Pundat is shipped as a Docker container image gtfierro/pundat:latest (most recent version is gtfierro/pundat:0.3.4). You can build this container yourself by running make container in a cloned copy of the Pundat repository.
Run with Kubernetes
If you are running Kubernetes on your node/cluster, then you can easily install Pundat by using its Kubernetes file
1
curl -O https://github.com/gtfierro/pundat/blob/master/kubernetes/pundat.yaml
2
# edit pundat.yaml
3
kubectl create -f pundat.yaml
Copied!
Run with Docker
If you are not running Kubernetes, you can invoke the Pundat container directly
1
# install jq:  https://stedolan.github.io/jq/
2
docker run -d --name pundat-mongo mongo:latest
3
MONGOIP=$(docker inspect pundat-mongo | jq .[0].NetworkSettings.Networks.bridge.IPAddress | tr -d '"')
4
docker run -d --name pundat -e BTRDB_SERVER=<btrdb ip>:4410 \
5
                            -e MONGO_SERVER=$MONGOIP:27017 \
6
                            -e GILES_BW_ENTITY=/etc/pundat/<archiver entity.ent> \
7
                            -e GILES_BW_NAMESPACE=<namespace to deploy on> \
8
                            -e GILES_BW_ADDRESS=172.17.0.1:28589 \
9
                            -e GILES_BW_LSTEN="space-separated list of namespaces to listen on" \
10
                            -e COLLECTION_PREFIX="pundat_" \
11
                            -v <host config directory>:/etc/pundat \
12
                            gtfierro/pundat:latest
Copied!
Archiving
A stream is the unit of archival and data access. A stream consists of:
a timeseries (a single progression of <time, value> pairs)
metadata (a bag of key = value pairs)
a UUID (a unique 16-byte identifier)
In BOSSWAVE, drivers simply push their data into the overlay network where anyone with permission can listen to the data. In order to archive data in BOSSWAVE, instead of explicitly instructing a driver to push data to a particular archiver (which would require prior knowledge of an archiver), we instead tell the archiver to listen to URIs on which drivers or services publish data.
Upon startup, Pundat is configured to listen to a set of BOSSWAVE namespaces. On each of these namespaces, Pundat listens for published messages on the !meta/archiverequest key (using the subscription <namespace/*/!meta/archiverequest). These messages, which take the form of ARs or Archive Requests (described below) are by convention persisted on URIs. ARs are added/removed/inspected using the pundat command-line tool.
Archive Request Structure
Field Name
Required?
Data Type
Role
ArchiveURI
yes
string
The URI the archiver subscribes to receive data. This URI can contain wildcards (+ and *)
Name
yes
string
A human-readable name for this stream that serves to disambiguate it from other streams that might be produced from the same URI
PO
yes
string
Which type of payload the archiver should extract from received messages. If multiple payloads with the PO in the same message, the archiver will operate on each of them
Value
yes
string
The OB expression used to extract the numerical value from the published message. This will be interpreted as a 64-bit float.
Unit
yes
string
The engineering units that characterize this stream of data
Time
no
string
The OB expression used to extract the time to be associated with the value. If provided, the archiver uses the TimeParse field to determine how to parse the extracted value. If this field is not provided, then the archiver just associates the server's current timestamp with the received record
There are two additional (optional) fields. URIMatch is a regular expression executed against each unique URI that pundat sees when subscribing to ArchiveURI. Capture groups are supported. URIReplace is a rewrite of the URI that gets used as the stream's name when it is inserted into BTrDB. By default, Pundat will use the full URI of the subscription, which can get unwieldy.
For published drivers, there is usually an archive.yml file already written and it should be a simple matter to edit these for your own deployments.
How to attach/remove Archive Requests
First, install pundat:
1
$ go get -u github.com/gtfierro/pundat
2
$ go install github.com/gtfierro/pundat
3
$ pundat
4
USAGE:
5
   pundat [global options] command [command options] [arguments...]
6
​
7
COMMANDS:
8
     ...
9
     listreq   List all archive requests persisted on the given URI pattern
10
     rmreq     Remove archive requests specified by file
11
     addreq    Load archive requests from config file
12
     nukereq   Remove ALL archive requests attached at the given URI
13
     ...
14
​
15
GLOBAL OPTIONS:
16
   --help, -h     show help
17
   --version, -v  print the version
Copied!
To add/remove archive requests, we use a special YAML file which we will refer to as archive.yml. This file contains archive requests and other relevant configuration.
Here is an example archive.yml file that contains all of the relevant components. This archive file is for a hypothetical service that publishes data for several sensors on URIs like scratch.ns/service/s.sensor/<sensorname>/i.board/signal/reading; we want to extract the "Supply Air Flow" and "Supply Air Temperature Setpoint" fields from the published messages for all sensors.
1
# For long URIs, it can be helpful to factor out a long shared prefix to make the rest of the file more readable.
2
# When interpreted, "Prefix" will be prepended to all URIs mentioned in the file
3
Prefix: scratch.ns/bms/
4
  - AttachURI:
5
    ArchiveURI: s.bms/+/i.xbos.ahu/signal/info
6
    Value: supply_air_flow_sensor
7
    Name: supply_air_flow_sensor
8
    Unit: cfm
9
    Time: time
10
    PO: 2.1.1.10/32
11
    URIMatch: .*/s.bms/(.*)/i.xbos.ahu/.*
12
    URIReplace: sdh/bms/ahus/supply_air_flow_sensor/$1
13
​
14
  - AttachURI:
15
    ArchiveURI: s.bms/+/i.xbos.ahu/signal/info
16
    Value: supply_air_temperature_setpoint
17
    Name: supply_air_temperature_setpoint
18
    Unit: F
19
    Time: time
20
    PO: 2.1.1.10/32
21
    URIMatch: .*/s.bms/(.*)/i.xbos.ahu/.*
22
    URIReplace: sdh/bms/ahus/supply_air_temperature_setpoint/$1
Copied!
Saving this as sensor_archive.yml, we can attach the Archive Requests using
1
$ pundat addreq -c sensor_archive.yml
Copied!
and then verify that it was attached by
1
$ pundat listreq scratch.ns/bms/*
2
RETRIEVING from scratch.ns/bms/*/!meta/archiverequest
3
┌────────────────
4
├ ARCHIVE REQUEST
5
├ PublishedBy: dgKG0DKVUw40PmpY2UqpBDFWdvKD5-KNyXQun2jQkNs=
6
├ Archiving: scratch.ns/bms/s.bms/+/i.xbos.ahu/signal/info
7
├ Name: supply_air_temperature
8
├ Unit: F
9
├ PO: 2.1.1.10/32
10
├ UUID: Autogenerating UUIDs
11
├ Value Expr: supply_air_flow_sensor
12
├┌
13
│├ Time Expr: time
14
│├ Parse Time:
15
│└
16
├ URI Match: .*/s.bms/(.*)/i.xbos.ahu/.*
17
├ URI Replace: scratch.ns/bms/ahus/supply_air_flow_sensor/$1
18
└────────────────
19
┌────────────────
20
├ ARCHIVE REQUEST
21
├ PublishedBy: dgKG0DKVUw40PmpY2UqpBDFWdvKD5-KNyXQun2jQkNs=
22
├ Archiving: scratch.ns/bms/s.bms/+/i.xbos.ahu/signal/info
23
├ Name: supply_air_temperature
24
├ Unit: F
25
├ PO: 2.1.1.10/32
26
├ UUID: Autogenerating UUIDs
27
├ Value Expr: supply_air_temperature
28
├┌
29
│├ Time Expr: time
30
│├ Parse Time:
31
│└
32
├ URI Match: .*/s.bms/(.*)/i.xbos.ahu/.*
33
├ URI Replace: scratch.ns/bms/ahus/supply_air_temperature/$1
34
└────────────────
Copied!
To remove requests, use rmreq instead of addreq when specifying the archive YAML file, or just use the nukereq command on a URI to remove all ARs on that URI.
NOTE: currently the archiver only respects removal of ARs on restart
Using
CLI tool
The Pundat commandline tool has a couple utilities you may find helpful for general usage
pundat scan: this locates the base URI of an archiver on a given namespace. This is helpful for liveness checks
1
  $ pundat scan ucberkeley
2
  Found Archiver at:
3
       URI        -> ucberkeley
4
       Last Alive -> 2018-01-20 18:27:16 +0000 UTC (9.829507254s ago)
Copied!
pundat check and pundat grant verify and grant permission for using the archiver to a BOSSWAVE entity
1
  $ pundat check -k gabe -u ucberkeley
2
  Found archiver at: ucberkeley (alive 8.508192267s ago)
3
  Hash: wTx2G1LVrAo_HVj6jVIFWY7Bl6Q7ge9bDft0Dt5w3UQ=  Permissions: C*    URI: nvrnSE4pJe4ZMO3WQdb-EPi5iwuzmTVUpk6XNNRGYsc=/*/s.giles/!meta/lastalive
4
  Hash: 908QbAUPnu5rm9h_3L65EQtPOxPVPvKibt2KpZxG_CQ=  Permissions: P     URI: nvrnSE4pJe4ZMO3WQdb-EPi5iwuzmTVUpk6XNNRGYsc=/s.giles/_/i.archiver/slot/query
5
  Hash: _s9XHmbnqk8RrEORpG84jW2qnAdc-6iP15esIP4rymI=  Permissions: C     URI: nvrnSE4pJe4ZMO3WQdb-EPi5iwuzmTVUpk6XNNRGYsc=/s.giles/_/i.archiver/signal/T8wqsqNgD_NmBeDp1n7Kx1b5yfXDWo8Oqb3y0AQ8-y0,queries
6
  Key dgKG0DKVUw40PmpY2UqpBDFWdvKD5-KNyXQun2jQkNs= has access to archiver at ucberkeley
Copied!
pundat range checks which ranges of data a BOSSWAVE entity can see
Query Language
Note: this is a legacy interface. For data access + metadata, we recommend using HodDB and MDAL
Documentation available on the Pundat wiki​
BTrDB
Configuration
Last modified 2yr ago
Copy link
Contents
Configuration
BOSSWAVE Entity
Environment Variables
Archive Request Structure
How to attach/remove Archive Requests
Using
CLI tool
Query Language