MDAL - XBOS Docs

Search…
MDAL
​MDAL retrieves, aggregates and formats timeseries data from BTrDB using queries that reference a Brick model. MDAL should be run on the same cluster or node as your BTrDB installation
To query a Brick model, MDAL can either create an embedded HodDB instance or access a remote one
Configuration
BOSSWAVE Entity
MDAL exposes an API over BOSSWAVE, so it needs to be configured with a BOSSWAVE entity ($MDAL_ENTITY) with publish/subscribe capabilities on a namespace. Create the entity and place it in a known location that you don't mind being mounted into a docker container (such as /etc/mdal).
The permissions MDAL needs are all permissions on some prefix ending in s.mdal, e.g.
1
bw2 mkdot -f $PRIVILEGED_ENTITY             \
2
          -t $MDAL_ENTITY                   \
3
          -u $NAMESPACE/services/s.mdal/*   \
4
          -x PC*                            \
5
          -m 'MDAL operation'
Copied!
The URI argument here (minus s.mdal, i.e. $NAMESPACE/services) should be included in the configuration for the MDAL service
Config File
MDAL uses a simple YAML config file canonically called mdal.yaml
1
# this is the reachable address:port (or hostname:port) of the BTrDB instance MDAL uses
2
BTrDBAddress: "localhost:4410"
3
# this is the base URI of the MDAL service. $NAMESPACE should be replaced with the extra
4
Namespace: $NAMESPACE/services
5
# this is the path to the BOSSWAVE entity MDAL uses. This will likely be a volume mount path
6
# inside a container
7
BW2_DEFAULT_ENTITY: path/to/$MDAL_ENTITY
8
# This is the address of the BOSSWAVE agent MDAL uses. 172.17.0.1:28589 is the default
9
BW2_AGENT: 172.17.0.1:28589
10
​
11
# if true, then the HTTP interface is enabled
12
HTTPEnabled: true
13
# the port used by the HTTP interface
14
Port: 8989
15
# the address MDAL listens on
16
ListenAddress: 127.0.0.1
17
# if true, binds to an IPv6 address
18
UseIPv6: false
19
# if specified, MDAL uses port 443 and uses a LetsEncrypt certificate using this hostname
20
TLSHost:
21
​
22
# If true, MDAL runs a local copy of HodDB with the given configuration file.
23
# If using this option, make sure that the Brick model files mentioned in the HodDB configuration
24
# are also mounted into the container (or otherwise reachable by MDAL)
25
EmbeddedBrick: false
26
# the configuration file used by MDAL if using an embedded HodDB instance
27
HodConfig: hodconfig/hodconfig.yaml
28
​
29
# If true, MDAL uses a hosted Brick model by instantiating a HodDB client accessing a HodDB
30
# instance hosted on BOSSWAVE at the given URI. See the HodDB documentation for how to check/grant
31
# $MDAL_ENTITY access to an instance of HodDB
32
RemoteBrick: true
33
BaseURI: xbos/hod
Copied!
Installation
MDAL is shipped as a Docker container image gtfierro/mdal:latest (most recent version is gtfierro/mdal:0.0.2. You can build this container yourself by running make container in a cloned copy of the MDAL repository.
Run with Kubernetes
If you are running Kubernetes on your node/cluster, then you can easily install MDAL by using its Kubernetes file.
Keep in mind that MDAL currently requires a volume mount where the mdal.yaml configuration file is stored.
1
# snippet of MDAL kubernetes file
2
...
3
    spec:
4
        containers:
5
            - name: mdal
6
              image:  gtfierro/mdal:0.0.2
7
              imagePullPolicy: Always
8
              volumeMounts:
9
                - name: mdal
10
                  mountPath: /etc/mdal  # <-- this is how your host folder gets mounted in the container.
11
        volumes:
12
            - name: mdal
13
              hostPath:
14
                path: /etc/mdal   # <-- create this host folder and place the mdal.yaml config file there
Copied!
To execute MDAL as a Kubernetes service, use the following:
1
curl -O https://github.com/gtfierro/mdal/blob/master/kubernetes/k8mdal.yaml
2
# edit /etc/mdal/mdal.yaml and k8mdal.yaml appropriately
3
kubectl create -f k8mdal.yaml
Copied!
Run with Docker
If you are not running Kubernetes, you can invoke the MDAL container directly
1
docker run -d --name mdal -v /etc/mdal:/etc/mdal gtfierro/mdal:latest
Copied!
Don't forget to forward the HTTP port if that interface is enabled
Using
Permissions
mdal check and mdal grant verify and grant permission to use the MDAL service at a given URI to a BOSSWAVE entity
1
$ mdal check -k T8wqsqNgD_NmBeDp1n7Kx1b5yfXDWo8Oqb3y0AQ8-y0= -u xbos/mdal
2
MDAL: Commit: 8284e13 Release: 0.0.3
3
T8wqsqNgD_NmBeDp1n7Kx1b5yfXDWo8Oqb3y0AQ8-y0=
4
Hash: 8z50T3ZQCMvpRH059d-DeZmCnuH9QKOkGsRU-Zz4rdc=  Permissions: C*    URI: 06DZ14k6dhRognGUoSHofD8oS8CUaWwq4tH-FPW13UU=/mdal/*/s.mdal/!meta/lastalive
5
Hash: 4e0ZrYpg2B-7oy_KGpxjxdnYPspKSFb_pr4yccJ-TdQ=  Permissions: P     URI: 06DZ14k6dhRognGUoSHofD8oS8CUaWwq4tH-FPW13UU=/mdal/s.mdal/_/i.mdal/slot/query
6
Hash: dKCtE_OY0QBm3Zq8eKxauaUy7GN1maeiMHL9Zb5eJ1E=  Permissions: C     URI: 06DZ14k6dhRognGUoSHofD8oS8CUaWwq4tH-FPW13UU=/mdal/s.mdal/_/i.mdal/signal/T8wqsqNgD_NmBeDp1n7Kx1b5yfXDWo8Oqb3y0AQ8-y0
7
Key T8wqsqNgD_NmBeDp1n7Kx1b5yfXDWo8Oqb3y0AQ8-y0= has access to archiver at xbos/mdal
Copied!
API
Requests are msgpack-serialized and look like:
1
query = {
2
    "Composition": ["temp"],
3
    "Selectors": [MEAN],
4
    "Variables": [
5
        {"Name": "meter",
6
         "Definition": """SELECT ?meter_uuid WHERE {
7
                        ?meter rdf:type/rdfs:subClassOf* brick:Electric_Meter .
8
                        ?meter bf:uuid ?meter_uuid .
9
                        };""",
10
         "Units": "kW",
11
        },
12
        {"Name": "temp",
13
         "Definition": """SELECT ?temp_uuid WHERE {
14
                        ?temp rdf:type/rdfs:subClassOf* brick:Temperature_Sensor .
15
                        ?temp bf:uuid ?temp_uuid .
16
                        };""",
17
         "Units": "C",
18
        },
19
    ],
20
    "Time": {
21
        "T0": "2017-07-21 00:00:00",
22
        "T1": "2017-08-30 00:00:00",
23
        "WindowSize": '2h',
24
        "Aligned": True,
25
    },
26
}
Copied!
Composition: the order of variables and UUIDs to be included in the response matrix. Variables are defined in Variables key (below) and resolve to a set of UUIDs. UUIDs are the pointers used by the timeseries database and represent a single timeseries sequence. If you want to apply unit conversion to the UUIDs, you will need to have instead define the UUIDs using a variable definition below.
Selectors: for each timeseries stream, we can get the raw data, or we can get resampled min, mean and/or max (as well as bucket count). Each item in the Composition list has a corresponding selector. This is a set of flags:
MEAN: selects the mean stream
MAX: selects the max stream
MIN: selects the min stream
COUNT: selects the count stream (how many readings in each resampled bucket)
RAW: selects the raw data. This cannot be resampled and is mutually exclusive with
the above flags
Combine flags like MEAN|MAX
Variables: each variable mentioned in Composition has a definition here. Each variable needs the following fields:
Name: the name of the variable. Refer to the variable in Composition using this name
Definition: the Brick query that will be resolved to a set of UUIDs. The returned variables need to end in _uuid
UUIDS: a list of additional UUIDs to append to this variable definition (can be used in addition or instead of the Brick query above). To perform unit conversion on data for these UUIDs, put those UUIDs here.
Units: the desired units for the stream; MDAL will perform the unit conversion if possible
Time: the temporal parameters of the data query
T0: the first half of the range (inclusive)
T1: the second half of the range (inclusive). These will be reordered appropriately by MDAL
WindowSize: window size as a Go-style duration, e.g. "5m", "1h", "3d".
Supported units are: d,h,m,s,us,ms,ns
Aligned: if true, then all timesetamps will be the same, else each stream (UUID) will have its own timestamps. Its best to leave this as True
Params: don't touch this for now
Supported unit conversions (case insensitive):
w/watt, kw/kilowatt
wh/watthour, kwh,kilowatthour
c/celsius,f/fahrenheit,k/kelvin
Python Client
1
from xbos.services import mdal
2
client = mdal.MDALClient("xbos/mdal")
3
query = {
4
    "Composition": ["temp"],
5
    "Selectors": [mdal.MEAN],
6
    "Variables": [
7
        {"Name": "temp",
8
         "Definition": "SELECT ?temp_uuid WHERE { ?temp rdf:type/rdfs:subClassOf* brick:Temperature_Sensor . ?temp bf:uuid ?temp_uuid . };",
9
         "Units": "C",
10
        },
11
    ],
12
    "Time": {
13
        "T0": "2017-07-21 00:00:00 PST",
14
        "T1": "2017-08-21 00:00:00 PST",
15
        "WindowSize": '30m',
16
        "Aligned": True,
17
    },
18
}
19
resp = client.do_query(query,timeout=300)
20
print resp.get('error') # see if there's an error
21
df = resp['df']
22
print df.describe()
Copied!
Using
Configuration
Last modified 2yr ago
Copy link
Contents