nrel / boptest-service

BOPTEST-Service

This software extends BOPTEST to a web service architecture, which enables support for multiple clients and multiple simultaneous tests at a large scale. This is a containerized design that can be deployed on a personal computer, however the software is targeted at commercial cloud computing environments such as AWS. For details about BOPTEST, refer to the project homepage.

The canonical BOPTEST source code is incorporated into this repository as a git subtree located at <project-root>/boptest and used under the terms of the license located at <project-root>/boptest/license.md.

BOPTEST-Service is a sibling of Alfalfa, which follows the same architecture, but adopts a more general purpose API to support interactive building simulation, whereas the BOPTEST API is designed around predetermined test scenarios.

flowchart LR
    A[API Client] <--> B[Web Frontend]
    subgraph cloud [Cloud Deployment]
            B <--> C[(Message Broker)]
            C <--> D[Worker 1]
            C <--> E[Worker 2]
            C <--> F[Worker N]
        subgraph workers [Worker Pool]
            D
            E
            F
        end
    end

Getting Started

A simple demonstration of using the BOPTEST API is available in this interactive tutorial. This tutorial leverages a publicly available deployment of BOPTEST-Service, which can be reached at https://api.boptest.net.

BOPTEST-Service APIs

The core BOPTEST APIs are documented as part of the upstream BOPTEST project. The BOPTEST-Service defines a number of APIs on top of BOPTEST for the purpose of managing test cases and running tests.

Description	Request
List official BOPTEST test cases.	GET testcases
List unofficial test cases in a namespace.	GET testcases/{namespace}
List private user test cases. (Auth required)	GET users/{username}/testcases/
Select a test case and begin a new test. (Auth optional)	POST testcases/{testcase_name}/select
Select a test case from the namespace and begin a new test. (Auth optional)	POST testcases/{namespace}/{testcase_name}/select
Select a private user test case and begin a new test. (Auth required)	POST users/{username}/testcases/{testcase_name}/select
Get test status as Running or Queued	GET status/{testid}
Stop a queued or running test.	PUT stop/{testid}
List tests for a user. (Auth required)	GET users/{username}/tests

The family of the select APIs are used to choose a test case and begin a running test. Select returns a testid which is required by all APIs that interact with the test or provide test information.

OpenAI Gym Interface

An OpenAI-Gym environment for BOPTEST is available.

Building and Running on a Personal Computer

1. Clone this repository.

git clone https://github.com/NREL/boptest-service.git

2. Install Docker.

3. Use Docker to build and run BOPTEST-Service.

docker compose up web worker provision

4. In a separate process, use the core BOPTEST APIs as well as BOPTEST-service APIs to interact with the test case using your test controller.

5. Shutdown the test case by the command docker compose down executed in the root directory of this repository.

Kubernetes Based Deployment

NREL maintains a helm chart for Kubernetes based deployments of BOPTEST-Service.

Running the developer Test Suite

Testing is based on the BOPTEST test suite with small adaptations to conform to the BOPTEST-Service API. Follow the README for more information.

Project Management

Development is tracked by three branches within the project repository.

• develop maintains the latest stable version of the project.
• main tracks the latest release version and corresponds to the current production deployment on https://api.boptest.net. A tag is also created for each release version.
• experimental is for versions of the project under evaluation and corresponds to a deployment on the "dev" cluster at https://api.dev.boptest.net.

boptest-service's Issues

socket API

I believe REST is less efficient than a WebSocket for BOPTEST's use case. The purpose of this issue is to implement and compare the performance of a WebSocket implementation of the BOPTEST API.

Inactivity timeout is not working

Need to be able to autoscale workers

Unauthorized GET users/{username}/testcases/{testcase_name}

"When I try GET users/{username}/testcases/{testcase_name} without providing any authorization I get a 404 response. But I think it should be a 401, which is what I get if I do GET users/{username}/testcases/. Basically, if I don't have permission to view all test cases with a username, I shouldn't have permission to check if a single test case is present either. "

This scenario should return a 401.

There is no record of test result metadata

BOPTEST Service stores test results as object storage in Minio/s3, however we lack key metadata. Currently the test objects are indexed only by testid. It would be nice to use the s3 tagging feature to record metadata for the user identifier and the test case name. If we did this we could search results by user and test case name. For example we could determine how many times a test has been run by a certain user.

@haraldwalnum

Improve the BOPTEST Solver time

We noticed that the inbuilt solver of BOPTEST is much slower/failed to converge compared to the Dymola solver, especially for many MegaBOP testcases. It would be great to see improvements here.

Setup GitHub Action to run tests

Add a WebSocket API

Use redis instead of sqs for job queing

We built around SQS (and the goaws clone) for queing jobs before Redis was introduced. Now redis is a key component and should be capable of providing a simple first in first out task queue that meets the requirements. I propose we switch to that, which will remove a dependency and simplify the software architecture.

Need to add APIs to change BOPTEST timeouts.

This is relevant when a large number of tests are being run simultaneously (like in MegaBOP). Waiting for all testcases to do something might cause a timeout with the default values.

Document service specific APIs and features

Need method to get all my test-ids

Remove vestiges of Alfalfa from the object storage

The Minio/S3 storage still uses some names carried over from the Alfalfa ancestry. Specifically,

1. The main bucket is named "alfalfa". We should change this to "boptest".
2. Test results are stored under "simulations", but in the context of BOPTEST it would be more conisistent to store test results under "tests"

Websocket crashes the web server if a timeout occurs while messaging the worker

Need to get status of my workers

Need a method to get status of all tests

Simulation output is going to the worker root

The working directory of a boptest-service test is the workers root directory. Meanwhile pyfmi generates most output to the current directory, therefore while we desire the output to be put into the /simulation/ directory, most of it is going to the worker root. This means that when simulation output is archived and pushed to object storage, we miss capturing most of the key output.

Add API to download the test results archive

This is a transfer from ibpsa/project1-boptest#462.

add tests for service specific APIs

Remove model.fmu from test results

Link BOPTEST user auth to dashboard

The basic links are currently in place, but we need better logging and diagnostics.

Remove Mongo

It is barely used. The one use case is to keep track of the testcases. Instead of mongo we can just listObjects on the s3 bucket.

ValueError: No JSON object could be decoded

Problem

When using boptest-service to run the test cases multizone_office_simple_air or multizone_residential_hydronic with scenario peak_heat_day and control step size equal to one month (2592000 seconds), the service eventually returns the error when using the /advance call:

Traceback (most recent call last):
  File "demo_service_error.py", line 43, in <module>
    y = requests.post("{0}/advance/{1}".format(url, testid), data=u).json()["payload"]
  File "/home/dhbubu18/.local/lib/python2.7/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

After the error ends the process, if I make a get /kpi request to the test case, it returns the expected KPI values for the given scenario.

Other Notes

When setting the control step to one day (86400 seconds, note that this is less than the length of the scenario, which is two weeks), the error does NOT occur, and the test runs to completion OK. Note also that when deploying BOPTEST v0.3.0 core locally and running the same experiment with multizone_residential_hydronic, the test runs to completion OK as well. This problem also wasn't seen when running single-zone test cases from BOPTEST.

Expected KPIs for multizone_residential_hydronic:

"cost_tot": 0.7932061254859891,
"emis_tot": 1.425504109372381,
"ener_tot": 8.141192268442058,
"idis_tot": 9114.594708949711,
"pdih_tot": null,
"pele_tot": 0.0017390231869758264,
"pgas_tot": 0.09592720495532536,
"tdis_tot": 22.003316309231238,
"time_rat": NaN

Python Code to Reproduce

# GENERAL PACKAGE IMPORT
import requests
#------------------------------------------------------------------------------

# SET PARAMETERS
# Set URL for service
url = "http://api.boptest.net"
# Set test case name to run
test_case = 'multizone_residential_hydronic'
# Set testing scenario
scenario = {"time_period":"peak_heat_day",
            "electricity_price":"dynamic"}
# Set control step
step = 3600*24*30
#------------------------------------------------------------------------------

# RUN TEST
# Select test case on service
testid = requests.post("{0}/testcases/{1}/select".format(url, test_case)).json()["testid"]
print('Selected test case {0} with test id: {1}'.format(test_case, testid))
# Set control step
requests.put("{0}/step/{1}".format(url, testid), data={"step":step})
print('Set control step.')
# Set test case scenario
print('Initializing test scenario...')
y = requests.put("{0}/scenario/{1}".format(url, testid), data=scenario).json()["payload"]["time_period"]
print('Done initializing test scenario.')
# Record test start time
start_time = y["time"]
# Simulation Loop
print('Running test...')
while y:
    u={}
    # Advance simulation with control signal
    y = requests.post("{0}/advance/{1}".format(url, testid), data=u).json()["payload"]
print('Done running test.')
#------------------------------------------------------------------------------

Need a method to kill all my running tests

GET /testscases/<namespace> for invalid namespace should return 404

For example https://api.dev.boptest.net/testcases/notanamespace returns empty list. It should instead return 404 not found.

User specific test cases

The official BOPTEST testcases should be public and available to everyone via the /testcases api, however we should also support user specific testcases.

Move boptest-service into the NREL/boptest-service repo

Currently there is a boptest-service branch here that should move into this repo. The canonical BOPTEST source code should be included as a git subtree.

get /testcases/<test_case_name> has ambiquous or maybe even wrong behavior

"When I try GET https://api.dev.boptest.net/testcases/bestest_air, I get an empty list in return (though bestest_air should exist). Same as when I try GET https://api.dev.boptest.net/testcases/bestest_airppp (though bestest_airppp doesn't exist). I'm not sure what the expected response is in terms of errors and list items, but I don't think both should return an empty list."

We need to clarify the design and then adjust the implmentation to make sure that it does what we want. This endpoint was created to confirm the existence of a test case.

Implement a WebSocket API

Need a "watchdog" for worker jobs

Sometimes jobs can get hung up and the event loop is stalled. This means api calls like stop won't work until the event loop is unblocked (which might not ever happen)

To solve this we should implement a watchdog on the worker. If the job doesn't provide a "tick" to indicate that it is alive, then the job needs to be terminated.