runwayml / hosted-models

Interact with Runway Hosted Models with only a few lines of code!

Hosted Models JavaScript SDK

A small library for interfacing with RunwayML Hosted Models using only a few lines of code. Works in both Node.js and the Browser.

Benefits

This library is a thin wrapper around the Hosted Models HTTP API. It provides a few benefits:

• Abstracts the HTTP requests to the Hosted Model, so you don't have to make them directly.
• Simplifies authorization for private models, just provide your token in the new HostedModels({ url, token }) constructor.
• Automatically retries failed requests if they timed out while the model wakes up or are blocked due to rate-limiting.
• Includes friendly error reporting with messages for humans, so you can better understand why something went wrong.

If your project is written in JavaScript, we encourage you to use this library!

Examples

See the examples/ directory for a full list of examples.

Node.js / Module Syntax

If you are using Node.js or packaging your front-end code via a bundler, you can install the module using npm.

npm install --save @runwayml/hosted-models

const { HostedModel } = require('@runwayml/hosted-models');

const model = new HostedModel({
  url: 'https://my-model.hosted-models.runwayml.cloud/v1',
  token: 'my-private-hosted-model-token',
});

const prompt = 'Hey text generation model, finish my sentence';
model.query({ prompt }).then(result => console.log(result));

Browser

If you prefer to access the library in the Browser using a <script> tag, you can include the code snippet below in your HTML files. Replace hosted-models.js with hosted-models.min.js if you prefer a minified build for production environments.

<script src="https://cdn.jsdelivr.net/npm/@runwayml/[email protected]/dist/hosted-models.js"></script>

This injects the library into the window and exposes it via the rw namespace.

const model = new rw.HostedModel({
  url: 'https://my-model.hosted-models.runwayml.cloud/v1',
  token: 'my-private-hosted-model-token',
});

const prompt = 'Hey text generation model, finish my sentence';
model.query({ prompt }).then(result => console.log(result));

Usage

This library is super simple to use; It exposes a single HostedModels class with only four methods:

• HostedModel
  • .info()
  • .query()
  • .isAwake()
  • .waitUntilAwake()

Note: Be sure to use rw.HostedModel() if you are including the library via a <script> in the Browser.

HostedModels Constructor

The HostedModel constructor takes a configuration object with two properties, url and token (required only if the model is private).

const model = new HostedModel({
  url: 'https://example-text-generator.hosted-models.runwayml.cloud/v1',
  token: 'my-private-hosted-model-token', // not required for public models
});

.info() Method

This method returns the input/output spec expected by this model's query() method.

const info = await model.info();
console.log(info);
//// Note: These values will be different for each model
// {
//   "description": "Generate text conditioned on prompt",
//   "name": "generate_batch",
//   "inputs": [
//     {
//       "default": "",
//       "description": null,
//       "minLength": 0,
//       "name": "prompt",
//       "type": "text"
//     },
//     ...
//   ],
//   "outputs": [
//     {
//       "default": "",
//       "description": null,
//       "minLength": 0,
//       "name": "generated_text",
//       "type": "text"
//     },
//     ...
//   ]
// }

.query() Method

query() is used to trigger the model to process input, and return output.

const result = await model.query({
  prompt: 'Hey text generation model, finish my sentence',
});
console.log(result);
//// Note: These values will be different for each model
// {
//   generated_text: 'Hey text generation model, finish my sentence please.',
//   encountered_end: true
// }

.isAwake() Method

The isAwake() method returns true if the model is already awake and processing requests quickly, or false if it is still waking up. A model that is waking up can still process all requests (e.g. info() and query()), they just might take a bit longer to respond. Learn more about Hosted Model states here.

if (!model.isAwake()) {
  showLoadingScreen(); // this is a fake function for demonstration
}

// A model doesn't have to be awake for you to make requests to it
await model.query(input);

Note: A model that is waking up can still process all requests (e.g. info() and query()), they just might take a bit longer to respond. Learn more about Hosted Model states here.

.waitUntilAwake() Method

The .waitUntilAwake() method returns a promise that will resolve as soon as the model is awake. This is useful if you'd prefer to wait to perform some action until after the model can be expected to process them quickly.

setLoading(true); // this is a fake function for demonstration
await model.waitUntilAwake();
setLoading(false);

setProcessing(true); // this is also a fake function for demonstration
await model.query(input);
setProcessing(false);

Note: A model that is waking up can still process all requests (e.g. info() and query()), they just might take a bit longer to respond. Learn more about Hosted Model states here.

License

This library is released under the terms of the MIT license.

CHANGELOG

You can view a list of changes here.

hosted-models's Issues

Inactive hosted model can't be woken up, 404s my request

This is similar to #3 but I see that one was closed because the user had a $0 balance. In my case I only have a single hosted model, in the unsubscribed mode. It set itself to inactive, and hovering over the status says:

Leading me to believe instantiating the model and calling waitUntilAwake would kick it into gear. However, any attempt to instantiate the model fails with:

Unhandled error during request: Error: Model not found. Make sure the url is correct and that the model is "active".
  at new NotFoundError (/vercel/workpath0/node_modules/@runwayml/hosted-models/build/HTTPErrors.js:38:23)
  at HostedModel.<anonymous> (/vercel/workpath0/node_modules/@runwayml/hosted-models/build/HostedModel.js:230:39)

So, how can I programmatically wake a model up???

Instantiating and/or querying the model results in 404

Hey there. Not sure if this is a bug in the JS library, Runway's backend, or if I'm getting silently rate-limited— so please let me know if there is a better place to post this!

My app encounters a PermissionDeniedError (as defined in src/HTTPErrors.ts:23) with the message 'Model not found. Make sure the url is correct and that the model is "active".' The API key is valid.

The network activity shows failures (404) for:

• GET https://MODEL-NAME.hosted-models.runwayml.cloud/v1/
• POST https://MODEL-NAME.hosted-models.runwayml.cloud/v1/query

Using waitUntilAwake makes no difference, as that call will fail, too.

Every time I encounter this error, I find that the hosted model has become Inactive, as shown on the dashboard at https://app.runwayml.com/hosted-models. I can then toggle it to Active, and perhaps make 1 request through my app before the model is once again Inactive.

This behavior makes me think there might be either a backend issue or some kind of rate-limiting. For reference, I have been making around 50-60 requests/hour while testing. If rate-limiting is the case here, it would be nice if the model returned 429 Too Many Requests instead of silently inactivating.