solid-contrib / solid-rest Goto Github PK

a client-side API which supports any backend for Solid requests

JavaScript 92.11% HTML 7.89%

solid-rest's Introduction

Solid-Rest

Treat any storage backend as a Solid pod

Overview

Solid-Rest translates Solid requests into backend requests and backend responses into Solid responses. This means that any storage system that has a Solid-Rest plugin may be treated as a pod. Currently there are plugins for file and dropbox which means that any app that uses Solid-Rest can address file:// and dropbox:// URIs the same way as a Solid pod https:// URI and expect the same responses with some exceptions : permissions are not handled by Solid .acls, they are based on the underlying file or cloud permissions; collaborative tools such as chat are not available. These backends can now be addressed with most Solid libraries (e.g. rdflib) and apps (e.g. the databrowser).

Plugins for ssh, in-memory storage, in-browser storage (indexedDB, localStorage, Native File API) are in development.

Although Solid-Rest can be used stand-alone, it is best used in conjunction with other libraries, especially Solid-Node-Client, a nodejs client for Solid. Solid-Node-Client comes preloaded with the Solid-Rest-File plugin, so it will be transparently included in anything using that library.

solid-rest's People

Contributors

Stargazers

Watchers

Forkers

kustomzone mikeadams1 cxres michielbdejong bourgeoa agentydragon standardgalactic intensifier doctorbud

solid-rest's Issues

file do not pass tests since commits from july 29

@jeff-zucker Since last commits tests do not pass for file://

Testing for parent folder exists getObjectType() always create the missing folders if not exists and return true
I rolled manually back the last commits and tests still fails
i tried different code for getObjectType() and always run in the same difficulties

I am running node.js v14.6 and I did not change version for a long time.

Tests for local storage ls:// succeeds all

Implement OPTIONS method and fix Allow header

See the discussion here : jeff-zucker/solid-auth-cli#2 (comment)

Rename default branch to main

Can someone with access to https://github.com/solid/solid-rest/settings/branches change the default branch name from 'master' to 'main'?

I'm doing this for all repos in the Solid OS stack, but don't have access to this one.

catch 409 error in PATCH

bug. See PR #54

update to solid v0.10 spec

solid-rest improvements :

n3-patch tests
contentType mapper when extension missing or incompatible --> $.ext
add metadata to container representation
http front server
- express server
- dataBrowser with an index.html at root
- redirect to dataBrowser programmatically like in NSS
- default card in server
- permanent location for /profile/ and /settings/ : see installDir in data-kitchen

It will be nice to have a way to run the test-suite and specification tests

Support of access restrictions (acl)

Is there some kind of .acl support intended in the future?

I don't quite know yet, how authentication would work, but I guess that it could be done to some extent (e.g. for apps with the origin).

implement delete resource with links with acl:write (atomic delete)

@jeff-zucker
Since NSS 5.4.0 solid as implemented atomic delete with acl:Write
Actually rest.js only delete the item resource and not all the auxiliary resources (ending with .acl, .meta and .meta.acl, ...)

To delete auxiliary resource you need to delete each resource independently with a need for acl:Control if you want to delete an ACL resource
For example this is done in solid-file-client that has implemented _deleteItemWithLinks()

  async _deleteItemWithLinks (itemUrl) {
    const links = await this.getItemLinks(itemUrl, { links: LINKS.INCLUDE })
    if (links.meta) {
      await this._deleteItemWithLinks(links.meta)
    }
    if (links.acl) {
      await this.delete(links.acl)
    }

    // Note: Deleting item after deleting links to make it work for folders
    //       Change this if a new spec allows to delete them together (to avoid deleting the permissions before the folder)
    return this.delete(itemUrl)
  }

The function should be replaced by

  async _deleteItemWithLinks (itemUrl) {
     return this.delete(itemUrl)
  }

A bunch of tests are failing because they use solid-rest and solid-rest only mock a simple delete and not an atomic delete
A solution is to implement a deleteItemWithLinks() in solid-rest

Fix broken links to solid-node-client

In memory storage

Have you already written an in-memory storage? Similar to localStorage but instead of storing it via setItem(key, string) to store it inside an object with data[key] = string? I don't know if there are any limitations regarding binary files, but I guess it's the same as for localStorage, so it should work fine and be easy to implement.

BrowserFS implementation

Hi guys, running opening tests/browser-test.html in the browser, i got a TypeError: can't access property "basename", u is undefined

and same with my webapp

Do you have any idea where it comes from ?
THxs

Reconsider handler initialization

Currently, handler initialization is based on checking the window object. I presume this has to do with detecting if the environment is a browser. However, this check is a problem in nwjs/electron environments where both window and node are defined. In such cases we still want to create file handlers and can even use native localstorage of the browser. At least checks should be made for both window and process.

I am not sure of the best strategy to manage this tough and would like to hear a bit more about the design intent in automatically initializing handlers. I see that one can pass options through solid-node-client (incl. options.handlers) but these are not documented. That is why I have refrained from putting up a PR!

Somewhat unrelated but relevant to the issue, I think in the long term solid-rest should be decoupled from solid-node-client (and other providers should be decoupled as well, as this list will become unwieldingly large if this project takes off). Each type of source then should be a plugin that is invoked based on user-installation (perhaps done so automatically on first invocation of the scheme) and url scheme. In fact, this is the approach I am taking in including solid in my application. Please consider this in your future plans!

Some remaining differences between NSS and solid-rest

@jeff-zucker
Apart from resourceMapper, here are the points I can raise for compatibility between NSS and solid-rest :
(I have no understanding of Link HEADER or OPTIONS)

slug
- without extension add ext to slug : slug = slug + ext (Tim's recommendation for all resources, can be forced for POST)
- 400 slug MUST not contain / or | ( slug.match(//|||:/) ) (I seem to remember that it is for mashlib)
.acl or .meta ext MUST have text/turtle contentType : 415 required (spec says RDF but it has been agreed text/turtle for NSS)
when creating Container create default META (everything works without it)
container content displayed as turtle doc contains .meta
content Type default to application/octet-stream and not text/turtle (since NSS v5.xx and default mime-type)
forbidden to delete root acl, root meta (NSS POD broken)
/profile/card contentType is text/turtle and file is card$.ttl ( the last point I suppose can be left to resourceMapper)
Resource name must be unique (Container without / at end and Resource) (not in NSS actually)

Use cross-fetch instead of isomorphic-fetch

(Opening this issue here, so it doesn't get lost)

Copied from jeff-zucker/solid-file-client#59 (comment):

Ok, after some digging into the the node_modules dependencies, I've found out that ismorphic-fetch is using an outdated version. It specifies "node-fetch": "^1.0.1" (=1.x) in its dependencies, so it sticks with the newest 1.x version of node-fetch. But node-fetch is already at v2.0.6, the blob support was added in v2.0.0 as experimental feature (see their changelog).

TL;DR: isomorphic-fetch uses an outdated version of node-fetch which means it doesn't support blobs

My suggestion for solid-auth-cli is, to use cross-fetch instead. It is basically the same as isomorphic-fetch, but with newer versions. And the last commit there was last week, whereas the last commit in isomorphic-fetch was in 2016.

Recursive Make-Containers in localStorage

Revisit @Otto-AA's patch at #10 which, for now is not implemented.

implement a resource mapper like solid NSS

This is a way to store in file and retrieve from file the Resource and contentType :

card -> card$.ttl
file.ttl text/turtle -> file.ttl
file.text text/turtle -> file.txt$.ttl

We could also implement a server slug creation with POST when resource exists.

Refactor to separate out utility functions

@bourgeoa wrote in another issue

a good idea to create a restUtils.js or a utils folder with some functions like _getParent() isLink() getExtension()

Yes, sounds like a good refactor eventually,

allow POST to create filenames if resource exists

POST on an existing resource or container should create an alternate filename (e.g. prepend a number) and save another version rather than overwriting.

Missing built files in npm releases, incorrect specification in package.json

The built files corresponding to module and browser properties are not being created by upon a npm release. I cannot identify why this is the case though, given build scripts are defined.

Also, per my understanding a module release is unnecessary given the code is still commonJS. It is also confusing for the bundler which given in solid-file-client module points to the unbuilt source (correctly, since that is esm); here it is main that points to unbuilt source and module points to a lib version which is missing but if it existed would be redundant.

broken link to rdflib in README

README.md contains this:

Solid-Rest is included automatically in [rdflib]() when a script is run outside a browser.

...but there is no definition for [rdflib] elsewhere in the file, so the link is broken.

Specification Targets

My summary of how Solid-Rest should behave to be spec-compliant (other than requirements for http, CORS, and WAC) :

https://solid.github.io/specification/#read-write (2020-10-29)

Updated 2020-12-15

READING 2.4.2

unsupported HTTP methods should return 405
trailing slash always denotes a container
can not have both "/path/foo/" and "/path/foo"
GET, HEAD, and OPTIONS MUST be supported
response headers for Accept-Patch & Accept-Post MUST list supported media
types in GET and HEAD requests and MAY list them in OPTIONS requests
MUST send 400 on PUT, POST, or PATCH that lacks content-type header (2.1.2)

WRITING 2.4.3

intermediate containers MUST be created for PUT and PATCH
a Link header indicating something is a container MUST be in the response
to requests to container URLs (those ending in slash)
POST should return 404 if the parent folder of the resource doesn't pre-exist
PUT and PATCH to an aux resource MUST create or update it
POST to an aux resource MUST return 403 and a body describing the error if
the request has a slug
requests to update a container's containment triples MUST faile with 409
MAY send If-None-Match header of "*" and support ETag headers
only those with Control on a resource may access its ACL resource

DELETING 2.4.4

send 405 on attempt to delete a storage's root container
do not send DELETE in Allow header for root container
when deleting a resource, its aux resources MUST be deleted
attempt to delete non-empty container MUST send 409 and show error in body
Write access is required to delete a resource (other than ACL)
Control access on a resource is required to deleta its ACL

REPRESENTATIONS 2.4.5

a resource's content-type MUST be determined from the header used in its creation
servers MUST honor a request's Accept header if it specifies "text/turtle" or "application/ld+json"

CROSS-ORIGN RESOURCE SHARING 2.8

MUST not block requests due to CORS or different origins

LDP Spec https://www.w3.org/TR/ldp/

Link http://www.w3.org/ns/ldp#Resource must be returned in all headers
should respond with text/turtle if no Aceept header is present

Error: Cannot find module '/home/jeff/solid/solid-rest/core/'

I think the most recent version of solid-rest had a build problem committed to NPM. Running the code gives me this error:

Error: Cannot find module '/home/jeff/solid/solid-rest/core/'
Require stack:
- /Users/jackson/O/clone-to-solid-pod/node_modules/@solid-rest/file/dist/cjs/index.js
- /Users/jackson/O/clone-to-solid-pod/node_modules/solid-node-client/dist/cjs/index.js
- /Users/jackson/O/clone-to-solid-pod/lib/index.ts

The file at /node_modules/@solid-rest/file/dist/cjs/index.js has this line:

var _core = _interopRequireDefault(require("/home/jeff/solid/solid-rest/core/"));

UUID fails (on some versions of node and on windows)

After I did npm install, I got this when trying to run npm build

[email protected] test /var/www/html/solid/solid-rest
node tests/all

/var/www/html/solid/solid-rest/node_modules/uuid/dist/esm-browser/index.js:1
export { default as v1 } from './v1.js';
^^^^^^

SyntaxError: Unexpected token 'export'
at wrapSafe (internal/modules/cjs/loader.js:1060:16)
at Module._compile (internal/modules/cjs/loader.js:1108:27)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1164:10)
at Module.load (internal/modules/cjs/loader.js:993:32)
at Function.Module._load (internal/modules/cjs/loader.js:892:14)
at Module.require (internal/modules/cjs/loader.js:1033:19)
at require (internal/modules/cjs/helpers.js:72:18)
at Object. (/var/www/html/solid/solid-rest/src/rest.js:8:14)
at Module._compile (internal/modules/cjs/loader.js:1144:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1164:10)
npm ERR! Test failed. See above for more details.

Solid-Rest-2.0.0

@bourgeoa , @Otto-AA , @CxRes, - I have followed some of each of your advices and completely rewritten solid-rest from the ground up. I need to do another re-factor in the future to make it typescript but this version moves from cjs to esm, modulerizes everything, gets rid of the horrid param passing of options, and other improvements. The new version is in the repo-cleanup branch.

Some of the big changes : it no longer handles multiple plugins - each solid-rest instance handles only one plugin and one kind of fetch, leaving the protocol switching to libraries above it like solid-node-client.

The logic is vastly improved. It first checks the request and then, if the request doesn't have errors, it checks the requested item and only if that doesn't have errors does it actually try to carry out the request. This means that we can apply most of the logic at the top rather than inside each action. So a request with a trailing slash on a slug can be rejected before even looking at the requested item. Things like attempt to patch non-RDF can be rejected by looking at the requestedItem without even touching the patch method. This makes the logic very clear (see handleRequest.js) which, AFAIK, implements close to all of the Solid spec logic in under 70 lines.

The other big change is I want users to import the plugin, not solid-rest. So new SolidRestFile() rather than new Rest( new SolidRestFile() ). This means that a SolidRest plugin that doesn't use the file-system doesn't need to import 'fs' and will make things cleaner for downstream libraries and apps.

I have not yet re-factored Mem (what used to be called localStorage) or BrowserFS. I have a new plugin due out this week Solid-Rest-Ssh (yes, you can ssh to any bash-like shell from solid-rest and treat it as a serverless pod). My intention is to keep the plugins in the same github repo (in a plugins folder) but have each of them be its own npm package. I am a bit confused if this would work in practice - wouldn't it mean that the user would have to install a solid-rest in each plugin's own node-modules space? Or maybe import solid-rest into the pugin with import(../core) or something. Thoughts welcomed.

In solid-rest-file, I've made an attempt at JSDoc but if anyone of you have some tips about it, let me know.

The only tests not passing are two text/n3 patch tests. SPARQL patch works great.

Handle JSON-LD

We need to

check that the correct content type is associated with json-ld files
handle the Accept header in requests
- send turtle/json depending on header
- reject requests for wrong type
make sure response.json() works as expected

Handle URL=valid Filename=notVallid

See : SolidOS/solid-panes#262

solid-rest add pim:Storage

see nodeSolidServer/node-solid-server#1629 and nodeSolidServer/node-solid-server#1632

Implement Dropbox as a Solid-accessible storage

@Otto-AA and I both thought of Dropbox as a natural fit for Solid-Rest. All that needs to be done is implement a thin layer similar to solid-rest/file or solid-rest/localStorage and coordinate with Dropbox's native access control mechanisms. Anyone that wants to give this a try, please get in touch. I'd be glad to help but can't fully take this on at the moment.

Refactor to have a RequestHanlder Class

@Otto-AA suggested the refactoring below in a comment on issue #24

I think this is a good idea and would welcome a PR later. Right now, I want to get a good stable release out. I will comment back here when I'm ready for a PR and if you're able to do it then, great, if not, oh well.

Otto wrote :
In general to make the code cleaner I'd suggest to split SolidRest into the two classes SolidRest and RequestHandler. The RequestHandler constructor could take the url, storage and options as parameter. The benefit would be, that you could assign the options to this from the RequestHandler and then access it from the methods without passing it as parameter. That would make it easier to split the fetch method it into smaller functions.

So it would look something like this:

class SolidRest {
constructor( handlers ) {
this.storageHandlers = {}
if( typeof handlers ==="undefined") return
handlers.forEach( handler => {
console.log(Installing Rest Handler ${handler.name} using prefix ${handler.prefix})
this.storageHandlers[handler.prefix] = handler
})
}

getStorage(uri){
const { protocol } = new URL(uri)
const prefix = protocol // TODO
if(!this.storageHandlers[prefix]) throw "Did not recognize prefix "+prefix
return this.storageHandlers[prefix]
}

async fetch(uri, options = {}) {
const storage = this.getStorage(uri)
const requestHandler = new RequestHandler(uri, storage, options)
return requestHandler.fetch()
}
}

class RequestHandler {
constructor(uri, storage, options) {
this.uri = uri
this.storage = storage
this.options = options
}

async fetch() {
switch(this.options.method) {
case 'GET':
return this.get()
case 'POST':
return this.post()
// ...
}
// ...
}
If you want a refactor like this I could make a PR for it. It's not necessary, but it should make the code easier to maintain.

server errors on PUT

L342 seems to be the source issue

solid-rest/file/src/index.js

Lines 340 to 343 in 49dadfd

 let s = new Readable(); 

 s.push(text); 

 s.push(null); 

 return s;

In javascript null is of type object

node:events:495
      throw er; // Unhandled 'error' event
      ^

TypeError [ERR_INVALID_ARG_TYPE]: The "chunk" argument must be of type string or an instance of Buffer or Uint8Array. Received an instance of Object
    at new NodeError (node:internal/errors:405:5)
    at readableAddChunk (node:internal/streams/readable:310:13)
    at Readable.push (node:internal/streams/readable:278:10)
    at SolidRestFile._makeStream (/mnt/d/github/bourgeoa/[email protected]/server/node_modules/@solid-rest/file/dist/cjs/index.js:371:7)
    at /mnt/d/github/bourgeoa/[email protected]/server/node_modules/@solid-rest/file/dist/cjs/index.js:258:24
    at new Promise (<anonymous>)
    at SolidRestFile.putResource (/mnt/d/github/bourgeoa/[email protected]/server/node_modules/@solid-rest/file/dist/cjs/index.js:207:12)
    at SolidRest.perform (/mnt/d/github/bourgeoa/[email protected]/server/node_modules/@solid-rest/core/dist/cjs/performRequestedMethod.js:120:34)
    at SolidRest.handleRequest (/mnt/d/github/bourgeoa/[email protected]/server/node_modules/@solid-rest/core/dist/cjs/handleRequest.js:86:29)
    at async SolidRest.fetch (/mnt/d/github/bourgeoa/[email protected]/server/node_modules/@solid-rest/core/dist/cjs/index.js:54:20)
Emitted 'error' event on Readable instance at:
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ERR_INVALID_ARG_TYPE'
}

Node.js v18.19.0

disallow creation of links with POST

POST MUST not be used to create linked resources and should return 403

When a PUTorPATCHmethod request targets an auxiliary resource, the server MUST create or update it. When aPOSTmethod request with theSlugheader targets an auxiliary resource, the server MUST respond with the403 status code and response body describing the error. [[Source](https://github.com/solid/specification/issues/42#issuecomment-616688848)]

accept-PATCH header

linkeddata/rdflib.js#473