ivoa-std / vospace

Virtual Observatory User Storage

VOSpace

VOSpace

What is it?

VOSpace

VOSpace is the IVOAinterface to distributed storage. It specifies how VO agents and applications can use network attached data stores to persist and exchange data in a standard way.

A VOSpace web service is an access point for a distributed storage network. Through this access point, a client can:

1. add or delete data objects in a tree data structure;

2. manipulate metadata for the data objects

3. obtain URIs through which the content of the data objects can be accessed

VOSpace does not define how the data is stored or transferred, only the control messages to gain access. Thus, the VOSpace interface can readily be added to an existing storage system.

When we speak of "a VOSpace", we mean the arrangement of data accessible through one particular VOSpace service.

Each data object within a VOSpace service is represented as a node and has a description called a representation. A useful analogy to have in mind when reading this document is that a node is equivalent to a file or a directory.

Status?

The last stable version is REC-2.1.

See also the section Releases of this GitHub Repository.

What about this repository?

This GitHub repository contains the sources of the IVOA document describing VOSpace.

Only the LaTeX version is available here. No output version (e.g. PDF, HTML, DOC) should be stored in this repository.

Want to contribute?

1. Raise a GitHub Issue on this repository

2. Fork this repository (eventually clone it on your machine if you want to)

3. Create a branch in your forked repository ; this branch should be named after the issue(s) to fix (for instance: issue-7-add-license)

4. Commit suggested changes inside this branch

5. Create a Pull Request on the official repository (note: a git push is needed first, if you are working on a clone)

6. Wait for someone to review your Pull Request and accept it

This process has been described and demonstrated during the IVOA Interoperability Meeting of Oct. 2019 in Groningen ; see slides)

License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

vospace's Issues

Clarify getNode pagination

In the current text, the description of how pagination works is buried in the Response sub-section of the getNode method. Although technically correct, this makes it easy to miss.

The description of how pagination works should be in a separate sub-section, clearly titled Pagination or Paged response.

Check for view properties (MD5 and length) applied to node body

In the example for setNode, section 6.3.2, the XML response shows length and MD5 properties applied to the main body of the node.

<vos:node uri="vos://cadc.nrc.ca!vospace/mydata1" xsi:type="vos:DataNode">
  <vos:properties>
    ....
    <vos:property uri="ivo://ivoa.net/vospace/core#length" readOnly="true">44</vos:property>
    <vos:property uri="ivo://ivoa.net/vospace/core#MD5"    readOnly="true">12c13e49f2fd37e2c91bcca6acb78a08</vos:property>
    ....
  </vos:properties>
  <vos:accepts>
    ....
  </vos:accepts>
  <vos:provides>
    ....
  </vos:provides>
</vos:node>

As dataNodes can have different representations (views) for the same content, e.g. binary file, csv file, zip file etc. the length and MD5 properties only make sense when applied to a specific view rather than the node itself.

<vos:node uri="vos://cadc.nrc.ca!vospace/mydata1" xsi:type="vos:DataNode">
  <vos:properties>
    ....
  </vos:properties>
  <vos:accepts>
    ....
  </vos:accepts>
  <vos:provides />
    <vos:view uri="ivo://ivoa.net/vospace/core#binaryview">
        <vos:property uri="ivo://ivoa.net/vospace/core#length" readOnly="true">44</vos:property>
        <vos:property uri="ivo://ivoa.net/vospace/core#MD5"    readOnly="true">12c13e49f2fd37e2c91bcca6acb78a08</vos:property>
    </vos:view>
  </vos:provides>
</vos:node>

Clarify response for getNode pagination

If clients appy strict schema rules, this change will require a major version step

In the current system there is no way to indicate if a server has applied a pagination limit to the list of child nodes in a getNode response.

This means that the following XML responses could be interpreted in two different ways:

<?xml version="1.0" encoding="UTF-8"?>
<vos:node uri="vos://example.com!vospace/mydir" xsi:type="vos:ContainerNode">
  ....
  <vos:nodes>
    <vos:node uri="vos://example.com!vospace/mydir/file-001" xsi:type="vos:DataNode"/>
    <vos:node uri="vos://example.com!vospace/mydir/file-002" xsi:type="vos:DataNode"/>
  </vos:nodes>
</vos:node>

<?xml version="1.0" encoding="UTF-8"?>
<vos:node uri="vos://example.com!vospace/mydir" xsi:type="vos:ContainerNode">
  ....
  <vos:nodes>
    <vos:node uri="vos://example.com!vospace/mydir/file-003" xsi:type="vos:DataNode"/>
    <vos:node uri="vos://example.com!vospace/mydir/file-004" xsi:type="vos:DataNode"/>
  </vos:nodes>
</vos:node>

These could be two separate getNode responses, separated by time, during which file-001 and file-002 were deleted and replaced by file-003 and file-004. Or, these could be two consecutive responses to a single getNode request with the pagination limit set to 2.

To resolve this we propose adding the start uri, limit count and a more flag as attributes to the <nodes> element of the getNode response.

<?xml version="1.0" encoding="UTF-8"?>
<vos:node uri="vos://example.com!vospace/mydir" xsi:type="vos:ContainerNode">
  ....
  <vos:nodes start="" limit="2" more="true">
    <vos:node uri="vos://example.com!vospace/mydir/file-001" xsi:type="vos:DataNode"/>
    <vos:node uri="vos://example.com!vospace/mydir/file-002" xsi:type="vos:DataNode"/>
  </vos:nodes>
</vos:node>

and

<?xml version="1.0" encoding="UTF-8"?>
<vos:node uri="vos://example.com!vospace/mydir" xsi:type="vos:ContainerNode">
  ....
  <vos:nodes start="file-003" limit="2" more="false">
    <vos:node uri="vos://example.com!vospace/mydir/file-003" xsi:type="vos:DataNode"/>
    <vos:node uri="vos://example.com!vospace/mydir/file-004" xsi:type="vos:DataNode"/>
  </vos:nodes>
</vos:node>

This makes it clear that the list in the first response is limited 2 children, starting from the first child and that there are more children in a subsequent page; and that the second response is limited 2 children, starting from file-003, and that the list is complete and there are no more children in the container.

These extra attributes help to make it clear that the following response starts from the first child, contains all the children, has not been limited and there are no more children after this:

<?xml version="1.0" encoding="UTF-8"?>
<vos:node uri="vos://example.com!vospace/mydir" xsi:type="vos:ContainerNode">
  ....
  <vos:nodes start="" limit="" more="false">
    <vos:node uri="vos://example.com!vospace/mydir/file-001" xsi:type="vos:DataNode"/>
    <vos:node uri="vos://example.com!vospace/mydir/file-002" xsi:type="vos:DataNode"/>
    <vos:node uri="vos://example.com!vospace/mydir/file-003" xsi:type="vos:DataNode"/>
    <vos:node uri="vos://example.com!vospace/mydir/file-004" xsi:type="vos:DataNode"/>
  </vos:nodes>
</vos:node>

Change SHALL to MAY for standard properties

The introductory text for section 3.2.4 Standard properties uses the word SHOULD to indicate that the use of these properties is recommended but not mandatory.

The following URIs SHOULD be used to represent the service properties:

However the text for the individual properties used the word SHALL, which suggests that the use of these properties is a mandatory requirement of the specification.

ivo://ivoa.net/vospace/core#title SHALL be used as the property URI denoting a name given to the resource

We should reserve the mandatory keywords MUST and SHALL for the parts of the specification that are essential for the protocol to function. The use of these common properties aids interoperability, but services and clients should be free to use different identifiers for these concepts if they need to.

See Key words for use in RFCs to Indicate Requirement Levels RFC 2119

Transfer from volute

The transfer from volute needs to be completed.

Clarify description for getNode detail

The current description for the getNode detail parameter needs clarification.

The text for the detail parameter suggests that it controls the level of detail for the requested node itself rather than any child nodes.

detail with values of:
- min: the returned record for the node contains minimum detail with all optional parts removed – the node type should be returned. e.g.,
- max : the returned record for the node contains the maximum detail, including any xsi:type specific extensions
- properties: the returned record for the node contains the basic node element with a list of properties but no xsi:type specific extensions

As is, this text seems to suggest that the following request:

http://server.example.com/vospace/nodes/mydir?detail=min

Would simply return the minimal response for that node.

<vos:node uri="vos://example.com!vospace/mydir" xsi:type="vos:ContainerNode">

On the other hand, the example given in the text returns the full content for the container node, including properties, accepts, provides and capabilities, and only applies the detail filter to the child nodes.

<vos:node xmlns:vos="...." uri="vos://example.com!vospace/mydir" xsi:type="vos:ContainerNode">
<vos:properties>
  <vos:property uri="ivo://ivoa.net/vospace/core#description">My award winning images</property>
</vos:properties>
<vos:accepts>
  <vos:view uri="ivo://ivoa.net/vospace/core#anyview"/>
</vos:accepts>
<vos:provides>
  <vos:view uri="ivo://ivoa/net/vospace/core#defaultview"/>
</vos:provides>
<vos:capabilities/>
<vos:nodes>
  <vos:node uri="vos://example.com!vospace/mydir/file3401" xsi:type="vos:DataNode"/>
  <vos:node uri="vos://example.com!vospace/mydir/file3406" xsi:type="vos:DataNode"/>
  <vos:node uri="vos://example.com!vospace/mydir/file3532" xsi:type="vos:DataNode"/>
</vos:nodes>

My guess is that the detail parameter only applies to the child nodes of a container node. In which case the text needs to specify this.

detail with values of:
- min: the returned record for the a container node contains minimum detail for any child nodes, with all optional parts removed – *only the child node type should be returned. e.g., <Node uri="vos://service/name" xsi:type="Node"/>
- max : the returned record for the a container node contains the maximum detail for any child nodes, including any xsi:type specific extensions
- properties: the returned record for the a container node contains the basic node element for any child nodes, with a list of properties but no xsi:type specific extensions

Clarify description for getNode pagination

Based on discussion in the VOSpace mailing list 2021-June/003092.html.

The description for pagination of the getNode response is not particularly clear and a number of people have reported missing or skipping over it [1].

In addition, while the description or the Request parameters lists uri and limit, the description in the Response section refers to uri and offset.

The proposal is to move the description of how pagination works into a specific sub-section just for this.

Add a standard property for immutable data

In the standard properties section it would be useful to add something like the following:

• ivo://ivoa.net/vospace/core#immutable: SHALL be used as the property URI denoting a node that can't be deleted or moved but whose metadata can be edited

Probably in this case there are some properties that should be not modifiable too. I think that groupread and groupwrite properties should be editable to be able to share an immutable node, however length property shouldn't be editable and maybe also mtime (modification time).

Clarify retrieval of supported protocols for a node

The getNode section also describes how to retrieve supported protocols for a node:

The list of supported protocols for a node can be retrieved with:

• A HTTP POST of an empty protocol representation to the URL of an existing node http://rest-endpoint/nodes/path

I think that, as happened for the paginated request, this piece of text could be easily missed.

Moreover it is not very clear what should be the content of the request and the response. I think that moving this to a dedicated subsection and adding a complete example should be preferable.

I also think that using POST method for retrieving data doesn't conform with RESTful semantic.

getNode detail=min example incorrect

In section 6.3.1, the example of a getNode with parameter detail=min is incorrect. No elements should be within the element because of this parameter.

Also, the element is not terminated.

Throw an Invalid Argument fault if the pagination uri is not a direct child

The current specification does not say what happens if the path of the uri parameter in a getNode request points to a node that is not a direct child of the target node.

GET /vospace/nodes/mydir?limit=2&uri=vos://example.com!vospace/other/file-003

In this example the path of the uri vospace/other/file-003 does not point to a direct child of the target node /vospace/nodes/mydir and so would match the criteria for throwing an exception.

This is different to the case where the path of the uri is within the scope of the target node, but points to a child that does not exist.

GET /vospace/nodes/mydir?limit=2&uri=vos://example.com!vospace/mydir/missing-file

In this example the path of the uri vospace/mydir/missing-file does point to a location within the target node /vospace/nodes/mydir. However the specific child it points to missing-file does not exist. This matches the case where missing-file was listed in a previous page of a paginated request for /vospace/nodes/mydir, but has been deleted before the client sent the request for the next page.

In this situation, should the server respond with:

1. An empty response, because the page start no longer exists.
   • This is a misleading response, because there may be more children after the missing node
2. Throw a NodeNotFound fault because the start child cannot be found
   • This is a misleading response, because this implies the parent container can't be found.
3. Try to find the next available start location given the name of the missing node and the pagination order.
   • This is hard to do reliably in a consistent way across different implementations.

Add code of conduct and contributor agreement

Many open source projects are adding these to their projects. Should we do the same ?

• code of conduct
• contributor agreement

Or, put it another way - is there any reason why not ?

Replace or link standard URIs with Dublin core equivalents

Many of the properties listed in section 3.2.4 Standard properties have direct equivalents in the Dublin Core metadata vocabulary.

In fact, the description for many of the properties listed in 3.2.4 are word for word identical to the description of the equivalent terms in the Dublin Core metadata. In which case should we consider either adopting the DublinCore term itself, by replacing the VOSpace property URI with the URI for the Dublin Core term. Or, given such a strong correlation in their descriptions, we should at least reference the Dublin Core term in the description of the VOSpace property.

VOSpace property	DublinCore term
ivo://ivoa.net/vospace/core#title	http://purl.org/dc/terms/title
ivo://ivoa.net/vospace/core#subject	http://purl.org/dc/terms/subject
ivo://ivoa.net/vospace/core#creator	http://purl.org/dc/terms/creator
ivo://ivoa.net/vospace/core#description	http://purl.org/dc/terms/description
ivo://ivoa.net/vospace/core#publisher	http://purl.org/dc/terms/publisher
ivo://ivoa.net/vospace/core#contributor	http://purl.org/dc/terms/contributor
ivo://ivoa.net/vospace/core#date	http://purl.org/dc/terms/date
ivo://ivoa.net/vospace/core#type	http://purl.org/dc/terms/type
ivo://ivoa.net/vospace/core#format	http://purl.org/dc/terms/format
ivo://ivoa.net/vospace/core#identifier	http://purl.org/dc/terms/identifier
ivo://ivoa.net/vospace/core#source	http://purl.org/dc/terms/source
ivo://ivoa.net/vospace/core#language	http://purl.org/dc/terms/language
ivo://ivoa.net/vospace/core#relation	http://purl.org/dc/terms/relation
ivo://ivoa.net/vospace/core#coverage	http://purl.org/dc/terms/coverage
ivo://ivoa.net/vospace/core#rights	http://purl.org/dc/terms/rights

Clarification on MD5 property

In the setNode example (section 6.3.2) the response payload contains the following property URI: ivo://ivoa.net/vospace/core#MD5. It looks like a standard property, however it is not listed in the standard properties section.

Should we add it to the list? It seems that it is already used by CADC implementation. Also INAF implementation is going to use it.

Should we add other properties for other hashing algorithms that could be used to check file integrity (e.g. SHA256)?