Comments (10)
The Tencent docs say the ETag is supported (see https://main.qcloudimg.com/raw/document/intl/product/pdf/436_14102_en.pdf), so I'd have expected it to work.
If you look through the DN logs, do you see a line like: "s3Client.get_key_stats, expected to find key: ETag"?
from hsds.
The Tencent docs say the ETag is supported (see https://main.qcloudimg.com/raw/document/intl/product/pdf/436_14102_en.pdf), so I'd have expected it to work.
If you look through the DN logs, do you see a line like: "s3Client.get_key_stats, expected to find key: ETag"?
I checked the DN log, but I didn't see anything related to ETag.
from hsds.
Do you see any DN lines like: INFO> head: ...?
The "..." should be a JSON including the ETag.
from hsds.
Do you see any DN lines like: INFO> head: ...? The "..." should be a JSON including the ETag.
@jreadey No, I don't see any INFO> head: ...
lines, which is weird.
from hsds.
I might have not correctly understood the issue... HSDS gets the ETags for each object as part of the "scanRoot" operation, but this is not returned in the GET /datasets response.
If you do hsstat <domain>
do you see an MD5 line?
from hsds.
I might have not correctly understood the issue... HSDS gets the ETags for each object as part of the "scanRoot" operation, but this is not returned in the GET /datasets response. If you do
hsstat <domain>
do you see an MD5 line?
@jreadey Yes, I have got MD5 line when I do hsstat <domain>
.
So doesn't return an Etag in the GET /datasets response, right?
But I can see from the example in this document that there is an Etag in the response information:
https://github.com/HDFGroup/hdf-rest-api/blob/master/DatasetOps/GET_Value.md
from hsds.
Ok, I see that. Not sure why the Etag key disappeared from the response header. Possibly this was the default in an earlier version of the aiohttp package, but not with the version we are using now.
It shouldn't be hard to restore the etag response, but let me ask about the semantics you are expecting...
With GET /datasets/datasetid/value, do you want the etag to represent the state of all the dataset values, or just the selection specified in the request?
For GET /datasets/datasetid, it seems clear that the etag should represent the dataset values combined with any metadata (e.g. attributes).
Are you thinking to use a HEAD request first to determine if anything has changed, and only do a GET when the values have changed?
from hsds.
Ok, I see that. Not sure why the Etag key disappeared from the response header. Possibly this was the default in an earlier version of the aiohttp package, but not with the version we are using now.
It shouldn't be hard to restore the etag response, but let me ask about the semantics you are expecting...
With GET /datasets/datasetid/value, do you want the etag to represent the state of all the dataset values, or just the selection specified in the request?
For GET /datasets/datasetid, it seems clear that the etag should represent the dataset values combined with any metadata (e.g. attributes).
Are you thinking to use a HEAD request first to determine if anything has changed, and only do a GET when the values have changed?
@jreadey Thank you very much for your reply.
Yes, what you said is exactly what I want, I want thinking to use a HEAD request first to determine if anything has changed, and only do a GET when the values have changed.
With GET /datasets/datasetid/value, I want the etag to represent the status of the selection specified in the request.
Thank you again!
from hsds.
Yes, that sounds useful. I created a feature issue for this here: #268.
In the meantime, you might find it useful to do a GET on the dataset and check the modification time. If that hasn't changed, you know the data will not have been updated. If it has, the data in your selection may or may not have been changed, so you'd need to do a GET value to verify.
from hsds.
Yes, that sounds useful. I created a feature issue for this here: #268. In the meantime, you might find it useful to do a GET on the dataset and check the modification time. If that hasn't changed, you know the data will not have been updated. If it has, the data in your selection may or may not have been changed, so you'd need to do a GET value to verify.
That's good idea! I will try to do a GET on the dataset and check the modification time.
from hsds.
Related Issues (20)
- Support FLETCHER32 filter
- Compound Type Subsetting HOT 4
- Support N-Bit and scale offset filters
- POST_Links does not return link information when following links recursively HOT 5
- Docker Image CI fails intermittently due to bad username/password
- Uninitialized variable length sequences are returned as scalars instead of empty arrays HOT 2
- Uploading 30 GB file takes too long to upload HOT 7
- not enough room in chunk cache - return 503 HOT 21
- Windows encoding error when writing sequence of variable length UTF-8 characters HOT 2
- Scripts use deprecated docker-compose v1
- Vlen sequence of variable-length UTF-8 strings cannot be written HOT 7
- Creation timestamps lack resolution on Windows HOT 2
- MAX_WAIT_TIME for rescan should be a config option HOT 1
- 409 Conflict during testCreateDomainNodeIds
- Example Requests and Authentication HOT 4
- typo, should be password_file HOT 2
- Improve resiliency for concurrent requests HOT 2
- Add support for numpy >= 2.x.x HOT 4
- HS_USERNAME assigned admin at startup HOT 2
- external connection to hsds with http HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hsds.