Comments (8)
CrCati
commented on July 20, 2024
Hi, @dipol0; as to read the mounted file, the error may caused by https://github.com/s3fs-fuse/s3fs-fuse/issues/1815, you can upgrade the s3fs to 1.88 or 1.93 and try again. BTW, the BTFS S3-compatible API only supports [AWS v4 signatures (AWS4-HMAC-SHA256)] for authentication and does not support AWS v2 signatures at this time. So, the command should be sudo s3fs iso-images /media/btfs-s3 -o passwd_file=~/passwd-s3fs,nonempty -o use_path_request_style -o sigv4 -o host=http://192.168.0.50:6001/
from go-btfs.
dipol0
commented on July 20, 2024
@CrCati , tnx for advise. may be i try it ))
PS + when a ise native aws-cli (aws) for upload file to BTFS-S3 - all fine
aws --endpoint=aws --endpoint=http://192.168.0.50:6001 --profile=BTFS-S3 s3 cp /home/manager/Yandex.deb s3://iso-images/Yandex.deb`
but then try download from get fail:
aws --endpoint=aws --endpoint=http://192.168.0.50:6001 --profile=BTFS-S3 s3 cp s3://iso-images/Yandex.deb /home/manager/Yandex.deb
download failed: s3://iso-images/Yandex.deb to ../home/manager/Yandex.deb An error occurred (NotImplemented) when calling the GetObject operation: A header you provided implies functionality that is not implemented
in btfs daeon log see:
s3-api: | 2023-10-02T11:41:34+03:00 | <HEAD> | /iso-images/Yandex.deb | HeadObjectHandler | &{UserId:a8ba917c-5fa1-43e6-8b84-b46a5c874c55 Bucket:iso-images Object:Yandex.deb WithBody:false} | <nil> | 425.755µs
s3-api: | 2023-10-02T11:41:34+03:00 | <GET > | /iso-images/Yandex.deb | GetObjectHandler | <nil> | param Range is unsported | 302.344µs
s3-api: | 2023-10-02T11:41:34+03:00 | <GET > | /iso-images/Yandex.deb | GetObjectHandler | <nil> | param Range is unsported | 197.373µs
s3-api: | 2023-10-02T11:41:34+03:00 | <GET > | /iso-images/Yandex.deb | GetObjectHandler | <nil> | param Range is unsported | 243.119µs
aws-cli/2.13.22 Python/3.11.5 Linux/6.2.0-33-generic exe/x86_64.ubuntu.22 prompt/off
from go-btfs.
CrCati
commented on July 20, 2024
Hi, @dipol0; I can't reproduce it. I used the aws-cli to upload and download different files, and all success:
aws-cli/1.29.57 Python/3.11.5 Darwin/22.5.0 botocore/1.31.57
BTW, I install aws-cli with this cmd python -m pip install awscli
from go-btfs.
dipol0
commented on July 20, 2024
@CrCati , you can show/share you BTFS node config file? Without critical info ;)
- and how OS you use and BTFS from (precompiled/self-compiled) ???
from go-btfs.
CrCati
commented on July 20, 2024
OK, the config file is:
My node is using pre-compiled version: https://github.com/bittorrent/go-btfs/releases/download/btfs-v2.3.3/btfs-linux-amd64
from go-btfs.
dipol0
commented on July 20, 2024
@CrCati , tnx. Hmmm...very strange. I have same version pre-compiled 2.3.3-linux-amd64.
but
aws-cli/2.13.22 Python/3.11.5 Linux/6.2.0-33-generic exe/x86_64.ubuntu.22 prompt/off
and upload worked, download - not. If use cli. WebUI up/down - work fine.
UPD
I found then standart "aws cp" can upload BIG size file to btfs s3 storage. And aws - btfs put it as multipart file. But then try "aws cp" from S3 to local place - it not woked by default.
With small files "aws cp" worked correct.
If need get BIG size file back from S3 to local i can like this (it worked for me):
manager@host:~/src$ aws --endpoint=http://192.168.0.50:6001 --profile=BTFS-S3 s3api get-object --bucket iso-images --key OPNsense-23.7-dvd-amd64.iso.bz2 /home/manager/OPNsense-23.7-dvd-amd64.iso.bz2
{
"AcceptRanges": "bytes",
"LastModified": "2023-10-02T21:06:46+00:00",
"ContentEncoding": "",
"ContentType": ", application/x-iso9660-image",
"Metadata": {
"Cid": "QmXH*******************evLU1Y"
}
}
I dont profi and dont know is issue awscli or btfs S3-API realization ))
Read AWS doc - Downloading an object - Using the AWS CLI
May be useful for somebody ))
@CrCati , if you can confirm my investigation ? Try upload big file ~200..300Mb to btfs and try download it back:
1 use "asw cp" (you get alerts about error downloading or not)
2. use "asw get-object"
Tnx.
UPD a little off topic))
Noob questions.
- There is a local BTFS node. I upload data/files to it via WebUI or awscli. In this case, these data/files are stored only locally or distributed in the BTFS nodes network?
- When deleting a file from storage, no reduction in space is displayed. Is this how it should be? Or what...?
from go-btfs.
CrCati
commented on July 20, 2024
Hi, @dipol0; I've tried that; the extra_args 'range' is not supported in this version. So, the 'aws cp' will get the 'A header you provided implies functionality that is not implemented' message. The feature to support more header formats is on the way. You can use 'aws get-object' for now.
As to 1. these data/files are stored only locally; 2. the space value should be updated; it will be fixed soon.
from go-btfs.
CrCati
commented on July 20, 2024
I'll close this GitHub issue now; if you have more information to share, please feel free to reopen it.
from go-btfs.
Related Issues (20)
- Node becomes stuck/stale? HOT 3
- [BUG] caught panic: runtime error: slice bounds out of range [:19] with capacity 1 HOT 8
- Peer Connections HOT 4
- [QUESTION] How to extend the storage duration of an uploaded file HOT 3
- [Feature] HOT 1
- [BUG] panic: runtime error: index out of range [1] with length 1 HOT 2
- Enable "API Endpoint" URL format with SSL and no port HOT 5
- [BUG] Can't build with GO v.1.21.1 HOT 7
- S3-API does not support metadata for PutObject operation HOT 3
- [BUG] CreateFile .btfs/api: The system cannot find the path specified - with custom config path HOT 1
- [BUG]Some contract status anomalies， status is DRAFT HOT 8
- go-btfs high RAM usege HOT 5
- caught panic HOT 1
- [BUG] HOT 2
- Installation error for BTFS beta version 2.3.4 HOT 3
- [Feature] actually have a s3 compatible api HOT 1
- [BUG] HOT 2
- [Feature] rclone s3 endpoint instruction documentation HOT 8
- [BUG] make: *** [check_go_version] Error 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-btfs.