espebra / filebin2 Goto Github PK

Not going too deep down the rabbit hole as yet. But I do see files in the AWS S3 bucket (yay!).

But I have this odd issue that every time I upload a file in the browser UI it also downloads a .gz file Not sure why ??

The save URL link on the home page also downloads a .gz file

A few errors in the log file as well for good measure.. Looks like some error in the template which I think is something else, I think the real issue I am seeing is the gorrilla/handlers but maybe they are linked..

Any how thought I would share todays findings/progress..

cconstab@cally:/filebin2$ go build -mod=vendor
cconstab@cally:/filebin2$ cp ../filebin2.old/env.sh ./
cconstab@cally:/filebin2$ source env.sh
cconstab@cally:/filebin2$ ./filebin2 -access-log ./logs --listen-host 0.0.0.0 -baseurl "http://192.168.1.149:8080"
Established session to S3AO at s3.us-west-1.amazonaws.com
Found S3AO bucket: filebin2-test
Starting Lurker process (interval: 5m0s)
Expiration: 168h0m0s
Starting HTTP server on 0.0.0.0:8080
Uploaded filename tent.jpg (582 kB) to bin gh5ha0767xby8095 (db in 0.008s, buffered in 0.016s, checksum in 0.005s, stored in 6.718s, total 6.746s)
Failed to execute template: template: :451:38: executing "bin" at <ne .CreatedAt .UpdatedAt>: error calling ne: invalid type for comparison
2021/02/16 01:00:43 http: superfluous response.WriteHeader call from github.com/gorilla/handlers.(*responseLogger).WriteHeader (handlers.go:65)

Hi @espebra,

I face this issue when uploading from Javascript and maybe you can help me:
The file boundaries (generated by browsers in standard FormData) are being copied into the file as you can see here:

You can reproduce it when using this Codepen example to upload files to a test Bin:
https://codepen.io/antoniandre/pen/GROWNMd?editors=0010

Would you have a way to strip this from the file on creation using Go? Or other ways to bypass?

My understanding is that when you receive the following header containing boundaries, you should extract only the content in between of these boundaries from the file sent via formData.

Just for what it is worth, I use the same frontend piece of code in a project that handles the backend file upload in PHP and it strips the boundaries by itself (I don't do anything for that). So the output like the original file, without the boundaries delimiters.

curl https://filebin.net
curl: (35) LibreSSL SSL_connect: SSL_ERROR_ZERO_RETURN in connection to filebin.net:443

Thx for fixing. Love your service!

Hello there,

I loved this project and I would like to contribute. What features can I make?

I wanted to use filebin to upload files to from my site and so send requests with jscript fetch, but I have attempted a couple of ways and I keep getting errors. I presume it might have to do with cors not being enabled? At least this is what I think, this curl command executes fine,
curl --data-raw "@test.txt" -H "filename: test" https://filebin.net/ -i -L
but when translated to a fetch command in javascript it seems to go wrong

await fetch("https://filebin.net", { method: "POST", body: "@test.txt", headers: { filename:"test" }});

@espebra ,

A few months ago I got Filebin2 on K8S up and working and a few months later @cpswan also contributed (Chris and I work at atsign.com). Just wanted to let you know that our application @mospherePro is now up and running and is also opensource.

https://github.com/atsign-foundation/atmosphere_pro

And it's also in the App/Play stores plus its coming soon to Mac/Windows.

@mospherePro uses our instance Filebin2 on k8s as ephemeral storage of encrypted files.

This app is not only a useful app but also a proof point for the opensource platform we have been building.

https://github.com/atsign-foundation

So why am I reaching out ? We are puling together a story/article that describes the @mospherePro journey and wondered if you would like/would mind to be included in that write up?

Thanks so much for Filebin2!

@cconstab

Hi!

It looks like the storage might be full. Is it possible to clear up some files or allocate more storage?

Thanks, Andrew.

Hello,

is there filebin Go binary that I can use to run it without Docker ? I would prefer to have this and all dependencies installed manually instead using Docker.

A.

Espen,

As you know we use filebin2 running on Kubernetes which means each instance is running in a container of its own and shares the DB and the S3 buckets.

Things have been working very well until this morning when someone reported they could not transfer a 300Mb+ file. On further inspection it looked like the container was being memory starved. So as a test of a hunch rather than anything else we upgraded the container to have a bit more memory and everything worked.

However, that got me testing even larger files and I noticed what I think is a pattern and wanted to see if you can confirm my theory.

I think when uploading files to filebin2 it holds those files in memory and then once received sends them on to S3 storage. Which is just fine on a chunky server with lots of memory and swap.

But a container is a little limited on infra .. For the moment each of our containers now has 4Gb of memory and handles a single 1GB file just fine (all/most) of the time.

Is this theory correct ? Is there anyway to stream straight through to S3 perhaps ? or if we need to cache locally use disk rather than memory ?

Looking forward to your reply..

I know I could look at the code :-) But I am not Go friendly yet !

Hi @espebra,

I'm pushing ahead with getting the local Docker environment up and running and documenting the process so that you can include it in the README.

I'd like to be able to MinIO on localhost:9000 as indicated in the current README. However, when I browse to that URL I immediately get redirected to another port (which is not exposed).

I don't know anything about MinIO, so I'm afraid that I can't fix this.

Thanks, Andrew.

Hi.

i run docker-compose up --build an error has occurred.

Creating network "filebin2_local" with the default driver
Pulling s3 (minio/minio:)...
latest: Pulling from minio/minio
8f403cb21126: Pull complete
65c0f2178ac8: Pull complete
6e32ce08526e: Pull complete
932fb72de569: Pull complete
71bfd33c61af: Pull complete
588b2addab38: Pull complete
093f7de724c9: Pull complete
Digest: sha256:fe69dcaed404faa1a36953513bf2fe2d5427071fa612487295eddb2b18cfe918
Status: Downloaded newer image for minio/minio:latest
Pulling db (postgres:)...
latest: Pulling from library/postgres
69692152171a: Pull complete
a31b993d5cc6: Pull complete
f65921886500: Pull complete
b9c1a94e4ca8: Pull complete
435dd99ceb68: Pull complete
d3ee8e88c67c: Pull complete
84b08674f942: Pull complete
7d358e850d3e: Pull complete
c7dcc5801f3b: Pull complete
f6eeca01c79c: Pull complete
392faa2e3ddd: Pull complete
3e77feaf6319: Pull complete
9b42e6c9c7ba: Pull complete
5fce2660d75c: Pull complete
Digest: sha256:117c3ea384ce21421541515edfb11f2997b2c853d4fdd58a455b77664c1adc20
Status: Downloaded newer image for postgres:latest
Building app
Sending build context to Docker daemon  26.94MB
Step 1/7 : FROM golang:buster
buster: Pulling from library/golang
d960726af2be: Pull complete 
e8d62473a22d: Pull complete 
8962bc0fad55: Pull complete 
65d943ee54c1: Pull complete 
f2253e6fbefa: Pull complete 
6d7fa7c7d5d3: Pull complete 
e2e442f7f89f: Pull complete 
Digest: sha256:ab32429d40c1b734ed4f036838ac516352182f9414563478fa88a1553c4a4414
Status: Downloaded newer image for golang:buster
 ---> 96129f3766cf
Step 2/7 : RUN export DEBIAN_FRONTEND=noninteractive && apt-get install -y make gcc libc-dev git
 ---> Running in 5ef270a72a52
Reading package lists...
Building dependency tree...
Reading state information...
gcc is already the newest version (4:8.3.0-1).
git is already the newest version (1:2.20.1-2+deb10u3).
libc6-dev is already the newest version (2.28-10).
make is already the newest version (4.2.1-1.2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Removing intermediate container 5ef270a72a52
 ---> 3be100a32645
Step 3/7 : RUN go get -u github.com/jstemmer/go-junit-report
 ---> Running in 69822ad38378
go: downloading github.com/jstemmer/go-junit-report v0.9.1
Removing intermediate container 69822ad38378
 ---> 6e44782a6e9d
Step 4/7 : RUN go get -u github.com/GeertJohan/go.rice/rice
 ---> Running in 1b7ba3b0da24
go: downloading github.com/GeertJohan/go.rice v1.0.2
go: downloading github.com/GeertJohan/go.incremental v1.0.0
go: downloading github.com/akavel/rsrc v0.8.0
go: downloading github.com/daaku/go.zipexe v1.0.0
go: downloading github.com/jessevdk/go-flags v1.4.0
go: downloading github.com/daaku/go.zipexe v1.0.1
go: downloading github.com/akavel/rsrc v0.10.2
go: downloading github.com/nkovacs/streamquote v1.0.0
go: downloading github.com/jessevdk/go-flags v1.5.0
go: downloading github.com/valyala/fasttemplate v1.0.1
go: downloading github.com/valyala/fasttemplate v1.2.1
go: downloading github.com/valyala/bytebufferpool v1.0.0
go: downloading golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4
go: downloading golang.org/x/sys v0.0.0-20210514084401-e8d321eab015
Removing intermediate container 1b7ba3b0da24
 ---> 4ed51b642e45
Step 5/7 : WORKDIR /app
 ---> Running in 486fc6babda5
Removing intermediate container 486fc6babda5
 ---> 7c36d13fa51d
Step 6/7 : EXPOSE 8080
 ---> Running in 26ce36fb0117
Removing intermediate container 26ce36fb0117
 ---> ee22974ba805
Step 7/7 : CMD make run
 ---> Running in 31d0d5d982a3
Removing intermediate container 31d0d5d982a3
 ---> e7c3ab4016e5
Successfully built e7c3ab4016e5
Successfully tagged filebin2_app:latest
Creating filebin2_s3_1 ... 
Creating filebin2_db_1 ... 
Creating filebin2_db_1 ... error

ERROR: for filebin2_db_1  Cannot start service db: driver failed programming external connectivity on endpoint filebin2_db_1 (fbdb954da035d23931a85cc36beac5b29f
Creating filebin2_s3_1 ... error
WARNING: Host is already in use by another container

ERROR: for filebin2_s3_1  Cannot start service s3: driver failed programming external connectivity on endpoint filebin2_s3_1 (bac11d8978a3957e88645ec2e1e34bba36b47ca77b14b0960cbebbb9df279c4e): Error starting userland proxy: listen tcp6 [::]:9000: socket: address family not supported by protocol

ERROR: for db  Cannot start service db: driver failed programming external connectivity on endpoint filebin2_db_1 (fbdb954da035d23931a85cc36beac5b29fa7908177ea8fe4f6225738f282b688): Error starting userland proxy: listen tcp6 [::]:5432: socket: address family not supported by protocol

ERROR: for s3  Cannot start service s3: driver failed programming external connectivity on endpoint filebin2_s3_1 (bac11d8978a3957e88645ec2e1e34bba36b47ca77b14b0960cbebbb9df279c4e): Error starting userland proxy: listen tcp6 [::]:9000: socket: address family not supported by protocol
ERROR: Encountered errors while bringing up the project.

How to solve this error? thanks

If I mess up a command line argument then the app very helpfully prints out usage instructions:

Usage of /usr/local/bin/filebin2:
app_1  |   -access-log string
app_1  |        Path for access.log output (default "/var/log/filebin/access.log")
app_1  |   -admin-password string
app_1  |        Admin password (default "changeme")
app_1  |   -admin-username string
app_1  |        Admin username (default "admin")
...

But the defaults printed out include information from correctly set environment variables, including secrets:

...
app_1  |   -s3-secret-key string
app_1  |        S3 secret key (default "Xt5F9x9UimLZNxtB4Zoo54TMTdjp85yZp3r2s59H")
...

Hello,

I noticed that filebin.net does not support non-English characters in file names. When attempting to upload a file with a file name containing non-English characters, the website replaces those characters with underscores. This is a problem for me, as I sometimes need to upload files with file names in different languages.

For example, I tried to upload a file named "使用手冊.pdf", which is a user manual in Traditional Chinese. However, the website replaced the non-English characters with underscores and renamed the file to "____.pdf".

It's important for me to be able to upload files with file names in different languages. Will support for non-English characters in file names be added in the future?

Thanks in advanced.

Thanks for the service at https://filebin.net/about
It works, however the basic feature to CANCEL once an upload has started is missing.

It was not clear to me if just closing the particular tab would stop the upload - but if it does, why not put some basic instructions or notice on the page to say so...

Works this only on docker or I can use it on apache web server also?

Hi! I’m not sure if this is the right place to report this issue, but I’m getting “You are blocked from using this service“ error whenever I open https://filebin.net in my web browser, independent of browser implementation (Safari/Firefox/Chrome), ISP (I’ve tried connecting from two different ISPs) and whether I enable Cloudflare WARP (or “VPN” in a layman’s terms). It’s possible to archive the front page (see https://archive.ph/QhYIR), so I guess it’s still accessible?

I’m not sure what’s the issue here since I don’t think I have abused the service (I’ve been only using it to share files between friends and occasionally as a workaround to transfer files from VM to host OS), and doesn’t look like a subnet ban to me too as I can’t access filebin with Cloudflare WARP.

If that would help with debugging, I’m currently leasing the following IP addresses from my primary ISP via DHCP:

• IPv4 95.84.246.62
• IPv6 subnet 2a02:2168:8fec:f600::/56
  Both are dynamically assigned, but are reasonably static given no power outages.

When I download apk its save as ZIP file

Hello,

I try to run docker-compose.yml on my synology. All it's ok on local.
With my domain name:
-App is ok
-Upload is ok
-Download is KO. Link is : https://storage:9000...
Someone can help me to solve this point :-)

Thank you

Hi @espebra, thanks for this great library!

I am trying to upload a file programatically via javascript, and get a CORS issue, just like #15.

Usually to allow cross-domain requests, the server must have this header set: Access-Control-Allow-Origin: *.
In php that would just be header('Access-Control-Allow-Origin: *'); for instance.

And as per this error, it states that no 'Access-Control-Allow-Origin' is present on the server.

Here is a Codepen you can play with after changing the 'Access-Control-Allow-Origin' header on the server side.
You should see 2 calls in the network panel: 1 for preflight "handshake" and one for the actual submission. But at the moment they fail because of CORS policy.

In my Codepen I also do a request to https://api.ipify.org which is another public API which has no CORS issue, for comparison.
In the network panel it shows that the server allows CORS from Codepen:

Here is the Codepen https://codepen.io/antoniandre/pen/GROWNMd?editors=0010, and the code (using Vue, but I've put comments if you're not familiar).

new Vue({
  waveui: new WaveUI(),
  data: () => ({
    files: [],
    loading: false
  }),

  methods: {
    // This function is called on send button press, and uploads to filebin.
    async onFormSuccess () {
      this.loading = true
      const formData = new FormData()

      formData.append('file', this.files[0].file)

      axios.post(
        `https://filebin.net/test-20220210/${this.files[0].name}`,
        formData,
        {
          headers: {
            'Access-Control-Allow-Origin': '*',
            'Content-Type': 'multipart/form-data',
            filename: this.files[0].name,
            bin: 'test-20220210'
          }
        }
      ).then(error => {
        this.loading = false
        console.log('ok')
      })
        .catch(error => {
        this.loading = false
        console.log(error)
      })
    }
  },
  
  mounted () {
    // This GET is called directly when the page is loaded.
    // Even a GET request returns this CORS error.
    axios.get(
      `https://filebin.net/test-20220210`,
      { headers: { 'Access-Control-Allow-Origin': '*' } }
    ).then(
      () => console.log('Success!'),
      () => console.log('Failure!')
    )

    // This is the same kind of GET request on another public API, which allows CORS. => no error.
    axios.get('https://api.ipify.org').then(data => {
      console.log(data)
    })
  }
}).$mount('#app')

I hope it makes sense.
I am really looking forward to be able to use this from Javascript! :)

Hello! I'm trying to deploy filebin via docker-compose but i cannot expose port 9000 for it's already occupied on my system.
I've managed to figure that out with minio but no matter what I change in the dociker-compose.yaml the app always goes for storage:9000 for whatever reason even though the -S3_ENDPOINT is set to storage:9010. How do i fix that?

Along with my @atsign-company colleague @cconstab I've been standing up a filebin2 instance on Google Cloud, using Google Kubernetes Engine (GKE), Cloud SQL, and Google Cloud Storage (GCS). It was a pleasant surprise to find that filebin2 just worked with (GCS).

But, after the recent change to use presigned URLS, things no longer work with GCS. Here's what I see when I click on a link to download a file:

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET 1624537054 /test-filebin2/eb318d739823cf741ecb8aad8f27266f916ba0561198a949c45ab5f3413fe51d/e27bf9b5dafdddc15b68ba198cc4be3c8c476461f6e2dfada5f976c991ab4388</StringToSign>
</Error>

I was able to setup the filebin app using my own docker instance. I am however running into a bit of an issue:
the filebin instance "believes" it is hosted at filebin.net (for QR codes for example).

Is there some way to set the app's hosting address (ideally as an environment variable in the docker-compose.yml)?

i want to increase time duration becasue it give only 60 min i want to increase