spreadshirt / backstage-plugin-s3

An S3 Viewer plugin for Backstage! 🪣

S3 plugins for Backstage

This repository contains a set of Backstage plugins for interacting with AWS S3 buckets.

The plugin provides:

• A backend plugin that exposes a bunch of endpoints that can be used by the frontend and also by end users.
• A common package containing several types shared by frontend and backend.
• A frontend page where the users can navigate through the different buckets and their objects. As well as previewing, downloading or getting object information.

Installation

Check the installation process for these plugins in the following links:

• Frontend Plugin Installation
• Backend Plugin Installation

Deploying new releases ( internal )

• After every change made in the plugins, execute yarn changeset and follow the steps.
• When everything is ready, execute locally yarn release to update all the plugin versions, update the CHANGELOG.md files & update the yarn.lock.
• Finally, run yarn publish-release to publish the packages to npm.

Development

It's also possible to test the s3-viewer plugin in an isolated mode. For that, you'll need to execute the following 2 commands:

    # From the root of the repository
    docker-compose -f demo/docker-compose.yaml up
    
    # Wait until the bucket is created and the files synced, then:
    yarn start

After executing these two commands, a new window in your browser will be opened and you will be able to use the s3-viewer frontend and backend plugins altogether. You should see a platform with a name test and a bucket inside called foobar. This bucket will contain the data from ./demo/examples inside.

License

Copyright 2020-2022 © sprd.net AG. All rights reserved.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

backstage-plugin-s3's Issues

Folder structure, problem

Hi

I would like to have a little guidance: so I have to upload a folder structure to s3:

my local structure is:

test1/
└── test2
    └── test3
        └── test.txt

on backstage gui structure

├─ test1
│  ├─ test2
│  │  ├─ test3
│  │  ├─ test3
│  │  ├─ test3
│  │  ├─ test3
│  │  ├─ test3
│  │  ├─ test3

So my question is, what could be the reson of that or how can I solve this?
I do not really understand this behaviour. ( I used minio s3 client)

Thanks

Abstract to also allow GCS and Azure?

Hey,

I recently forked your implementation and refactored it to work with Google Cloud Storage. Maybe we can collaborate and use some sort of file system abstraction library? I am not that into Typescript, so I am not aware of a good Abstraction Library. In the PHP World, we have very good ones and we use them for all kinds of storage abstractions.

WDYT?

Missing 'Authorization' header when download file

<LinkButton
      style={{ textDecoration: 'none' }}
      variant="outlined"
      title={`Download ${objectInfo.downloadName}`}
      to={objectInfo.downloadUrl}
      download={objectInfo.downloadName}
>

When click this linkbutton, the Authorization didn't set into the header. It will return 401 on production env

{
  "error": {
  "name": "AuthenticationError",
  "message": "Missing 'Authorization' header in request"
  },
  "request": {
  "method": "GET",
  "url": "[/stream/test/report%2Fscan_20_report.html?endpoint=test-report](https://portal.8293f99d.nip.io/stream/test/report%2Fscan_20_report.html?endpoint=test-report)"
  },
  "response": {
  "statusCode": 401
  }
}

buckets, 404 Not Found

Hi,

Should I do any further changes after the install based on your doc?
Becase I got this message:

log:
"/api/s3/buckets/grouped? HTTP/1.1" 404 "

GUI:
Request failed for buckets/grouped, 404 Not Found

And I do not really understand the cause, the config is working fine and the permission have already been added, I am currently use openshift.

Do you have any suggestions?

Version @spreadshirt/[email protected] breaks the frontend

Overview

The backend plugin in version 0.8.0 appears to be malfunctioning after the latest release. It can fetch the buckets as expected and shown in the logs, however, they are not being displayed in the frontend. Instead, a 404 error is displayed.
This error is not encountered in version 0.7.6, the same behavior of the backend finding the buckets and not displaying occurs in a production environment.

Environment

Locally with (fedora:39)
Production (Kubernetes:1.24.10 - @backstage/[email protected])

Steps to Reproduce

app-config.yaml used

# Stub app config needed to run with yarn start
app:
  title: backstage example app
  baseUrl: http://localhost:3000

backend:
  baseUrl: http://localhost:7007
  listen:
    port: 7007

permission:
  enabled: true

s3:
  bucketLocatorMethods:
    - type: config
      platforms:
        - endpoint: http://127.0.0.1:8080
          name: test
          accessKeyId: 70VkRWd3IHDxEafKZFX9 # Has to match `CEPH_DEMO_ACCESS_KEY` when using docker compose
          secretAccessKey: v0GerzwTw0cD2Dcq4m0aGeNzQVnpyzc0zW4Mc05A # Has to match `CEPH_DEMO_SECRET_KEY` when using docker compose
          region: 'us-east-1'
  allowedBuckets:
    - platform: test
      buckets:
        - foobar

• git checkout @spreadshirt/[email protected] / git checkout @spreadshirt/[email protected]
• Launch the bucket container / wait for it to be ready
• yarn start

Expected

The bucket should be seen in backstage frontend

Actual

Frontend returns a message with "Request failed for buckets/grouped, 404 Not Found"

Logs

Logs from the container with the bucket

Restarting RESTful API server...
c2752236-ef0d-44ef-92c1-f4efdb3f4190
changed ownership of '/etc/ceph/ceph.client.crash.keyring' from root:root to ceph:ceph
ownership of '/var/lib/ceph/crash/posted' retained as ceph:ceph
ownership of '/var/lib/ceph/crash' retained as ceph:ceph
nohup: ignoring input and appending output to 'nohup.out'
2024-04-05 16:31:00  /opt/ceph-container/bin/entrypoint.sh: SUCCESS
exec: PID 4078: spawning ceph --cluster ceph -w
exec: Waiting 4078 to quit
  cluster:
    id:     3915a179-4e6e-4511-9f25-b60b7b658423
    health: HEALTH_OK
 
  services:
    mon:        1 daemons, quorum bac395a105be (age 2m)
    mgr:        bac395a105be(active, since 2m)
    mds:        1/1 daemons up
    osd:        2 osds: 2 up (since 2m), 2 in (since 2m); 1 remapped pgs
    rbd-mirror: 1 daemon active (1 hosts)
    rgw:        1 daemon active (1 hosts, 1 zones)
    rgw-nfs:    1 daemon active (1 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   9 pools, 233 pgs
    objects: 238 objects, 579 KiB
    usage:   53 MiB used, 200 GiB / 200 GiB avail
    pgs:     233 active+clean
 
  io:
    recovery: 37 B/s, 1 objects/s
 

10.0.2.100 - - [05/Apr/2024 16:32:12] "GET /buckets.html HTTP/1.1" 200 -
10.0.2.100 - - [05/Apr/2024 16:32:12] "GET /css/bootstrap.min.css HTTP/1.1" 200 -
10.0.2.100 - - [05/Apr/2024 16:32:12] "GET /css/font-awesome.min.css HTTP/1.1" 200 -
10.0.2.100 - - [05/Apr/2024 16:32:12] "GET /css/style.css HTTP/1.1" 200 -
10.0.2.100 - - [05/Apr/2024 16:32:12] "GET /css/jquery.dataTables.min.css HTTP/1.1" 200 -
10.0.2.100 - - [05/Apr/2024 16:32:12] "GET /css/validationEngine.jquery.css HTTP/1.1" 200 -
10.0.2.100 - - [05/Apr/2024 16:32:12] "GET /image/ceph-nano-logo-horizontal.svg HTTP/1.1" 200 -
10.0.2.100 - - [05/Apr/2024 16:32:12] "GET /js/lib/require.js HTTP/1.1" 200 -
10.0.2.100 - - [05/Apr/2024 16:32:12] "GET /js/lib/require.config.js HTTP/1.1" 200 -

Logs from yarn start in branch @spreadshirt/[email protected]

 xxx@fedora  ~/Downloads/backstage-plugin-s3  ◈ @spreadshirt/[email protected]  yarn start
yarn run v1.22.22
$ concurrently "yarn start:backend" "yarn start:frontend"
$ yarn workspace @spreadshirt/backstage-plugin-s3-viewer-backend start
$ yarn workspace @spreadshirt/backstage-plugin-s3-viewer start
$ LEGACY_BACKEND_START=1 backstage-cli package start
$ backstage-cli package start
[0] Build succeeded
[1] Loaded config from app-config.yaml
[1] <i> [webpack-dev-server] Project is running at:
[1] <i> [webpack-dev-server] Loopback: http://localhost:3000/, http://[::1]:3000/
[1] <i> [webpack-dev-server] Content not from webpack is served from '/home/xxx/Downloads/backstage-plugin-s3/plugins/s3-viewer/public' directory
[1] <i> [webpack-dev-server] 404s will fallback to '/index.html'
[1] <i> [webpack-dev-middleware] wait until bundle finished: /
[0] Loaded config from app-config.yaml
[0] 2024-04-05T17:01:38.269Z s3-viewer-backend info Found 1 new secrets in config that will be redacted 
[0] 2024-04-05T17:01:38.282Z s3-viewer-backend warn Generated a secret for service-to-service authentication: DEVELOPMENT USE ONLY. 
[0] 2024-04-05T17:01:38.283Z s3-viewer-backend info Initializing S3 backend 
[0] 2024-04-05T17:01:38.285Z s3-viewer-backend info Fetching S3 buckets... 
[0] 2024-04-05T17:01:38.311Z s3-viewer-backend info CORS is disabled, allowing all origins 
[0] 2024-04-05T17:01:38.325Z backstage info Listening on :7007 
[0] 2024-04-05T17:01:38.443Z s3-viewer-backend info Fetched 1 S3 buckets 
[1] webpack compiled successfully
[0] 2024-04-05T17:01:48.553Z backstage info ::ffff:127.0.0.1 - - [05/Apr/2024:17:01:48 +0000] "GET /api/s3-viewer/buckets/grouped? HTTP/1.1" 404 - "http://localhost:3000/" "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0" type=incomingRequest
[0] 2024-04-05T17:01:55.270Z backstage info ::ffff:127.0.0.1 - - [05/Apr/2024:17:01:55 +0000] "GET /api/s3-viewer/buckets/grouped? HTTP/1.1" 404 - "http://localhost:3000/" "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0" type=incomingRequest

Logs from yarn start in branch @spreadshirt/[email protected]

xxx@fedora  ~/Downloads/backstage-plugin-s3  ◈ @spreadshirt/[email protected]  yarn start                                              
yarn run v1.22.22
$ concurrently "yarn start:backend" "yarn start:frontend"
$ yarn workspace @spreadshirt/backstage-plugin-s3-viewer-backend start
$ yarn workspace @spreadshirt/backstage-plugin-s3-viewer start
$ LEGACY_BACKEND_START=1 backstage-cli package start
$ backstage-cli package start
[0] Build succeeded
[1] Loaded config from app-config.yaml
[1] <i> [webpack-dev-server] Project is running at:
[1] <i> [webpack-dev-server] Loopback: http://localhost:3000/, http://[::1]:3000/
[1] <i> [webpack-dev-server] Content not from webpack is served from '/home/xxx/Downloads/backstage-plugin-s3/plugins/s3-viewer/public' directory
[1] <i> [webpack-dev-server] 404s will fallback to '/index.html'
[1] <i> [webpack-dev-middleware] wait until bundle finished: /
[0] Loaded config from app-config.yaml
[0] 2024-04-05T16:33:30.677Z s3-viewer-backend info Found 1 new secrets in config that will be redacted 
[0] 2024-04-05T16:33:30.688Z s3-viewer-backend warn Generated a secret for service-to-service authentication: DEVELOPMENT USE ONLY. 
[0] 2024-04-05T16:33:30.689Z s3-viewer-backend info Initializing S3 backend 
[0] 2024-04-05T16:33:30.690Z s3-viewer-backend info Fetching S3 buckets... 
[0] 2024-04-05T16:33:30.704Z s3-viewer-backend info CORS is disabled, allowing all origins 
[0] 2024-04-05T16:33:30.719Z backstage info Listening on :7007 
[0] 2024-04-05T16:33:30.840Z s3-viewer-backend info Fetched 1 S3 buckets 
[1] webpack compiled successfully
[0] 2024-04-05T16:33:42.037Z backstage info ::1 - - [05/Apr/2024:16:33:42 +0000] "POST /api/permission/authorize HTTP/1.1" 200 74 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type=incomingRequest
[0] 2024-04-05T16:33:42.044Z backstage info ::ffff:127.0.0.1 - - [05/Apr/2024:16:33:42 +0000] "GET /api/s3/buckets/grouped? HTTP/1.1" 304 - "http://localhost:3000/" "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0" type=incomingRequest
[0] 2024-04-05T16:33:59.251Z backstage info ::1 - - [05/Apr/2024:16:33:59 +0000] "POST /api/permission/authorize HTTP/1.1" 200 74 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" type=incomingRequest
[0] 2024-04-05T16:33:59.256Z backstage info ::ffff:127.0.0.1 - - [05/Apr/2024:16:33:59 +0000] "GET /api/s3/buckets/grouped? HTTP/1.1" 304 - "http://localhost:3000/" "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0" type=incomingRequest
^C[0] 2024-04-05T16:35:06.221Z backstage info CTRL+C pressed; exiting. 
[1] <i> [webpack-dev-server] Gracefully shutting down. To force exit, press ^C again. Please wait...

Aditional info

In version 0.7.6 in a production environment there seems to be another issue regarding permissions, could you provide a working "app-config.yaml" in a production environment.

Shall we consider setting the default region as `us-east-1`?

This what backstage does.

https://github.com/backstage/backstage/blob/999936b3915d807206f833f44c8397775780a22e/packages/backend-common/src/reading/AwsS3UrlReader.ts#L50

export const DEFAULT_REGION = 'us-east-1';

Shall we consider this?

With this approach, the region is not a breaking change any more.

Configure with IAM

Hi team, can you guide me on what is the correct way to configure this plugin to use IAM role (without passing AWS secret key and access key)? I did implement Custom Credentials Providers which worked for listing buckets but for listing inner contents it uses S3 Client which I am unable to implement. Also, I wanna know is this the correct way? Any help would be appreciated.

Add yourself to Community Plugin Directory

Hej Spreadshirt-Team,

awesome to see this continous plugin development & maintainance by your team. I wasn't really aware this plugin exists & stumbled over it by luck. Would you be interested in adding yourself to the community plugin directory (Guide). This could make it easier for others to discover this plugin & hopefully contributing back to improve it further! :)

Special shout-out to @ivangonzalezacuna for contributing a lot to the community by being active in the Backstage Discord as well as directly contributing to the upstream improving Backstage overall! 🎉