remko / dsadmin

Google Cloud Datastore Emulator Administration UI

DSAdmin: Google Cloud Datastore Emulator Admin UI

Administration GUI for the Google Cloud Datastore Emulator.

• Supports browsing, editing, creating, deleting, querying (using GQL), import, export, ...
• Supports formatted display of JSON properties & compressed properties
• Supports large databases
• Does not have problematic dependencies (such as gRPC, which are not available on all platforms)
• Portable: Implemented entirely in the frontend as an SPA. Uses the Datastore REST API directly from the frontend. The only thing a server is used for is for proxying to the datastore emulator (to avoid CORS problems), and to serve the HTML and JS files. The NPM package uses a small Node.js server. A small self-contained binary server is also provided, avoiding the need for any system dependencies. If for some reason you need this to be available in a specific environment (Java, Python, ...), it should be easy to create a similar server to run this in.

📷 Screenshots

  

❓ Why?

Since Google stopped shipping an admin interface for their Datastore Emulator, there have been some external projects trying to fill the gap. However, all of the ones I tried were either partially or completely broken, lacked in core features, had performance issues, were painful to set up, or had dependencies that prevented them from e.g. being installed in a non-x86 Docker image.

This project tries to fix all of the issues with the other Datastore admin interfaces, and bring improvements on the original Google interface.

🚧 Not yet implemented

• Editing nested entities
• UI for filtering & projecting

📖 Usage

Using NPM

Using the environment from the emulator:

eval $(gcloud beta emulators datastore env-init --data-dir=DATA-DIR)
npx dsadmin

Using command-line arguments:

npx dsadmin --project=my-datastore-project --datastore-emulator-host=localhost:8081

Using a pre-built binary

Download the correct binary for your OS from the Releases page.

Start using the environment from the emulator:

eval $(gcloud beta emulators datastore env-init --data-dir=DATA-DIR)
./dsadmin

Start using command-line arguments:

./dsadmin --project=my-datastore-project --datastore-emulator-host=localhost:8081

Using Docker

docker run -p 8080:8080 ghcr.io/remko/dsadmin:latest \
  --project=my-project --datastore-emulator-host=host.docker.internal:8081

Using Docker Compose

Create a docker-compose.yml that starts the Datastore Emulator and the Datastore Admin container:

version: "3.9"
services:
  # DSAdmin container
  dsadmin:
    image: "ghcr.io/remko/dsadmin:latest"
    depends_on:
      - datastore
    ports:
      - "8080:8080"
    environment:
      DATASTORE_PROJECT_ID: my-datastore-project
      DATASTORE_EMULATOR_HOST: "datastore:8081"

  # Datastore Emulator container
  datastore:
    image: "gcr.io/google.com/cloudsdktool/cloud-sdk:latest"
    volumes:
      - datastore_data:/opt/datastore/data
    ports:
      - "8081:8081"
    command: [
      "gcloud", "--quiet", "beta", "emulators" ,"datastore", "start", 
      "--host-port=0.0.0.0:8081", "--data-dir=/opt/datastore/data"
    ]
    environment:
      CLOUDSDK_CORE_PROJECT: my-datastore-project

volumes:
  datastore_data:

💻 Development

Install all dependencies

yarn

Start all development servers (datastore emulator, proxy, and frontend build):

yarn start

If you want to run against your own running instance of the Datastore emulator, start the backend server and the build server separately:

eval $(gcloud beta emulators datastore env-init --data-dir=DATA-DIR)
./bin/dsadmin.js
yarn run start-build

or using command-line arguments:

./bin/dsadmin.js --project=my-datastore-project --datastore-emulator-host=localhost:8081
yarn run start-build

🔋 Powered by ...

This project uses some great open source projects. Check them out if you don't know them.

                

dsadmin's Issues

Error: unsupported value "entityValue"

I get an error when trying to view an entity from properties.js in the function valueToEditValue, as there is no handling for values of type "entityValue".

Adding basepath

Is there any way you can add basepath support for reverse proxying? The script includes on the index.html are hardcoded to the root.

Copy 'indexed' value when creating new array value

When adding an array value to an indexed array, copy the 'exclude from indexes' flag from the other values. Datastore requires that either all or none of the values have to be indexed.

'Delete' from GQL query view does not refresh list

triggerUncaughtException while running command

Steps:

1. Running command npx dsadmin --project=my-datastore-project --datastore-emulator-host=localhost:8081
2. Project - https://github.com/oppia/oppia/

npx dsadmin --project=my-datastore-project --datastore-emulator-host=localhost:8081 
dsadmin (project my-datastore-project) listening on http://localhost:8080
node:internal/process/promises:227
          triggerUncaughtException(err, true /* fromPromise */);
          ^

Error: connect ECONNREFUSED 127.0.0.1:8081
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1139:16) {
  errno: -61,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 8081,
  statusCode: 503,
  connectedSocket: false,
  reusedSocket: false
}
npm ERR! code 1
npm ERR! path /Desktop/opensource/oppia
npm ERR! command failed
npm ERR! command sh -c dsadmin "--project=my-datastore-project" "--datastore-emulator-host=localhost:8081"

npm ERR! A complete log of this run can be found in:
npm ERR!     /.npm/_logs/2021-12-17T02_11_46_504Z-debug.log

Chrom console log

api.js:17 POST http://localhost:8080/v1/projects/my-datastore-project:runQuery net::ERR_EMPTY_RESPONSE
Wu @ api.js:17
id @ api.js:30
(anonymous) @ api.js:65
P @ react-query.js:1050
E @ react-query.js:620
ww @ react-query.js:680
t.fetch @ react-query.js:1081
n.executeFetch @ react-query.js:2336
n.onSubscribe @ react-query.js:2189
t.subscribe @ react-query.js:34
(anonymous) @ react-query.js:2960
o5 @ index-3eae4d6e.js:276
t.unstable_runWithPriority @ index-3eae4d6e.js:14
ho @ index-3eae4d6e.js:136
Ri @ index-3eae4d6e.js:275
(anonymous) @ index-3eae4d6e.js:275
Fe @ index-3eae4d6e.js:12
I.port1.onmessage @ index-3eae4d6e.js:8
react-query.js:1107 TypeError: Failed to fetch
    at Wu (api.js:17)
    at id (api.js:30)
    at Object.queryFn (api.js:65)
    at Object.P [as fn] (react-query.js:1050)
    at E (react-query.js:620)
    at new ww (react-query.js:680)
    at e.t.fetch (react-query.js:1081)
    at t.n.executeFetch (react-query.js:2336)
    at t.n.onSubscribe (react-query.js:2189)
    at t.subscribe (react-query.js:34)
onError @ react-query.js:1107
f @ react-query.js:592
(anonymous) @ react-query.js:656
Promise.catch (async)
E @ react-query.js:640
ww @ react-query.js:680
t.fetch @ react-query.js:1081
n.executeFetch @ react-query.js:2336
n.onSubscribe @ react-query.js:2189
t.subscribe @ react-query.js:34
(anonymous) @ react-query.js:2960
o5 @ index-3eae4d6e.js:276
t.unstable_runWithPriority @ index-3eae4d6e.js:14
ho @ index-3eae4d6e.js:136
Ri @ index-3eae4d6e.js:275
(anonymous) @ index-3eae4d6e.js:275
Fe @ index-3eae4d6e.js:12
I.port1.onmessage @ index-3eae4d6e.js:8
api.js:17 POST http://localhost:8080/v1/projects/my-datastore-project:runQuery net::ERR_CONNECTION_REFUSED
Wu @ api.js:17
id @ api.js:30
(anonymous) @ api.js:50
P @ react-query.js:1050
E @ react-query.js:620
ww @ react-query.js:680
t.fetch @ react-query.js:1081
n.executeFetch @ react-query.js:2336
n.onSubscribe @ react-query.js:2189
t.subscribe @ react-query.js:34
(anonymous) @ react-query.js:2960
o5 @ index-3eae4d6e.js:276
t.unstable_runWithPriority @ index-3eae4d6e.js:14
ho @ index-3eae4d6e.js:136
Ri @ index-3eae4d6e.js:275
(anonymous) @ index-3eae4d6e.js:275
Fe @ index-3eae4d6e.js:12
I.port1.onmessage @ index-3eae4d6e.js:8
react-query.js:1107 TypeError: Failed to fetch
    at Wu (api.js:17)
    at id (api.js:30)
    at Object.queryFn (api.js:50)
    at Object.P [as fn] (react-query.js:1050)
    at E (react-query.js:620)
    at new ww (react-query.js:680)
    at e.t.fetch (react-query.js:1081)
    at t.n.executeFetch (react-query.js:2336)
    at t.n.onSubscribe (react-query.js:2189)
    at t.subscribe (react-query.js:34)
onError @ react-query.js:1107
f @ react-query.js:592
(anonymous) @ react-query.js:656
Promise.catch (async)
E @ react-query.js:640
ww @ react-query.js:680
t.fetch @ react-query.js:1081
n.executeFetch @ react-query.js:2336
n.onSubscribe @ react-query.js:2189
t.subscribe @ react-query.js:34
(anonymous) @ react-query.js:2960
o5 @ index-3eae4d6e.js:276
t.unstable_runWithPriority @ index-3eae4d6e.js:14
ho @ index-3eae4d6e.js:136
Ri @ index-3eae4d6e.js:275
(anonymous) @ index-3eae4d6e.js:275
Fe @ index-3eae4d6e.js:12
I.port1.onmessage @ index-3eae4d6e.js:8
:8080/favicon.ico:1 GET http://localhost:8080/favicon.ico net::ERR_CONNECTION_REFUSED

DSAdmin crashes when changing namespace from default

Hi! Thanks for making this killer application.

We're trying to use it with the datastore emulator at my company, but we're running into a problem when switching namespaces. The entire application simply crashes.

Let me know what information you need to debug this and I'll happily provide:)

Changing sort with a namespace selected looses the namespace

When I select a namespace, i can see all the data in that namespace.
When I then click on the tableheader to change the order, the namespace resets back to the default namespace.

The /namespaces/namespaceID/ part in the url seems to be missing.
When I manually add this namespace part to the url, is see the expected data in the expected order.

I just noticed: The same issue seem to occur when clicking on the next/previous buttons or selecting a different rows per page.

Thank you for creating this great tool. This really is the best tool I found.

Editing nested entities?

Hi, this business with editing nested entities, is there and idea of how that could/should be solved?

Keep getting Error: Could no load default credential

I'm sure this is something that I'm doing wrong but I'm having issues getting this to work. I was able to use docker-compose to start both the dsadmin and datastore services. I can view it running on the ui and make changes fine. When I start my web app in development it seems like it is still trying to reach out to the production datastore and not the emulator. I'm assuming that I'm not setting environment variables correctly, I set them to match what was in the docker-compose file. Any suggestions

Datastore UI is not working with datastore in firestore mode

I am trying to make the UI work with datastore in firestore mode. This is my docker-compose.yaml

version: "3.9"
services:
  # DSAdmin container
  dsadmin:
    image: "ghcr.io/remko/dsadmin:latest"
    depends_on:
      - datastore
    ports:
      - "3334:8080"
    environment:
      DATASTORE_PROJECT_ID: "project-dev"
      DATASTORE_EMULATOR_HOST: "datastore:3333"

  # Datastore Emulator container
  datastore:
    image: "gcr.io/google.com/cloudsdktool/google-cloud-cli:latest"
    volumes:
      - datastore_data:/opt/datastore/data
    ports:
      - "3333:3333"
    command: [
      "gcloud", "emulators", "firestore", "start","--database-mode=datastore-mode", "--host-port=0.0.0.0:3333"
    ]
    environment:
      CLOUDSDK_CORE_PROJECT: "project-dev"

volumes:
  datastore_data:

But for some reason I keep on getting this Error.

What can I do about it?