bluesky / bluesky-httpserver

HTTP API to Queueserver

Bluesky — An Experiment Specification & Orchestration Engine

Source	https://github.com/bluesky/bluesky
PyPI	pip install bluesky
Documentation	https://bluesky.github.io/bluesky
Releases	https://github.com/bluesky/bluesky/releases

Bluesky is a library for experiment control and collection of scientific data and metadata. It emphasizes the following virtues:

• Live, Streaming Data: Available for inline visualization and processing.
• Rich Metadata: Captured and organized to facilitate reproducibility and searchability.
• Experiment Generality: Seamlessly reuse a procedure on completely different hardware.
• Interruption Recovery: Experiments are "rewindable," recovering cleanly from interruptions.
• Automated Suspend/Resume: Experiments can be run unattended, automatically suspending and resuming if needed.
• Pluggable I/O: Export data (live) into any desired format or database.
• Customizability: Integrate custom experimental procedures and commands, and get the I/O and interruption features for free.
• Integration with Scientific Python: Interface naturally with numpy and Python scientific stack.

Bluesky Documentation.

The Bluesky Project enables experimental science at the lab-bench or facility scale. It is a collection of Python libraries that are co-developed but independently useful and may be adopted a la carte.

Bluesky Project Documentation.

See https://bluesky.github.io/bluesky for more detailed documentation.

bluesky-httpserver's Issues

The "config_schemas" path not included in packaged distribution.

If providing a server configuration json file the service_configuration.yml schemas is required:

bluesky-httpserver/bluesky_httpserver/config.py

Line 22 in b2db63e

schema_path = os.path.join(here, "config_schemas", "service_configuration.yml")

However, the config_schemas path hasn't been included in the manifest.

How to allow full unauthenticated access to the server

Unauthenticated public access to the server should be disabled in production deployments. It was intentionally made inconvenient to enable to prevent security issues. Basic anonymous access is enabled by setting authentication/allow_anonymous_access in the config file, but the default permissions only allows reading status of the server, which is generally safe. But since there is some interest in running the server without authentication in local test environments, the documentation should include the instructions how to do it. The following config file allows anonymous access and adds all API scopes to unauthenticated_public group. This allows unauthenticated access to all API. Remove scopes from the list to block access to respective API.

The example also shows how to specify control and info addresses (0MQ sockets) of RE Manager.

qserver_zmq_configuration:
  control_address: tcp://localhost:60615
  info_address: tcp://localhost:60625
authentication:
  allow_anonymous_access: True
api_access:
  policy: bluesky_httpserver.authorization:BasicAPIAccessControl
  args:
    roles:
      unauthenticated_public:
        scopes_add:
          - read:status
          - read:queue
          - read:history
          - read:resources
          - read:config
          - read:monitor
          - read:console
          - read:lock
          - read:testing
          - write:queue:edit
          - write:queue:control
          - write:manager:control
          - write:plan:control
          - write:execute
          - write:history:edit
          - write:permissions
          - write:scripts
          - write:config
          - write:lock
          - write:manager:stop
          - write:testing
          - user:apikeys
          - admin:apikeys
          - admin:read:principals
          - admin:metrics

Requirements for authorization management

This issue contains a minimum list of authorization data required by different modules of the queue server. The purpose is to find appropriate way to store this data on a secured system. The list could be expanded with more data to incorporate future functionality. It is undesirable to remove any items from the list, since it will effectively disable already existing features of the Queue Server. It is assumed that the system already provides secure user authentication and the server can use user ID to access the data.

• The list of users currently allowed to access the beamline. The scope of access is determined by the API access group name, which is set separately (see below). For example, some users may be allowed to monitor execution of plans, but not control the beamline. Adding and removing users from the list should not change API access group name.

• API access group name (string) for each user for each beamline. The HTTP server can provide fine-grained control of user access to API. The API are split into logical groups and the scope for each user may include any combination of those groups (see the examples of scopes for the default groups used in demo

  bluesky-httpserver/bluesky_httpserver/database/core.py

  Lines 27 to 85 in 5277d51

  	db.add(
  	Role(
  	name="user",
  	description="Default Role for users.",
  	scopes=[
  	"read:queue",
  	"read:history",
  	"read:resources",
  	"read:config",
  	"read:monitor",
  	"read:console",
  	"read:status",
  	"read:lock",
  	"write:queue:edit",
  	"write:queue:control",
  	"write:manager:control",
  	"write:plan:control",
  	"write:execute",
  	"write:history:edit",
  	"write:permissions",
  	"write:scripts",
  	"write:config",
  	"write:lock",
  	"write:unsafe",
  	"apikeys",
  	],
  	),
  	)
  	db.add(
  	Role(
  	name="admin",
  	description="Role with elevated privileges.",
  	scopes=[
  	"read:queue",
  	"read:history",
  	"read:resources",
  	"read:config",
  	"read:monitor",
  	"read:console",
  	"read:status",
  	"read:lock",
  	"write:queue:edit",
  	"write:queue:control",
  	"write:manager:control",
  	"write:plan:control",
  	"write:execute",
  	"write:history:edit",
  	"write:permissions",
  	"write:scripts",
  	"write:config",
  	"write:lock",
  	"write:unsafe",
  	"admin:apikeys",
  	"read:principals",
  	"metrics",
  	],
  	),
  	)
  	db.commit()

  ).

• Definitions of scopes for API access user groups (dictionary) for each beamline. While it is possible to define fixed set user groups in the source code to cover the basic access needs, it is likely that beamline staff may request more fine grained control with ability to add new groups and change scopes for the existing groups.

• Plan/device access group name (string) for each user for each beamline. This name is different from API access group name and determines which plans and devices the user can access. The user groups may be defined in a YAML file in profile_collection directory and automatically loaded by the Queue Server at startup or stored in the database and uploaded to the Queue Server using API. The plan/device access groups are defined mostly for convenience (avoid errors due to accidental access and reduce clutter in lists of allowed plans and devices and related GUI menus), may be changed dynamically if needed and are not considered secured information.

bluesky_httpserver crashing on startup with current version of pydantic 2.x.x

Expected Behavior

Server should startup normally and print some info like:

$ uvicorn --host localhost --port 60610 bluesky_httpserver.server:app
...
INFO:     Application startup complete.
...

Current Behavior

$ uvicorn --host localhost --port 60610 bluesky_httpserver.server:app

Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 416, in main
    run(
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 587, in run
    server.run()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve
    config.load()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 467, in load
    self.loaded_app = import_from_string(self.app)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.11/site-packages/bluesky_httpserver/server.py", line 10, in <module>
    from .app import build_app
  File "/usr/local/lib/python3.11/site-packages/bluesky_httpserver/app.py", line 18, in <module>
    from .authentication import Mode
  File "/usr/local/lib/python3.11/site-packages/bluesky_httpserver/authentication.py", line 24, in <module>
    from pydantic import BaseModel, BaseSettings
  File "/usr/local/lib/python3.11/site-packages/pydantic/__init__.py", line 207, in __getattr__
    return _getattr_migration(attr_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pydantic/_migration.py", line 288, in wrapper
    raise PydanticImportError(
pydantic.errors.PydanticImportError: `BaseSettings` has been moved to the `pydantic-settings` package. See https://docs.pydantic.dev/2.0.3/migration/#basesettings-has-moved-to-pydantic-settings for more details.

For further information visit https://errors.pydantic.dev/2.0.3/u/import-error

Possible Solution

Setting a fixed v1 version of pydantic in the requirements of the bluesky_httpserver package could make it usable again when installing from pip.

Quick fix:

pip install pydantic==1.10.11

Steps to Reproduce (for bugs)

1. python3 -m venv .venv
2. source .venv/bin/activate
3. pip install bluesky-httpserver==0.0.10
4. uvicorn --host localhost --port 60610 bluesky_httpserver.server:app

Your Environment

python 3.11

Enforce limits on API keys, Sessions

The API keys and Sessions implementation, copied from Tiled, might benefit from this recent change to Tiled as well:

bluesky/tiled#252