Comments (8)
I like the method @fubuloubu gave, but historically I've always used environment variables or a .env file to get pulled in and managed by pydantic settings. Then all of the settings handle building your config. I'd be for either method of dealing with this
I think what essentially you can do is something like this:
class SilverbackConfig(BaseSettings):
BROKER_CLASS: str
BROKER_ARGS: dict[str, Any] = {}
$ SILVERBACK_BROKER_CLASS="..." SILVERBACK_BROKER_ARGS='{"uri": "https://..."}' silverback run ...
and then it would essentially give you a dict like {"uri": "https://..."}
you can destructure with **
:
broker = broker_class(**broker_args)
from silverback.
SBK-458 More dynamic broker configuration
from silverback.
One of the options would seem to be a config file of some sort. For ape core, one thing I want to do in 0.9 is allow config via pyproject.toml "tools", which pydantic-settings supports
https://docs.pydantic.dev/latest/concepts/pydantic_settings/#pyprojecttoml
from silverback.
I like the method @fubuloubu gave, but historically I've always used environment variables or a .env file to get pulled in and managed by pydantic settings. Then all of the settings handle building your config. I'd be for either method of dealing with this
from silverback.
Never really been a fan of the .env
file pattern. Secret settings that override env vars in a hidden file that sometimes accidentally get committed to the repo isn't super ideal.
I'm a little curious if we could have a config file that merges with our settings, but not sure that really works well either. The problem isn't that we can't use env vars (we can), but how do we provide arbitrary arguments to the constructor using said settings.
For instance, here's the new signature that brought this on
def.__init__(
self,
sqs_queue_url: str,
wait_time_seconds: int = 0, # Used for long polling
max_number_of_messages: int = 1, # size of batch to receive from the queue
result_backend: Optional[AsyncResultBackend] = None,
task_id_generator: Optional[Callable[[], str]] = None,
) -> None:
If we used something like env vars we'd need to do some fancy parsing to get that to work.
from silverback.
Never really been a fan of the
.env
file pattern. Secret settings that override env vars in a hidden file that sometimes accidentally get committed to the repo isn't super ideal.I'm a little curious if we could have a config file that merges with our settings, but not sure that really works well either. The problem isn't that we can't use env vars (we can), but how do we provide arbitrary arguments to the constructor using said settings.
For instance, here's the new signature that brought this on
def.__init__( self, sqs_queue_url: str, wait_time_seconds: int = 0, # Used for long polling max_number_of_messages: int = 1, # size of batch to receive from the queue result_backend: Optional[AsyncResultBackend] = None, task_id_generator: Optional[Callable[[], str]] = None, ) -> None:If we used something like env vars we'd need to do some fancy parsing to get that to work.
I don't think there is a good solution for parsing complex classes as types to add to args that doesn't involve doing something complex to handle it
Perhaps that is just never how the tools are configured to work, you would be expected to do the conversion upstream before you initialize a class that needs to pass a more complex class as an arg
from silverback.
oops, just saw your reply.
class SilverbackConfig(BaseSettings): BROKER_CLASS: str BROKER_ARGS: dict[str, Any] = {}
That's really interesting. Had no idea we could provide complex types as env vars. From the pydantic docs:
Complex types like list, set, dict, and sub-models are populated from the environment by treating the environment variable's value as a JSON-encoded string.
Another way to populate nested complex variables is to configure your model with the env_nested_delimiter config setting, then use an environment variable with a name pointing to the nested module fields. What it does is simply explodes your variable into nested models or dicts. So if you define a variable FOO__BAR__BAZ=123 it will convert it into FOO={'BAR': {'BAZ': 123}} If you have multiple variables with the same structure they will be merged.
We could even provide them individually.
This at least wouldn't require much in the way of changes to support immediately. 👍
from silverback.
We could even provide them individually.
I think this requires a specific submodel to have it work, it doesn't work with a generic dict
from silverback.
Related Issues (20)
- [SBK-314] Add REST API for command/control/monitoring
- bug(client): rounding error on allowance #227 [SBK-341]
- bug(client): Transaction processing error on approve() #228 [SBK-342]
- discussion: start_block parameter in decorator [SBK-360] HOT 2
- bug: BlockNotFoundError: Missing latest block. Reason: when sending a str, it must be a hex string. Got: 'latest' HOT 2
- Not a valid TransactionType error with goerli/sepolia testnet HOT 14
- feat: Logging results to console HOT 2
- Allow filtering for Event Log tasks HOT 1
- Refactor Runner to be stateless HOT 1
- Managed Parameters HOT 1
- Runtime Data Acquistion HOT 3
- Gracefully resolve shutdown HOT 1
- use GHCR instead of Dockerhub HOT 1
- `silverback worker` does not use independent processes per worker HOT 4
- bug: websockets.exceptions.ConnectionClosedError: no close frame received or sent HOT 2
- Track `app.signer` transactions in `TaskResult` HOT 1
- Add autosign to runner HOT 2
- !breaking: reference newly created base ape image for the silverback image HOT 1
- feat: add support for defining tools and settings in toml HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from silverback.