Comments (3)
I think we would get rid of the KEYS_PER_PERIOD
concept and probably handle each strategy different.
- all strategies would be "throttled" by their
mpsc::Channel
being "full" so they would select as fast as they can and then block once their channel is full. - random can just select as fast as possible, probably in some kind of batch size since we probably don't want to hammer the database with single row lookups.
- latest should probably get "smarter" and work backwards from the front of the chain stopping either once it hits the previous latest or some other maximum batch size.
- oldest un-audited can probably select as fast as it wants as long as it has protection from filling the queue with duplicates.
from glados.
I have an alternate suggestion that I believe gives us both of the following.
- ability to control overall audit rate
- ability to throttle each source/strategy in relation to each other (e.g. 50% towards
latest
, 30% towardsrandom
, 20% towardsoldest-unaudited
I suggest the --concurrency
flag remains to determine how many lookups should happen in parallel.
I suggest the following flags where N
is a positive integer argument:
--latest-strategy-weight=N
--random-strategy-weight=N
- ... (etc)
We then interpret these as the respective weights for how many jobs we pull from each source.
Implementation detail: This would also imply that we give each strategy its own
mpsc::Channel
to enable the logic for spreading the audits across the different sources.
Each strategy can then produce audit candidates at maximum rate all of the time and will self-throttle when the channel is full.
from glados.
Ok that sounds good. So to check:
If there are two strategies (A
weight=1, B
weight=2), with KEYS_PER_PERIOD=10
. Then:
- Total weight = 1 + 2 = 3
A
= 10 * 1//3 = 3 keys per periodB
= 10 * 2//3 = 7 keys per period
Or you could pass 10 and 20 for weights respectively to get 30 and 70 keys per period respectively
from glados.
Related Issues (20)
- Add Radius and "Should store" column in Audit page
- Web view for DHT census data HOT 1
- A way to view audit metrics over time
- Glados doesn't look good on mobile devices
- Client Pie Chart turns into a black circle when only one type of client is found HOT 1
- Backfill mode
- Relative timestamps HOT 1
- Show census deadzones
- calculate stats per strategy type
- Improvements to content key page
- Improvements to audit page
- Generate graphs from census data.
- Show "delta" in census detail
- glados-web unavailable due to panicks HOT 3
- Enr Decoder custom javascript long term maintenance concerns/issue
- fluffy doesn't work for latest audit
- used fluffy on glados and audit client tag was still said trin
- Initial state-root based auditing strategy for state network. HOT 12
- FourFours data showing invalid "First Available At" value
- Provide database backups for researchers
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glados.