Comments (10)
Note, the intent of this feature request is to only provide basic filtering capabilities when looking at a queue in a particular state, although it would be nice to think on solving that problem in a way that it may open new features as the ones that @krazyjakee is suggesting. But I think it'd be better to solve a small problem at a time.
I'm thinking that a solution for this that is efficient for very large queues is to use SCAN
cursors. I'm not sure if bee
supports job listing with a cursor, the idea is that since you have to do a full scan for filtering anyway, we can pull 10 jobs in some state like we do with pagination, and then apply the filter on these 10 jobs. Then, if 3 jobs were filtered out of these 10 jobs, then get the next set of 10 jobs with the SCAN
cursor until you have a set of 10 jobs.
Pagination can become tricky though because we can't know how many pages a particular search has without doing a full scan which is expensive. I'm thinking that we can replace pagination with a cursor hash that represents the filter and the redis SCAN
cursor together, then a "next page" button or a "load more" button will fetch more jobs until the cursor is exhausted. The tradeoff is that we can't know how many results a filter has but I believe that it is fine to exclude traditional pagination when doing searches
from arena.
Great idea.
Sadly, redis does not support value searches but we could hand the entire job set over to the client in the form of a clusterize table which can support hundreds of thousands of rows without a performance hit. A form could then be constructed to perform the filtering and update results in the table. It would need a lot of testing.
If a client-side solution doesn't work, the server would need to create an internal database that syncs with redis and could be queried via ajax. We could also keep and improve pagination this way.
from arena.
Some of our queues are 3GB, so sending all that data to the client for filtering won't work.
Why not have the server just read each job from redis, perform the filter, and then stream results to the client?
from arena.
@bradvogel ok, 3GB is a great example to think about. It's worth testing at least but I doubt we could have such functionality as an autocomplete. I guess we'd need a syncd database loaded with indexes to achieve that.
from arena.
Nah - it'd have to stream each job (key) from redis and run the filter in the backend, then stream it to the frontend if it matches.
from arena.
@bradvogel ah! I hear you. Yeah I'm picturing loading indicator, progress bar, options to filter by job data by values & response content. Could be a great addition.
from arena.
Hello!
Is there still a feature ongoing for this issue? :)
Would be very handy!
Thanks for the good work :)
from arena.
Any news on this issue?
from arena.
ping
from arena.
While I understand that this ticket has a lot of interest, please refrain from simply bumping - it's not a priority for us, and we don't really have resources to fully support this project. I'm locking and restricting to collaborators. The best way to get progress on this ticket would be to submit a pull request! Linked it to this ticket :)
from arena.
Related Issues (20)
- Bull arena inside docker just loading HOT 7
- [Security] Critical security issue in "handlebars" dependency HOT 2
- Search filter for job name
- Progress Reporting for BeeQueue
- Update docker image HOT 1
- Views directory is pointing to the wrong place HOT 1
- Can we replace the add job with a simple textArea? HOT 2
- [SECURITY] Critical security vulnerability in express-handlebars dependency HOT 2
- 404 when clicking on job HOT 1
- REGRESSION ERROR: Arena Bootstrap bug HOT 2
- `url` option no longer works for connection to BullMQ
- TypeError when using Bull < v3.15.0
- How to add/remove queues dinamically in the UI? HOT 1
- Filtering jobs with Javascript queries, if there is interest we can do a PR for this in the bee-queue project
- Not able to perform delete/select all/remove job from dashboard? HOT 1
- Ability to handle redis server unavailability
- THIS PROJECT IS LOOKING FOR A NEW OWNER HOT 1
- Adding support for printing logs on UI
- ERR: "Converting circular structure to JSON" when the JSON button is clicked HOT 1
- Delayed jobs: Cannot read properties of undefined (reading 'getState') HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arena.