milvus-io / birdwatcher Goto Github PK

Is your feature request related to a problem? Please describe.

Add a "web --port ${port}" command to listen on the specified port after starting Milvus. Users can view the status of Milvus through a web page or API.

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

Current Behavior

birdwatcher may panic when writing audit log

Expected Behavior

Birdwatcher shall not panic

Steps To Reproduce

Run write/delete operations

Environment

No response

Anything else?

No response

What would you like to be added?

Add build options to exclude CGO part

Why is this needed?

cgo id dedicated for kafka cp repairing. If it's not needed, related code could be excluded to avoid cgo compilation complexity.

Anything else?

No response

What would you like to be added?

Use "github.com/cockroachdb/errors" to replace other errors package

Why is this needed?

To unify error handling

Anything else?

No response

What would you like to be added?

add command to display index information

Why is this needed?

It's important to known collection/ segment related index info

Anything else?

No response

Current Behavior

As I typed "birdwatcher --version" and "birdwatcher version", they both start the interactive mode:

And no way to print the version, even in the interactive mode:

Expected Behavior

Only print the version with the two commands

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

Current Behavior

For each state, command flags are not reset after execution

Expected Behavior

Flag value shall be renewed after each command execution

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

Current Behavior

My Milvus is deployed by helm, so the default rootPath is by-dev.

But I refer to the following command, set the rootPath as instance name.

However, it does not raise any errors. It can not show any info when I type show collections(already create many collections).

This makes me confused: It does not work, but there is no error happened.the

Expected Behavior

Raise an error when the config is wrong.

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

What would you like to be added?

Add command for release-dropped-collection

Why is this needed?

Fix querycoord cannot start if collection is dropped during querycoord down time.

Anything else?

No response

What would you like to be added?

Add dry-mode to switch/select milvus instance

Why is this needed?

If user cannot determine rootPath of milvus, birdwatcher could provided help to select some possible rootPath.

Anything else?

No response

What would you like to be added?

support to check loaded collection

Why is this needed?

if a cluster maintains many collections, it's important to see which one is loaded.

Anything else?

No response

What would you like to be added?

support v2.2 segment binlog meta format

Why is this needed?

in Milvus 2.2, segment binlogs meta stores in different path aside segment info. Birdwatcher shall be able to read them as well.

Anything else?

No response

Current Behavior

Only first level command has auto complete feature

Expected Behavior

all level command shall support auto complete

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

Is your feature request related to a problem? Please describe.

I can't re-run command just by traveling through the history

Describe the solution you'd like.

hint based on history

and

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

Current Behavior

show segments command prints start pos info for dml pos

Expected Behavior

show segments shall print dml position info correctly

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

What would you like to be added?

Add audit log for any delete/put operation

Why is this needed?

For the case of mistake operation we shall have an final resolve for revert the change.

Anything else?

No response

Current Behavior

instance restart, GetPartitions interface get null value for collection whose partition is not null

Expected Behavior

No response

Steps To Reproduce

No response

Environment

No response

Anything else?

@bigsheeper

Current Behavior

Field list is empty since collection schema is separated into several kvs.

Expected Behavior

Show collection could print correct field schema

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

Current Behavior

Expected Behavior

No response

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

What would you like to be added?

supported dynamic change log level

Why is this needed?

in some use case, we can get debug log without restart system.

Anything else?

No response

Current Behavior

Show collections output doesn't end with a new line (without pager)

Expected Behavior

No response

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

What would you like to be added?

Add command to verify oss files and etcd meta matches

Why is this needed?

To verify oss storage and meta integrity and detect error fast

Anything else?

No response

Current Behavior

load-backup command prints false alarm error message even when load succeeds

/tmp/birdwatcher2038238499/.bw_project not exist, writing a new one/tmp/birdwatcher2038238499/.bw_project cannot setup as workspace meta, err: stat /tmp/birdwatcher2038238499/.bw_project: no such file or directory

Expected Behavior

Just print no workspace config found

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

What would you like to be added?

1. Add a functionality to check if segment loaded on querynode, leaderview and querycoord memory state is consistent.
2. if not consistent and for some reason stucked, need to be abled to force release/load on a querynode.

Why is this needed?

This would be very helpful to verify cluster state and fix if it has some issue.

Anything else?

No response

What would you like to be added?

When you execute a show segments, you'll have a nice output for the segment count like the belowing:

It would be much better to have the count output for show collections , show collection-loaded

Why is this needed?

It's hard to count by yourself.

Anything else?

No response

Current Behavior

show segment may timeout if segment number is huge for Milvus2.2

Expected Behavior

Birdwatcher shall display info even segment number is huge

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

What would you like to be added?

Adapt Milvus 2.2 channel checkpoint

Why is this needed?

Since Milvus 2.2 uses channel checkpoint, birdwatcher shall analysis them when showing checkpoints for collections

Anything else?

No response

Current Behavior

List segment will list extra info from statslog fields

Expected Behavior

show segment command shall only return segment items

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

we can use milvus Go SDK to get internal states of milvus, such as show loaded segments of QueryNode, show channel state of DataNode, index building progress etc.

Is your feature request related to a problem? Please describe.

When there are a lot of collections, I want to see the details of a collection by a collection name I know. Now I need to "show collections" and look for them one by one.

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

What would you like to be added?

Add unit tests for all commands

Why is this needed?

Bird watcher need unit test to keep constraints met

Anything else?

No response

Current Behavior

When I connected etcd deploied by docker-compose and run "show-log-level",I got the following error:

How do I fix it?
Look forwarding the applying~

Expected Behavior

No response

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

What would you like to be added?

Add a reusable workspace feature to avoid repeatably reload backup file for analysis

Why is this needed?

Some backup file maybe large and would be slow to reload from time to time.

Anything else?

No response

What would you like to be added?

Adapt segment index information for milvus 2.2 meta

Why is this needed?

No response

Anything else?

No response

What would you like to be added?

Add the function of modifying the metadata of ectd

Why is this needed?

Lack of tools to modify data in etcd

Anything else?

No response

What would you like to be added?

make it easy and possible to do performance analysis and tuning base on birdwatcher!!!

Why is this needed?

No response

Anything else?

No response

Current Behavior

checkpoint might be not accurate since dropped segment is not excluded

Expected Behavior

checkpoint shall only consider healthy segments

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

What would you like to be added?

Add output pager for each command execution

Why is this needed?

The output for one command(say show segment) may be too long to be printed in one screen and hard to inspect
Adding a pager will let user has control when browsing the whole output.

Anything else?

No response

Current Behavior

cannot enter dry mode since pingEtcd check fails

Expected Behavior

Dry mode shall by-pass etcd ping check

Steps To Reproduce

connect to etcd with bad rootPath in dry mode

Environment

No response

Anything else?

No response

What would you like to be added?

show segment according to segment state, eg Flushed, Sealed, Dropped

Why is this needed?

In the case where a large number of segments exist in the cluster, show segment order may return too many lines at a time including all possible status of segments like Sealed, Flushed, Dropped. If the user just want to get segments in specific status, this function would be useful

Anything else?

No response

Current Behavior

• The command doesn't specify the version
• The go.mod contains a replace directive, which is not supported by go install

Expected Behavior

No response

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

Current Behavior

force release cannot work with 2.2.x milvus

Expected Behavior

Force release could handle querycoord v2 meta as well

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

What would you like to be added?

Support one-line-command mode

Why is this needed?

Support one-line-command will make Birdwatcher easier to integrate with other devops toolchain

Anything else?

No response

Is your feature request related to a problem? Please describe.

When I'm at the first/last item in the completion hints menu, and try to navigate up/down, the focus just lost, and and no easy way to do "navigate to the last item"

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

Current Behavior

No error reported when connect to an invalid etcd address.
Birdwatcher will only report error when try to execute actual command, e.g. show.

Expected Behavior

Fail fast when user provided an invalided ETCD address.

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

Current Behavior

When output is too large press "q" for pager may block forever

Expected Behavior

Pager shall not block birdwatcher

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

Is your feature request related to a problem? Please describe.

Many users have encountered the problem of excessive storage consumption by object storages such as MinIO/S3. Adding a command to analyze the usage of object storage, including the following functions:

1. View the size of each collection classified by collection.
2. Classify according to segment and mark which files are dropped segments that can be recycled.
3. Able to analyze the sizes of insert_log, delta_log, and stats_log.

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

Current Behavior

I have go version go1.19.3 installed.
I have run the below commands:
git clone https://github.com/milvus-io/birdwatcher
cd birdwatcher
go build -o birdwatcher main.go
mv -v birdwatcher /usr/local/bin

Above step produces a excutable file with a name of birdwatcher
I run birdwatcher in bash session, the session enter command mode while displaying "Offline >", then I type "exit" to go back to bash session, now when I type in bash session, nothing will display, but if I accidently type something recognized as system command, it will run and display the result.

Expected Behavior

Bash session should display when typing.

Steps To Reproduce

git clone https://github.com/milvus-io/birdwatcher
cd birdwatcher
go build -o birdwatcher main.go
mv -v birdwatcher /usr/local/bin
./birdwatcher
Offline > exit

Environment

Ubuntu 20.04.5 LTS
Ubuntu 22.04.5 LTS

Anything else?

No response

What would you like to be added?

Add probe Milvus function

Why is this needed?

Health check may not be accurate for Milvus system
Birdwatcher can utilize internal api to probe certain functionality of system

Anything else?

No response

Is your feature request related to a problem? Please describe.

There is no convenient way to parse and analysis Milvus index files(binlog tool might help with segment binlog)

Describe the solution you'd like.

Birdwatcher provides commands to analysis index files

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response