Comments (9)
hi,
yes attu was doing deletes on my test data, when i started to add real data and it became around 65k , i figured some meta fields were added as "undefined" , when i tried to delete the records with data as undefined ,records were not deleting , i realized that these records were around 18K , with python i could delete 8k in one go , by finding the ids first then deleting , but i guess as the field value was "undefined" JavaScript might be interpreting it differently while using delete entities?, i will test further and let you know , thanks for your reply.
consistency level is default i am just starting to use milvus!
from milvus-sdk-node.
otherwise the id will be changed by node system. I have an issue for that, but I don't have time to give a better solution. so, please use string id if autoID
is enabled.
from milvus-sdk-node.
Hi there,
Since attu is using the same node sdk, I tested deletion on attu, it works well.
One question: What's the consistency level of your collection?
from milvus-sdk-node.
Hi,
I've started using Milvus recently and stumbled upon the same issue that deleting entities don't seem to work. After some research I also tried a recommended approach of creating the collection with a consistency level of Strong, but it still didn't solve the issue.
All other operations (i.e. creating collections, querying data) could be successfully carried out except for delete so far.
I tried both the delete
as well as the deleteEntities
method as described in the Milvus docs but neither one yielded a result.
In fact the operation runs without errors and I also get an appropriate response including the delete_cnt which is set to the number of entities that should have been removed. But upon querying the collection after some time, the entities still exist.
Am I missing some configurations?
Steps to reproduce:
- create a collection
- add data (using LangChain JS and OpenAI Embeddings API)
- query data and obtain primaryIds
- delete entities for given primaryIds (succeeds without error)
- query again and same data is displayed as if they weren't deleted
Milvus Node SDK version: 2.3.1
Milvus version (standalone docker): 2.3.0
from milvus-sdk-node.
@shanghaikid could you please explain this behavior?
from milvus-sdk-node.
Could you try this file https://github.com/milvus-io/milvus-sdk-node/blob/main/examples/milvus/DataQuery.ts
On my testing, If I set the consistency_level
to Strong
, I can get the correct result.
from milvus-sdk-node.
from milvus-sdk-node.
@shanghaikid thank you for providing the example file.
I've done some tests and also changed the code to rather simulate the conditions I use in my own project.
Generally the code works fine and the selected entities are also deleted from the collection. But as soon as I use the autoID
feature on the id field, then the delete doesn't seem to work anymore and the subsequent query shows all 6 entities.
Here's my modified version of the code:
import { MilvusClient, InsertReq, DataType } from '@zilliz/milvus2-sdk-node';
const COLLECTION_NAME = 'data_query_example';
(async () => {
// build client
const milvusClient = new MilvusClient({
address: 'localhost:19530',
username: 'username',
password: 'Aa12345!!',
});
console.log('Node client is initialized.');
// create collection
const create = await milvusClient.createCollection({
collection_name: COLLECTION_NAME,
consistency_level: 'Strong',
fields: [
{
name: 'id',
description: 'ID field',
data_type: DataType.Int64,
is_primary_key: true,
autoID: true,
},
{
name: 'vector',
description: 'Vector field',
data_type: DataType.FloatVector,
dim: 8,
},
{ name: 'height', description: 'int64 field', data_type: DataType.Int64 },
{
name: 'source',
description: 'VarChar field',
data_type: DataType.VarChar,
max_length: 128,
},
],
});
console.log('Create collection is finished.', create);
// build example data
const vectorsData = [
{
vector: [
0.11878310581111173, 0.9694947902934701, 0.16443679307243175,
0.5484226189097237, 0.9839246709011924, 0.5178387104937776,
0.8716926129208069, 0.5616972243831446,
],
height: 20405,
source: 'page1',
// id: 1,
},
{
vector: [
0.9992090731236536, 0.8248790611809487, 0.8660083940881405,
0.09946359318481224, 0.6790698063908669, 0.5013786801063624,
0.795311915725105, 0.9183033261617566,
],
height: 93773,
source: 'page1',
// id: 2,
},
{
vector: [
0.8761291569818763, 0.07127366044153227, 0.775648976160332,
0.5619757601304878, 0.6076543120476996, 0.8373907516027586,
0.8556140171597648, 0.4043893119391049,
],
height: 85122,
source: 'page2',
// id: 3,
},
{
vector: [
0.5849602436079879, 0.5108258101682586, 0.8250884731578105,
0.7996354835509332, 0.8207766774911736, 0.38133662902290566,
0.7576720055508186, 0.4393152967662368,
],
height: 92037,
source: 'page2',
// id: 4,
},
{
vector: [
0.3768133716738886, 0.3823259261020866, 0.7906232829855262,
0.31693696726284193, 0.3731715403499176, 0.3300751870649885,
0.22353556137796238, 0.38062799545615444,
],
height: 31400,
source: 'page3',
// id: 5,
},
{
vector: [
0.0007531778212483964, 0.12941566118774994, 0.9340164428788116,
0.3795768837758642, 0.4532443258064389, 0.596455163143,
0.9529469158782906, 0.7692465408044873,
],
height: 1778,
source: 'page3',
// id: 6,
},
];
const params: InsertReq = {
collection_name: COLLECTION_NAME,
fields_data: vectorsData,
};
// insert data into collection
await milvusClient.insert(params);
console.log('Data is inserted.');
// create index
const indexCreateParams = {
index_type: 'HNSW',
metric_type: 'L2',
params: JSON.stringify({ M: 8, efConstruction: 64 }),
};
const createIndex = await milvusClient.createIndex({
collection_name: COLLECTION_NAME,
field_name: 'vector',
extra_params: indexCreateParams,
});
console.log('Index is created', createIndex);
// need load collection before search
const load = await milvusClient.loadCollectionSync({
collection_name: COLLECTION_NAME,
});
console.log('Collection is loaded.', load);
// do the query
console.time('Query time');
const query = await milvusClient.query({
collection_name: COLLECTION_NAME,
filter: 'source == "page1"',
output_fields: ['id', 'source', 'vector'],
limit: 100,
});
console.timeEnd('Query time');
console.log('query result', query);
// delete data
const { data } = query;
const ids: number[] = data.map((entity) => +entity['id']);
const del = await milvusClient.delete({
collection_name: COLLECTION_NAME,
ids,
});
console.log('del', del);
// do the query
console.time('Query after del');
const queryAfterDel = await milvusClient.query({
collection_name: COLLECTION_NAME,
filter: 'id > 0',
output_fields: ['id', 'source', 'vector'],
limit: 100,
});
console.timeEnd('Query after del');
console.log('query after del', queryAfterDel);
// drop collection
await milvusClient.dropCollection({
collection_name: COLLECTION_NAME,
});
})();
Setting autoID
to false and uncommenting the ids to use fixed integer values will work as I said.
I had the assumption, that maybe the autoID feature affects the way the generated ids behave, but the type is still set to Int64, so I have no clues why this shouldn't work!
Would you please look into this issue?
from milvus-sdk-node.
@sadwsh the type of id is int64, which is not supported in node.
const ids: number[] = data.map((entity) => +entity['id']);
You can use string type for ids array. I tested your code after change this part, everything works.
const ids: string[] = data.map((entity) => entity['id']);
from milvus-sdk-node.
Related Issues (20)
- useDatabase is not available HOT 1
- getVersion Should be removed from API? HOT 1
- Ignore Growing Segment
- Provide a count function HOT 4
- Can not work on `grpc/grpc-js: 1.8.19` default dependency HOT 3
- `milvusClient.delete()` can not delete entities when the primary key is string.
- Fail to run langchain milvus vector store sample HOT 9
- Lack of error handling on invalid collection names
- output_fields error
- search api only return `score` and `id` HOT 7
- It should handle int64 properly
- support missing AlterCollection
- failed to get channels, collection not loaded HOT 3
- MilvusClient instantiation is not catchable HOT 1
- The dynamicSchema in the example is OK? HOT 2
- Please support search for multiple vectors HOT 2
- [Langchain] - How to set collection vector dimension? HOT 2
- Build a pool of connections <> Check if the connection is dead to revive it. HOT 1
- Error: ENOENT: no such file or directory, open '/admin/.next/server/app/api/proto/proto/schema.proto HOT 13
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from milvus-sdk-node.