Fails to insert value into table with a 2 nodes cluster sometimes about scylladb

[slivne]

@asias when you can please check if "[PATCH] Fix a race condition in node state for bootstrapped nodes" solves your issue

[asias]

No, it does not help.

On Tue, Jul 21, 2015 at 8:33 PM, slivne [email protected] wrote:

@asias https://github.com/asias when you can please check if "[PATCH]
Fix a race condition in node state for bootstrapped nodes" solves your issue

—
Reply to this email directly or view it on GitHub
#30 (comment)
.

Asias

[vladzcloudius]

What happens above is that one can't assume that keyspace/table is immediately available on a Node2 once CREATE command has successfully completed on Node1. Gossiper will eventually (!!!) propagate this information to all other nodes but it may take some time. As a result the above test periodically hits the "column_family not found" case and as a result the corresponding mutation times out.

Adding sleep(3) after creating the keyspace and another one after creating a table "resolves" the issue.

[penberg]

@asias I sent a patch titled "service/migration_manager: Fix announce() delay" on the mailing list that fixes the issue for me.

[penberg]

@asias Never mind, the patch was bogus. Still investigating...

[asias]

On Thu, Jul 23, 2015 at 5:33 PM, Pekka Enberg [email protected]
wrote:

@asias https://github.com/asias Never mind, the patch was bogus. Still
investigating...

sure.

—
Reply to this email directly or view it on GitHub
#30 (comment)
.

Asias

[penberg]

@asias, the cases I've seen with dtest is either RPC starting to fail or other nodes not actually joined the cluster when we attempt to push out the schema changes. What does gms::get_live_members() return in migration_manager::announce() when the test fails?

[asias]

On Thu, Jul 23, 2015 at 8:36 PM, Pekka Enberg [email protected]
wrote:

@asias https://github.com/asias, the cases I've seen with dtest is
either RPC starting to fail or other nodes not actually joined the cluster
when we attempt to push out the schema changes. What does
gms::get_live_members() return in migration_manager::announce() when the
test fails?

Do you want to see this debug info when I see insert fails?

0 // Returns a future on the local application of the schema
1 future<>
migration_manager::announce(distributedservice::storage_proxy& proxy,
std::vector schema)
2 {
3 return gms::get_live_members().then([&proxy, schema =
std::move(schema)](std::setgms::inet_address live_members) {
4 for (auto& x : live_members) {
5 print("====>>>> migration_manager::announce %s\n", x);
6 }

Asias

[penberg]

Yes. I assume the other node is not there when the test fails.

[asias]

On Thu, Jul 23, 2015 at 9:04 PM, Pekka Enberg [email protected]
wrote:

Yes. I assume the other node is not there when the test fails.

Here, the test fails (key 3-6 are not inserted), but node 1 knows node 2.

================ STREAM_PLAN TEST ==============
WARNING: Not implemented: AUTH
WARNING: Not implemented: METRICS
WARNING: Not implemented: PERMISSIONS
====>>>> migration_manager::announce 127.0.0.1
====>>>> migration_manager::announce 127.0.0.2
CREATE KEYSPACE = KS DONE
====>>>> migration_manager::announce 127.0.0.1
====>>>> migration_manager::announce 127.0.0.2
CREATE TABLE = TB DONE
WARNING: Not implemented: TRIGGERS
INSERT VALUE DONE: 1
INSERT VALUE DONE: 2

—
Reply to this email directly or view it on GitHub
#30 (comment)
.

Asias

[penberg]

@asias AFAICT, Cassandra drivers issue a schema refresh on other nodes after a schema change. As you're doing schema changes directly, the expected behavior is exactly what you're seeing.

[asias]

On Tue, Jul 28, 2015 at 4:27 PM, Pekka Enberg [email protected]
wrote:

@asias https://github.com/asias AFAICT, Cassandra drivers issue a
schema refresh on other nodes after a schema change. As you're doing schema
changes directly, the expected behavior is exactly what you're seeing.

Can we issue such schema refresh in my test code after schema change?

—
Reply to this email directly or view it on GitHub
#30 (comment)
.

Asias

[penberg]

I am completely not sure how the driver does the refresh. Once we figure that out, we can certainly do the same thing in test code.