error: multiple rlib candidates for `kafka` found about kafka-rust

[spicavigo]

I think you added kafka dependency to kafka library itself. i.e. you added kafka="*" to the cargo.toml of kafka-rust library.

To test the library out, you should

1. cd /Users/admin/myrustdemo
2. cargo new kafkademo --bin
3. cd kafkademo
4. cp ../kafka-rust-master/src/main.rs src/
5. Add kafa dep to cargo.toml
6. cargo build

Let me know if this doesn't work

[Lingling7]

Thank you so much! I followed all your steps, and add [dependencies] kafka = "*" in cargo.toml of kafkademo, I ran into this error

    Updating registry `https://github.com/rust-lang/crates.io-index`
   Compiling rustc-serialize v0.3.16
   Compiling winapi-build v0.1.1
   Compiling winapi v0.2.4
   Compiling byteorder v0.3.13
   Compiling libc v0.1.10
   Compiling gcc v0.3.17
   Compiling advapi32-sys v0.1.2
   Compiling rand v0.3.11
   Compiling miniz-sys v0.1.6
   Compiling flate2 v0.2.9
   Compiling num v0.1.27
   Compiling kafka v0.1.6
   Compiling kafkademo v0.1.0 (file:///Users/admin/myrustdemo/kafkademo)
src/main.rs:5:39: 5:74 error: mismatched types:
 expected `collections::vec::Vec<collections::string::String>`,
    found `&collections::vec::Vec<collections::string::String>`
(expected struct `collections::vec::Vec`,
    found &-ptr) [E0308]
src/main.rs:5     let mut client = KafkaClient::new(&vec!("localhost:9092".to_string()));
                                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/main.rs:5:39: 5:74 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to previous error
Could not compile `kafkademo`.

is that means I need to delete &? (As what I saw in soulfly's post)
but I ran into a new question

   Compiling kafkademo v0.1.0 (file:///Users/admin/myrustdemo/kafkademo)
src/main.rs:6:5: 6:32 warning: unused result which must be used, #[warn(unused_must_use)] on by default
src/main.rs:6     client.load_metadata_all();
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~
error: linking with `cc` failed: exit code: 1
note: "cc" "-m64" "-L" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib" "/Users/admin/myrustdemo/kafkademo/target/debug/kafkademo.0.o" "-o" "/Users/admin/myrustdemo/kafkademo/target/debug/kafkademo" "-Wl,-force_load,/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libmorestack.a" "-Wl,-dead_strip" "-nodefaultlibs" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libkafka-73c7ae9da4235a7d.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libflate2-98f57e2f7af2d432.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libnum-397f282c1d72fe58.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/librustc_serialize-7ff5bfc027146194.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/librand-bdfcc55d3466feb3.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libminiz_sys-fa48ce1a538dca79.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/liblibc-144c435538abd757.rlib" "/Users/admin/myrustdemo/kafkademo/target/debug/deps/libbyteorder-3e27c88aa235985f.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libstd-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libcollections-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/librustc_unicode-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/librand-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/liballoc-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/liblibc-198068b3.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libcore-198068b3.rlib" "-L" "/Users/admin/myrustdemo/kafkademo/target/debug" "-L" "/Users/admin/myrustdemo/kafkademo/target/debug/deps" "-L" "/Users/admin/myrustdemo/kafkademo/target/debug/build/miniz-sys-fa48ce1a538dca79/out" "-L" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib" "-L" "/Users/admin/myrustdemo/kafkademo/.rust/lib/x86_64-apple-darwin" "-L" "/Users/admin/myrustdemo/kafkademo/lib/x86_64-apple-darwin" "-l" "snappy" "-l" "c" "-l" "m" "-l" "System" "-l" "pthread" "-l" "c" "-l" "m" "-l" "compiler-rt"
note: ld: warning: directory not found for option '-L/Users/admin/myrustdemo/kafkademo/.rust/lib/x86_64-apple-darwin'
ld: warning: directory not found for option '-L/Users/admin/myrustdemo/kafkademo/lib/x86_64-apple-darwin'
ld: library not found for -lsnappy
clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: aborting due to previous error
Could not compile `kafkademo`.

[xitep]

hello,

yes ... that & is currently a misleading point in the documentation (see #18) and will be resolved soon (i hope).

regarding the linking error: i think you'll need to have the native snappy library installed (don't know how to do this on windows, but on unix like systems it'd expect you to have /usr/lib/libsnappy.so.1.1.3 or similar).

[Lingling7]

It works, thanks!!!

[Lingling7]

this is what I put in main.rs now

extern crate kafka;
use kafka::utils;
use kafka::client::KafkaClient;
fn main() {
    let mut client = KafkaClient::new(vec!("localhost:9092".to_string()));
    //client.load_metadata_all();
    // OR
    client.load_metadata(vec!("rocksdbtest".to_string())); // Loads metadata for vector of topics
    let con = kafka::consumer::Consumer::new(client, "test-group".to_string(), "rocksdbtest".to_string())
             .partition(0);
    for msg in con {
        println!("{:?}", msg);
    }
 }

this is the warning when I type cargo build

src/main.rs:4 use kafka::utils;
                  ^~~~~~~~~~~~
src/main.rs:9:5: 9:59 warning: unused result which must be used, #[warn(unused_must_use)] on by default
src/main.rs:9     client.load_metadata(vec!("rocksdbtest".to_string())); // Loads metadata for vector of topics
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I have .exe now, but I cannot see anything when I ran cargo run, are there anything I miss?

This is the tree of kafademo

.
├── Cargo.lock
├── Cargo.toml
├── src
│   ├── consumer.rs
│   └── main.rs
└── target
    └── debug
        ├── build
        │   ├── advapi32-sys-cfef7a1f30f1e5f6
        │   │   ├── build-script-build
        │   │   ├── build_script_build.dSYM
        │   │   │   └── Contents
        │   │   │       ├── Info.plist
        │   │   │       └── Resources
        │   │   │           └── DWARF
        │   │   │               └── build_script_build
        │   │   ├── out
        │   │   └── output
        │   └── miniz-sys-fa48ce1a538dca79
        │       ├── build-script-build
        │       ├── build_script_build.dSYM
        │       │   └── Contents
        │       │       ├── Info.plist
        │       │       └── Resources
        │       │           └── DWARF
        │       │               └── build_script_build
        │       ├── out
        │       │   ├── libminiz.a
        │       │   └── miniz.o
        │       └── output
        ├── deps
        │   ├── libadvapi32-cfef7a1f30f1e5f6.rlib
        │   ├── libbuild-304afb6bdff23d72.rlib
        │   ├── libbyteorder-3e27c88aa235985f.rlib
        │   ├── libflate2-98f57e2f7af2d432.rlib
        │   ├── libgcc-f53b41fe21770ed8.rlib
        │   ├── libkafka-73c7ae9da4235a7d.rlib
        │   ├── liblibc-144c435538abd757.rlib
        │   ├── libminiz_sys-fa48ce1a538dca79.rlib
        │   ├── libnum-397f282c1d72fe58.rlib
        │   ├── librand-bdfcc55d3466feb3.rlib
        │   ├── librustc_serialize-7ff5bfc027146194.rlib
        │   └── libwinapi-21b078e9a1931364.rlib
        ├── examples
        ├── kafkademo
        ├── kafkademo.dSYM
        │   └── Contents
        │       ├── Info.plist
        │       └── Resources
        │           └── DWARF
        │               └── kafkademo
        └── native

[xitep]

i think we need to improve the documentation a little bit. in your particular case, however ...

a)
... you can get rid of the warnings by 1) deleting that unused use kafka::utils and 2) capturing the result of the call to client.load_metadata. The quick and dirty way could be just something like let _ = client.load_metadata(...) (here you merely capture the result into a variable named _ and that makes the compiler happy). The correct way to deal with the result value would be to see whether load_metadata method resulted in an error or not. A good resource of this topic (error handling) is surely the rust book and Andrew Gallant's "Error Handling in Rust".

b)
regarding the fact that your program doesn't print anything, i would suspect that the topic rocksdbtest does not exist in your kafka broker. the consumer silently skips requests for non-existing topics. the topic is the third argument in your call to kafka::consumer::Consumer::new. I would suggest to verify it exists.

[Lingling7]

Thanks for the very detail info, xitep!!!
I can use fetch_messages and print out now!!!^_^
But still cannot use the consumer, probably the "test-group", I use the default name in consumer.properties.java "test-consumer-group".
This is just test on the local, and I only build one topic "test"

extern crate kafka;
use kafka::client::KafkaClient;
use kafka::consumer::Consumer;
//use kafka::utils;
fn main() {
    // 1. Connect to Kafka
    let mut client = KafkaClient::new(vec!("localhost:9092".to_owned()));

    let kafka_metadata =client.load_metadata(vec!("test".to_owned())); 


    // client.load_metadata(vec!("my-topic".to_owned())); // 

    // 2. fetch messages: single (Topic, Partition, Offset)
    let offsetmsg= client.fetch_messages("test".to_owned(), 0, 0);
    println!("offsetmsg: {:?}", offsetmsg);
    //3. consumer

    let con = Consumer::new(client, "test-consumer-group".to_owned(), "test".to_owned()).partition(0);
    for msgs in con {
       println!("{:?}", msgs);
    }
}

This is partial result

offsetmsg: Ok([TopicMessage { topic: "test", partition: 0, offset: 0, error: None, message: [91] }, TopicMessage { topic: "test", partition: 0, offset: 1, error: None, message: [9, 123] },...TopicMessage { topic: "test", partition: 0, offset: 155, error: None, message: [93] }])

No result form the consumer.

[spicavigo]

@Lingling7 Could you fetch the offsets for this consumer and verify that its less than the greatest offset for the partition you are consuming?

[Lingling7]

@spicavigo Do you mean check the offset numbers (id:0-155 ) with the consumed messages numbers(as below, 156)?

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Consumed 156 messages

In addition, the info in "messages: [91] " (mylast comment) is not the message in kafka, messages here looks like the metadata.

[spicavigo]

@Lingling7 I mean this https://github.com/spicavigo/kafka-rust/blob/master/src/client.rs#L550

[Lingling7]

@spicavigo thanks, I add this in main.rs

let resp = client.fetch_group_topic_offset("test-consumer-group".to_owned(),"test".to_owned());
    println!("resp: {:?}", resp);

result like this
resp: Ok([TopicPartitionOffsetError { topic: "test", partition: 0, offset: -1, error: Some(UnknownTopicOrPartition) }])

[Lingling7]

I can use the consumer now, because I didn't set the offesetkey to the earliest, so I used

bin/kafka-run-class.sh kafka.tools.UpdateOffsetsInZK earliest config/consumer.properties test

But how can I set this through rust, should I set from the consumer?

For the messages form consumer, I just got the numbers, how can I get the true text messages?

msgs: TopicMessage { topic: "test", partition: 0, offset: 1, error: None, message: [32, 32, 32, 32, 125] }
msgs: TopicMessage { topic: "test", partition: 0, offset: 2, error: None, message: [109, 115, 103, 116, 101, 115, 116, 95, 50] }

And if I want to load the data to rocksdb, should I use fetch or consumer?

[Lingling7]

The numbers in message is vector of bytes (utf-8), I need to transfer them to string

use std::str