amqp-rs / lapin Goto Github PK

Hello!

I'm trying to figure out with using basic_consumer method for extracting a message, but can't . I'm trying to deal with using basic_consumer method for extracting a message. In my case, I want to implement something similar to Request / Response pattern with RabbitMQ and want to after creating a special queue for a response, extract the first incoming message (with basic_ack invoke) and close the consumer. Then I'd like to unbind the queue from the exchange, delete the response queue and close the channel on the last stage for processing a client request.

On the first steps of my project I'd like to make it in this way, because it more or less easy to understand how it works in general and keeps other server parts asynchronously. Moreover, the server processing is also divided onto small pieces and build as conveyor, which is doing something on each stage. And on the final stage some existing worker will send a response message to client.

Does it possible to write something like this and save a readability of the code?

// This part won't compile
.and_then(|channel| {
     channel.basic_consume(
         &queue_name,
         "response_consumer",
         &BasicConsumeOptions::default(),
         &FieldTable::new()
     )
        .map(|stream| {
            stream
                // get one message
                .take(1) 
                // go to the next stage and process the message
                .map(move |message| { 
                    channel.basic_ack(message.delivery_tag);
                    (channel, message)
                })
        })
})
.map_err(|err| {
    let message = format!("Error during consuming the response message: {}", err);
    error!("{}", message);
    err
})

// This part is working as expected, without any issues
// Prepare a response for a client, serialize and sent it to client
.and_then(|(channel, message)| {
    println!("Message: {:?}", message);
    // ... something useful here
    Ok(channel)
})
.map_err(|err| {
    let message = format!("Error during sending a message to the client: {}", err);
    error!("{}", message);
    err
})

Most amqp libs in different langs and also rust-amqp have this interface for publish:

channel.basic_publish(exchange='', routing_key='task_queue')

Right now lapin specifies a queue as the first argument for basic_publish, declaring the queue upfront is not useful for more complex topologies like when doing exchange to exchange routing.

I used the sample code as follows:

extern crate futures;
extern crate tokio_core;
extern crate lapin_futures as lapin;

use futures::Stream;
use futures::future::Future;
use tokio_core::reactor::Core;
use tokio_core::net::TcpStream;
use lapin::client::ConnectionOptions;
use lapin::channel::{BasicConsumeOptions,BasicPublishOptions,QueueDeclareOptions};
use lapin::types::FieldTable;

fn main() {

  // create the reactor
  let mut core = Core::new().unwrap();
  let handle = core.handle();
  let addr = "127.0.0.1:5672".parse().unwrap();

  core.run(

    TcpStream::connect(&addr, &handle).and_then(|stream| {

      // connect() returns a future of an AMQP Client
      // that resolves once the handshake is done
      lapin::client::Client::connect(stream, &ConnectionOptions::default())
    }).and_then(|(client, heartbeat_future_fn)| {
      // The heartbeat future should be run in a dedicated thread so that nothing can prevent it from
      // dispatching events on time.
      // If we ran it as part of the "main" chain of futures, we might end up not sending
      // some heartbeats if we don't poll often enough (because of some blocking task or such).
      let heartbeat_client = client.clone();
      handle.spawn(heartbeat_future_fn(&heartbeat_client).map_err(|_| ()));

      // create_channel returns a future that is resolved
      // once the channel is successfully created
      client.create_channel()
    }).and_then(|channel| {
      let id = channel.id;
      info!("created channel with id: {}", id);

      let ch = channel.clone();
      channel.queue_declare("hello", &QueueDeclareOptions::default(), &FieldTable::new()).and_then(move |_| {
        info!("channel {} declared queue {}", id, "hello");

        // basic_consume returns a future of a message
        // stream. Any time a message arrives for this consumer,
        // the for_each method would be called
        channel.basic_consume("hello", "my_consumer", &BasicConsumeOptions::default(), &FieldTable::new())
      }).and_then(|stream| {
        info!("got consumer stream");

        stream.for_each(move|message| {
          debug!("got message: {:?}", message);
          info!("decoded message: {:?}", std::str::from_utf8(&message.data).unwrap());
          ch.basic_ack(message.delivery_tag);
          Ok(())
        })
      })
    })
  ).unwrap();
}

If I manually close rabbit (testing purpose) I see this error:
ERROR 2018-02-13T16:20:24Z: lapin_futures::transport: upstream poll gave Ready(None)

The core.run() function remains blocked without any error catched, is stuck forever from this point. How I can handle this error and break the call after any type of error occurs?

paste code:

rust
extern crate lapin_async as lapin;

use lapin::buffer::Buffer;
use lapin::connection::*;
use lapin::generated::basic;

use lapin::types::*;
use std::{thread, time};
use std::io::Read;
use std::net::TcpStream;
#[test]
fn connection() {
let mut stream = TcpStream::connect("127.0.0.1:5672").unwrap();
stream.set_nonblocking(true).unwrap();

let capacity = 8192;
let mut send_buffer = Buffer::with_capacity(capacity as usize);
let mut receive_buffer = Buffer::with_capacity(capacity as usize);

let mut conn: Connection = Connection::new();
conn.set_frame_max(capacity);
assert_eq!(conn.connect().unwrap(), ConnectionState::Connecting(ConnectingState::SentProtocolHeader));
loop {
	match conn.run(&mut stream, &mut send_buffer, &mut receive_buffer) {
		Err(e) => panic!("could not connect: {:?}", e),
		Ok(ConnectionState::Connected) => break,
		Ok(state) => println!("now at state {:?}, continue", state),
	}

	thread::sleep(time::Duration::from_millis(100));
}
println!("CONNECTED");
println!("conn.configuration.frame_max = {}", conn.configuration.frame_max);

//now connected
let frame_max = conn.configuration.frame_max;
if frame_max > capacity {
	send_buffer.grow(frame_max as usize);
	receive_buffer.grow(frame_max as usize);
}

let channel_a: u16 = conn.create_channel();
let channel_b: u16 = conn.create_channel();

//send channel
conn.channel_open(channel_a, "".to_string()).expect("channel_open");
println!("channel [{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());

thread::sleep(time::Duration::from_millis(100));
println!("channel [{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());

//receive channel
conn.channel_open(channel_b, "".to_string()).expect("channel_open");
println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());
thread::sleep(time::Duration::from_millis(100));
println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());

//create the hello queue
conn.queue_declare(channel_a, 0, "hello-async".to_string(), false, false, false, false, false, FieldTable::new())
    .expect("queue_declare");
println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());
thread::sleep(time::Duration::from_millis(100));
println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());


//purge the hello queue in case it already exists with contents in it
conn.queue_purge(channel_a, 0, "hello-async".to_string(), false).expect("queue_purge");
println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());
thread::sleep(time::Duration::from_millis(100));

println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());
conn.queue_declare(channel_b, 0, "hello-async".to_string(), false, false, false, false, false, FieldTable::new())
    .expect("queue_declare");

println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());
thread::sleep(time::Duration::from_millis(100));
println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());


println!("will consume");
conn.basic_consume(channel_b, 0, "hello-async".to_string(), "my_consumer".to_string(), false, true, false, false, FieldTable::new())
    .expect("basic_consume");

println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());
thread::sleep(time::Duration::from_millis(100));

//println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());

println!("will publish");
conn.basic_publish(channel_a, 0, "".to_string(), "hello-async".to_string(), false, false)
    .expect("basic_publish");
let mut data = b"Hello world!";
let mut payload: Vec<u8> = vec![];
payload.resize(8888, 65);
payload.push(12);
payload.append(&mut data.to_vec());
send_buffer.grow(payload.len());
conn.send_content_frames(channel_a, 60, &payload, basic::Properties::default());
println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());

thread::sleep(time::Duration::from_millis(100));
println!("[{}] state: {:?}", line!(), conn.run(&mut stream, &mut send_buffer, &mut receive_buffer).unwrap());
let msg = conn.next_message(channel_b, "hello-async", "my_consumer").unwrap();

println!("received message: {:?}", msg);
println!("data: {}", std::str::from_utf8(&msg.data).unwrap());

}

my code:
rust
let mut data = b"Hello world!";
let mut payload: Vec = vec![];
payload.resize(8888, 65);
payload.push(12);
payload.append(&mut data.to_vec());
send_buffer.grow(payload.len());
conn.send_content_frames(channel_a, 60, &payload, basic::Properties::default());

The future returned by lapin::client::Client::connect doesn't always complete.

In some instances when running something like the following code, the application will just stall.

core.run(
    TcpStream::connect(&addr, &handle).and_then(|stream| {
        lapin::client::Client::connect(stream, &ConnectionOptions::default())
    })
)

log file: https://www.dropbox.com/s/gti4kf327vfepib/log.html?dl=0

Hey, I am thinking about leveraging this crate, but right now it is hard (without compiling the code) to follow where the breaking changes are and what new features are available and so forth. Accordingly, I think a changelog could help with the individuals following this project to get a quick update on the new versions that are getting published.

Hi!

I'm using latest master, and I randomly see failed to handle frame: NotConnected before any messages are received using what is practically the futures example. I see absolutely nothing in the debug logs beyond this:

 INFO  my_project                > Starting consumer with id 2...
 DEBUG tokio_reactor             > loop process - 1 events, 0.000s
 DEBUG tokio_reactor             > loop process - 1 events, 0.000s
 DEBUG tokio_reactor             > loop process - 1 events, 0.000s
 DEBUG tokio_reactor             > loop process - 1 events, 0.000s
 INFO  my_project                > got consumer stream
 DEBUG tokio_reactor             > loop process - 1 events, 0.000s
 DEBUG tokio_reactor             > loop process - 1 events, 0.000s
thread 'tokio-runtime-worker-0' panicked at 'failed to handle frame: NotConnected', src/main.rs:114:36
note: Run with `RUST_BACKTRACE=1` for a backtrace.
 DEBUG tokio_reactor             > dropping I/O source: 0
 DEBUG tokio_reactor             > loop process - 1 events, 0.000s
 DEBUG tokio_reactor::background > shutting background reactor on idle
 DEBUG tokio_reactor::background > background reactor has shutdown

Any ideas what might be doing this? I used my client to create queues, etc, so it appears that the channels should work fine. Happy to debug as needed!

I'm trying to combine tokio-postgres with lapin-futures(version 0.8.2, using RMQ 3.6.9), the use case is listening to postgres notifications and publishing to a RMQ queue with lapin, this example works:

fn main() {
  env_logger::init().unwrap();
  let mut core = Core::new().unwrap();
  let handle = core.handle();

  let addr = "127.0.0.1:5672".parse().unwrap();

  core.run(
    TcpStream::connect(&addr, &handle)
      .and_then(|stream| lapin::client::Client::connect(stream, &ConnectionOptions::default()))
      .and_then(|client| client.create_channel())
      .and_then(|channel| {
        channel.queue_declare("hello", &QueueDeclareOptions::default(), FieldTable::new()).and_then(|_| {
          Connection::connect("postgres://postgres@localhost", TlsMode::None, &handle)
          .then(|conn| conn.unwrap().batch_execute("LISTEN pgchannel1").map_err(|_|Error::new(ErrorKind::Other, "")))
          .and_then(|conn|
            conn.notifications().map_err(|_|Error::new(ErrorKind::Other, "")).for_each(move |n|{
              println!("Publishing payload: {}", n.payload);
              channel.basic_publish(
                  "hello",
                  n.payload.as_bytes(),
                  &BasicPublishOptions::default(),
                  BasicProperties::default()
              );
              Ok(())
            })
          )
        })
      })
  ).unwrap();
}

But after 3 mins of being idle, the RMQ tcp connection is dropped(disappears from RMQ management web interface), and when trying to send another notification to the queue the following message is shown:

ERROR:lapin_futures::transport: Failed to send frame: Error { repr: Os { code: 104, message: "Connection reset by peer" } }
ERROR:lapin_futures::transport: Failed to send heartbeat: Error { repr: Os { code: 32, message: "Broken pipe" } }
ERROR:lapin_futures::transport: Failed to send frame: Error { repr: Os { code: 32, message: "Broken pipe" } }
ERROR:lapin_futures::transport: Failed to send frame: Error { repr: Os { code: 32, message: "Broken pipe" } }
ERROR:lapin_futures::transport: upstream poll gave Ready(None)
ERROR:lapin_futures::transport: Failed to send frame: Error { repr: Os { code: 32, message: "Broken pipe" } }
ERROR:lapin_futures::transport: upstream poll gave Ready(None)

I've checked and the 3 mins are because of 3 failed heartbeats(60s each one, same error can be obtained faster by reducing heartbeat).

How can I keep the lapin connection alive while consuming another stream? Any pointers would help. Thanks.

CloudAMQP and similar hosted versions of RabbitMQ use a vhost for multi-tenant environments.

When trying out lapin, I think the lack of vhost support was the only thing "technically" keeping me from establishing a connection.

I think all of the major outstanding bugs have been fixed

@kureuil @whitfin @jeffparsons you either contributed or were affected by bugs that should be fixed in current master. Can you give it a try and see if you notice anything that could block the release?

Carrying the actual state and the expected state would help debugging a lot

Hello,

I'm trying to publish a message via using basic_publish method of a channel, but can't figure out how to ack the already sent message (and retry a few times it if will be necessary).

The basic_publish method contains inside a small piece of code which is intended for desribing a state of the message, but used it only one time. Also, this method returns an Option as a result of sending a message, but doesn't returning any additional information about the message (for our case is delivery tag).

The following future I'm using for sending a message:

channel.basic_publish(
    &endpoint_publish.get_request_exchange(),
    &endpoint_publish.get_microservice(),
    message["content"].dump().as_bytes(),
    &publish_message_options,
    basic_properties
)
    .map(|_confirmation| channel)  // <-- return a channel for a further using

Current output in a console:

Listening on: 0.0.0.0:9000
[2018-02-08][20:57:32][lapin_futures::channel][INFO] message with tag 1 still in unacked: {1}

We have a server running on tokio-threadpool, where we keep all the connections in thread local variables. With Kafka and Aerospike this works quite nicely, but I've been now fighting with lapin to exit the heartbeat future cleanly. Keep in mind, while using thread_local! I don't really have an easy ownership to a oneshot channel, so I've been banging my head against the wall for the last couple of days. Here's one try with tokio_signal:

let _hb_thread: JoinHandle<()> = {
    let heartbeat_client = client.clone();
    let control = Signal::new(SIGINT).flatten_stream().into_future();

    thread::Builder::new()
        .name("rabbitmq heartbeat".to_string())
        .spawn(move || {
            let mut core = Core::new().unwrap();

            match core.run(heartbeat_future_fn(&heartbeat_client).select2(control)) {
                Ok(_) => {
                    info!("Producer heartbeat thread exited cleanly");
                },
                Err(_) => {
                    error!("Producer heartbeat thread crashed, going down...");
                }
            }
        })
        .unwrap()
};

But with SIGINT, this thread keeps the app running forever and you must SIGKILL.

The other thing I tried was to create a oneshot channel in the initializer for my RabbitMQ producer, receiver in the select2 of the heartbeat and sender in an Option wrapped in my struct, so I can take it out in drop, take the ownership and send the signal to kill the heartbeat thread. This caused a segmentation fault.

There is not really any nice ways to just have the connections in the service's self until we get Pin to the stable Rust, and using Arc for the lapin producer in the main service is not feasible due the amount of traffic we get and everybody racing the reference counter.

What we really need is a blocking way to form a connection when the thread starts, store the connection in thread local and have a producing function returning a future for the threadpool to execute. Without the heartbeat thread the system starts nicely, produces events and exits cleanly. But when you wait for a while, the producing connections will be dead due to heartbeats missing.

There has been no release on crates.io for almost 9 months. Wouldn't it be a good idea to make a new one in order to benefit from the more recent changes?

I'm not sure if this would work or not, but I'm working on a project where I want to load my configuration from a config file. It would be nice if things like ConnectionOptions had #[derive(Deserialize)] from serde to allow reading the configuration through directly.

Do you think this can/should be added?

I know very little about Rust but this client uses a very uncommon (for OO-ish languages) API for channel operations (say, queue.declare).

In Java, C#, Ruby (Bunny), Python (Pika) queueDeclare and similar are methods on a channel. In Erlang, Clojure (Langohr), Haskell they are functions that accept a channel as the first parameter and are unaware of the connection.

According to this example, in Lapin queue_declare is a connection method that accepts a channel. What's the idea behind
this decision? Even if the idea is to introduce a special lower-level API, most protocol operations
are channel-scoped (not counting connection.start or channel.open, of course).

It is not unheard of for applications that use the aforementioned clients to accept channels and be very much unaware of what connection said channel is on.

I see that default frame size is set to 8192 while in most clients it is set to 0 which means the server default (131072) will be used.

What's the idea behind using such small frames? This value does not control TCP buffer size and therefore you won't reduce connection memory usage that way.

queue_bind is there but queue_unbind isn't yet

I'm using Lapin on Heroku and had to bodge together some boilerplate for translating the amqp://* URIs that are passed in through env vars. There is an official-ish specification (https://www.rabbitmq.com/uri-spec.html) and other libraries like bunny for Ruby have built-in support. (Though there's mention here that there are some implementation differences, but I can't imagine they wouldn't be that hard to paper over). I'd be willing to take this on, but I wanted to raise the issue first and see if anyone had any comments. I could envision this either being a method on ConnectionOptions, or even a separate helper function that returns ConnectionOptions and SockAddr together.

Hi there,

I am new to Rust and lapin. Now I am investigating the lapin-futures.
Say if I want to send 1000 messages through channel.basic_publish inside

channel
.queue_declare(...)
.and_then(...

What should be the proper approach? Thanks!

• channel_flow
• channel_flow_ok

Those aren't implemented in lapin-futures, do we want/need them?

Instead of looping while there are frames available, we should break after like 30 frames or so (number taken randomly).
This avoids getting stuck if we're flooded with frames for some reason.

• tx_select
• receive_tx_select_ok
• tx_commit
• receive_tx_commit_ok
• tx_rollback
• receive_tx_rollback_ok

Isn't implemented yet, not sure whether we need it or not.

I've run into some confusion around a consumer intermittently not consuming any messages. Thankfully I was able to reproduce it in a really simple test case (so I can leave the rest of my program's code behind), by turning https://github.com/sozu-proxy/lapin/blob/master/futures/examples/client.rs "upside-down" such that it starts trying to consume before publishing any message.

The end result looks pretty simple, so I'm guessing this is me misunderstanding something basic about how to use the library. But if that's the case, hopefully there's some way to detect whatever silly thing I'm doing, or some way to guide other travellers away from my mistake.

Here's the code:

extern crate lapin_futures as lapin;
extern crate futures;
extern crate tokio_core;

use futures::future::Future;
use futures::Stream;
use tokio_core::reactor::Core;
use tokio_core::net::TcpStream;
use lapin::types::FieldTable;
use lapin::client::ConnectionOptions;
use lapin::channel::{BasicConsumeOptions,BasicPublishOptions,BasicProperties,ExchangeBindOptions,ExchangeUnbindOptions,ExchangeDeclareOptions,ExchangeDeleteOptions,QueueBindOptions,QueueDeclareOptions};
use std::thread;

fn main() {
    let mut core = Core::new().unwrap();

    let handle = core.handle();
    let addr = "192.168.50.100:5672".parse().unwrap();

    core.run(
        TcpStream::connect(&addr, &handle).and_then(|stream| {
            lapin::client::Client::connect(stream, &ConnectionOptions {
                username: "guest".to_string(),
                password: "guest".to_string(),
                frame_max: 65535,
                ..Default::default()
            })
        }).and_then(|(client, heartbeat_future_fn)| {
            let heartbeat_client = client.clone();
            thread::Builder::new().name("heartbeat thread".to_string()).spawn(move || {
                Core::new().unwrap().run(heartbeat_future_fn(&heartbeat_client)).unwrap();
            }).unwrap();

            client.create_channel().and_then(|channel| {
                let id = channel.id;
                println!("created channel with id: {}", id);

                let ack_channel = channel.clone();
                let publish_channel = channel.clone();

                channel.queue_declare("hello", &QueueDeclareOptions::default(), &FieldTable::new()).and_then(move |_| {
                    println!("channel {} declared queue {}", id, "hello");

                    handle.spawn(
                        channel.basic_consume("hello", "my_consumer", &BasicConsumeOptions::default(), &FieldTable::new()).and_then(|stream| {
                            println!("got consumer stream");

                            stream.for_each(move |message| {
                                println!("got message: {:?}", message);
                                println!("decoded message: {:?}", std::str::from_utf8(&message.data).unwrap());
                                ack_channel.basic_ack(message.delivery_tag)
                            })
                        }).map_err(|err| {
                            println!("uh oh: {:?}", err);
                            ()
                        })
                    );
                    futures::future::ok::<(), std::io::Error>(())
                }).and_then(move |_| {
                    publish_channel.exchange_declare("hello_exchange", "direct", &ExchangeDeclareOptions::default(), &FieldTable::new()).and_then(move |_| {
                        publish_channel.queue_bind("hello", "hello_exchange", "hello_2", &QueueBindOptions::default(), &FieldTable::new()).and_then(move |_| {
                            publish_channel.basic_publish(
                                "hello_exchange",
                                "hello_2",
                                b"hello from tokio",
                                &BasicPublishOptions::default(),
                                BasicProperties::default().with_user_id("guest".to_string()).with_reply_to("foobar".to_string())
                            ).map(|confirmation| {
                                println!("publish got confirmation: {:?}", confirmation)
                            }).and_then(move |_| {
                                publish_channel.exchange_bind("hello_exchange", "amq.direct", "test_bind", &ExchangeBindOptions::default(), &FieldTable::new()).and_then(move |_| {
                                        publish_channel.exchange_unbind("hello_exchange", "amq.direct", "test_bind", &ExchangeUnbindOptions::default(), &FieldTable::new()).and_then(move |_| {
                                                publish_channel.exchange_delete("hello_exchange", &ExchangeDeleteOptions::default()).and_then(move |_| {
                                                        publish_channel.close(200, "Bye")
                                                })
                                        })
                                })
                            })
                        })
                    })
                })
            })
        })
    ).unwrap();

    core.run(futures::future::empty::<(), ()>()).unwrap();
}

If I'm lucky, then it'll print this:

created channel with id: 1
channel 1 declared queue hello
got consumer stream
got message: Message { delivery_tag: 1, exchange: "hello_exchange", routing_key: "hello_2", redelivered: true, properties: Properties { content_type: None, content_encoding: None, headers: None, delivery_mode: None, priority: None, correlation_id: None, reply_to: Some("foobar"), expiration: None, message_id: None, timestamp: None, type_: None, user_id: Some("guest"), app_id: None, cluster_id: None }, data: [104, 101, 108, 108, 111, 32, 102, 114, 111, 109, 32, 116, 111, 107, 105, 111] }
decoded message: "hello from tokio"

But lots of the time it'll print this...

created channel with id: 1
channel 1 declared queue hello
got consumer stream
publish got confirmation: None

...and then make no further progress, with the message sitting in the queue until I kill my program and it gets auto-deleted.

Is it obvious from looking at this that I'm doing something wrong? Perhaps something to do with spawning my consumer?

Do you know how to use basic_get in a blocking way? If the queue is empty and I call basic_get it returns "basic get returned empty" error. I'd like to block the calling thread in this case.

According to Keruspe in #73:

Hi,

for the basic_get stuff, please open a separate issue.
Blocking is not a viable option in a futures-based environment though, I think the right thing to do is rather to wrap the value in an Option or such as basic_get is expected to optionally return something

I don't necessarily need basic_get if I can work it around with something else.

I can imagine something like the following code, but it doesn't seem to be the optimal solution:

loop {
  msg = basic_get()
  if msg != "empty error" {
    break
  }
  sleep()
} 

Do you have any better idea? Thanks!

This allows receiver to check who sent the message.

The futures Channel.basic_publish method has a parameter called queue, which, when passed down to the underlying async connection, is used as the routing_key parameter. I don't know why there is a difference in nomenclature, but calling it queue is definitely quite deceptive.

According to the AMQP 0.9.1 reference, routing_key is definitely the correct nomenclature for the basic publish argument.

Excellent work on these libraries so far!

• tx_select
• tx_commit
• tx_rollback

Are there plans to extend the support to 1.0 version of the spec?

The function basic_get returns a Box<Future<Item=Message...>> but the type Message isn't exposed in lapin_futures. I don't really want to depend on lapin_async myself just to be able to write type signatures for processing messages.

Not sure whether we want/need this or not

• receive_basic_publish
• receive_basic_return
• receive_basic_reject
• receive_basic_recover_async
• receive_tx_select
• receive_tx_commit
• receive_tx_rollback

Hello!

I'm curious about using the Lapin client for a communication with RabbitMQ and have a couple of questions around this library:

1. Does it possible to make chained and flatten calls for futures from this library?

For example, when we have the following code:

use lapin_futures_rustls::lapin;
use futures::future::Future;
use lapin::types::FieldTable;
use lapin::channel::{ConfirmSelectOptions, QueueDeclareOptions, QueueDeleteOptions};
use tokio_core::reactor::Core;
use engine::rabbitmq::client::{RabbitMQClient};
use uuid::{Uuid};

fn main() {
    let cli = CliOptions::from_args();
    let mut core = Core::new().unwrap();
    let handle   = core.handle();

    let client = RabbitMQClient::new(&cli);
    let new_future = client.get_future(&handle);  // Returns a connection here 
    let queue_name = format!("{}", Uuid::new_v4());

    core.run(
        new_future
            .and_then(|client| {
                client.create_confirm_channel(ConfirmSelectOptions::default())
            })
            .map_err(|err| {
                let message = format!("Error during creating a channel: {}", err);
                error!("{}", message);
                err
            })

            .map(|channel| {
                let queue_declare_options = QueueDeclareOptions {
                    passive: true,
                    durable: true,
                    exclusive: true,
                    auto_delete: false,
                    ..Default::default()
                };

                channel.queue_declare(&queue_name, &QueueDeclareOptions::default(), &FieldTable::new())
                    .map(|_| channel) // <--- I'd like to return here a channel for using further
            })
            .map_err(|err| {
                let message = format!("Error during declaring the queue: {}", err);
                error!("{}", message);
                err
            })

            .map(|channel| {
                let queue_delete_options = QueueDeleteOptions {
                    if_unused: false,
                    if_empty: false,
                    ..Default::default()
                };

                channel.queue_delete(&queue_name, &QueueDeleteOptions::default())
                    .map(|_| channel)  // <--- and here too
            })
            .map_err(|err| {
                let message = format!("Error during deleting the queue: {}", err);
                error!("{}", message);
                err
            })

            .map(|channel| {
                println!("Channel is closed.");
                channel.close(200, "Close the channel.")
            })

    ).unwrap();
}

And trying to compile it, we're getting here the following error of compilation:

error[E0599]: no method named `queue_delete` found for type `futures::Map<std::boxed::Box<futures::Future<Item=(), Error=std::io::Error>>, [closure@src/main.rs:112:26: 112:47 channel:_]>` in the current scope
   --> src/main.rs:128:25
    |
128 |                 channel.queue_delete(&queue_name, &QueueDeleteOptions::default())
    |                         ^^^^^^^^^^^^

error: aborting due to previous error

2. Can I somehow receive a unique name of a queue, that was generated by RabbitMQ automatically, when the queue_name parameter was specified as an empty string, instead of generating a UUID4?

The future returned by lapin::client::Client::connect doesn't always complete.

This is very similar to #28, but I think it might be a different bug.

log file 1: https://www.dropbox.com/s/bhhbxmwx1lm5kk3/a_log.html?dl=0
log file 2: https://www.dropbox.com/s/xyat551tmos6du6/b_log.html?dl=0

When running the example client from lapin-futures, the consumer gets all messages currently in the queue as expected, but if I publish some new ones manually afterwards, all I get in the logs are:

DEBUG:tokio_core::reactor: loop poll - Duration { secs: 4, nanos: 898193532 }
DEBUG:tokio_core::reactor: loop time - Instant { tv_sec: 6526, tv_nsec: 962204220 }
DEBUG:tokio_core::reactor: loop process - 1 events, Duration { secs: 0, nanos: 27551 }
DEBUG:tokio_core::reactor: consuming notification queue
DEBUG:tokio_core::reactor: scheduling direction for: 0
TRACE:lapin_futures::consumer: consumer[my_consumer] not ready
DEBUG:tokio_core::reactor: loop poll - Duration { secs: 0, nanos: 23599 }
DEBUG:tokio_core::reactor: loop time - Instant { tv_sec: 6526, tv_nsec: 962389949 }
DEBUG:tokio_core::reactor: loop process - 0 events, Duration { secs: 0, nanos: 7419 }
DEBUG:tokio_core::reactor: loop poll - Duration { secs: 4, nanos: 897467435 }
DEBUG:tokio_core::reactor: loop time - Instant { tv_sec: 6531, tv_nsec: 859868088 }
DEBUG:tokio_core::reactor: notifying a task handle
DEBUG:tokio_core::reactor: loop process - 1 events, Duration { secs: 0, nanos: 100422 }
DEBUG:tokio_core::reactor: loop poll - Duration { secs: 0, nanos: 8267 }
DEBUG:tokio_core::reactor: loop time - Instant { tv_sec: 6531, tv_nsec: 859981547 }
DEBUG:tokio_core::reactor: loop process - 1 events, Duration { secs: 0, nanos: 8820 }
TRACE:lapin_futures::transport: decoded frame: Method(2, Basic(Deliver(Deliver { consumer_tag: "my_consumer", delivery_tag: 3, redelivered: false, exchange: "", routing_key: "hello" })))
TRACE:lapin_futures::consumer: consumer[my_consumer] not ready
DEBUG:tokio_core::reactor: loop poll - Duration { secs: 0, nanos: 4363 }
DEBUG:tokio_core::reactor: loop time - Instant { tv_sec: 6531, tv_nsec: 860151903 }
DEBUG:tokio_core::reactor: loop process - 0 events, Duration { secs: 0, nanos: 22663 }
DEBUG:tokio_core::reactor: loop poll - Duration { secs: 0, nanos: 101380774 }
DEBUG:tokio_core::reactor: loop time - Instant { tv_sec: 6531, tv_nsec: 961557809 }
DEBUG:tokio_core::reactor: loop process - 1 events, Duration { secs: 0, nanos: 100444 }
TRACE:lapin_futures::transport: decoded frame: Header(2, 60, ContentHeader { class_id: 60, weight: 0, body_size: 4, property_flags: 12288, property_list: {} })
TRACE:lapin_futures::consumer: consumer[my_consumer] not ready
DEBUG:tokio_core::reactor: loop poll - Duration { secs: 0, nanos: 6701 }
DEBUG:tokio_core::reactor: loop time - Instant { tv_sec: 6531, tv_nsec: 961856626 }
DEBUG:tokio_core::reactor: loop process - 0 events, Duration { secs: 0, nanos: 6336 }

The content header is received but not the body.

• basic_cancel
• basic_recover_async
• basic_recover
• basic_nack

While running the example client of the lapin-futures crate, the CPU usage stays at 100%, when it should be passively waiting for I/O readiness.

I believe that the commit that introduced this bug is bf6913c where the current task is rescheduled when a delivery couldn't be found in the transport's connection.

Hello,

I have an issue with message consuming.
I send messages on queue using rabbitmq, and sometime, I need to restart my rust instance because it don't catch any messages on the queue.
I have a simple configuration (local RabbitMQ, using exchange). regarding RabbitMQ logs the worker looks connected and registered as a consumer on the queue.

Do you have any ideas ? Already seen that ?

Marc-Antoine

Hi,

I tried to connect to one of my rabbitmq, it allow only one user with credentials.
I tried to define wrong credentials and the consumer process is still working, I clearly see in rabbitmq logs:

=INFO REPORT==== 26-Apr-2017::09:44:02 ===
accepting AMQP connection <0.1532.0> (172.17.0.1:37890 -> 172.17.0.2:5672)

=ERROR REPORT==== 26-Apr-2017::09:44:02 ===
Error on AMQP connection <0.1532.0> (172.17.0.1:37890 -> 172.17.0.2:5672, state: starting):
PLAIN login refused: user 'myusername' - invalid credentials

If I fix credentials but I don't update the vhost:

=INFO REPORT==== 26-Apr-2017::09:50:06 ===
accepting AMQP connection <0.1345.0> (172.17.0.1:37892 -> 172.17.0.2:5672)

=ERROR REPORT==== 26-Apr-2017::09:50:06 ===
Error on AMQP connection <0.1345.0> (172.17.0.1:37892 -> 172.17.0.2:5672, user: 'iadvize', state: opening):
access to vhost '/' refused for user 'myusername'

Is the consumer process blind to credentials failing?

Another problem, if I kill my rabbitmq, the consumer is still running.

Let me know if you want more details.

I'm wrote this code

[dependencies]
futures = "0.1.14"
lapin-futures = "0.10.0"
tokio-core = "0.1.8"

extern crate futures;
extern crate lapin_futures as lapin;
extern crate tokio_core;

use futures::Future;
use lapin::client::{Client as RabbitClient, ConnectionOptions};
use lapin::channel::{BasicPublishOptions, QueueDeclareOptions, BasicProperties};
use lapin::types::FieldTable;
use tokio_core::net::TcpStream;
use tokio_core::reactor::Core;

fn main() {
    let mut core = Core::new().unwrap();
    let handle = core.handle();

    let ch = TcpStream::connect(&"127.0.0.1:5672".parse().unwrap(), &handle)
        .and_then(|stream| {
            RabbitClient::connect(
                stream,
                &ConnectionOptions {
                    username: "rabbitmq".into(),
                    password: "rabbitmq".into(),
                    ..Default::default()
                },
            )
        })
        .map(|(client, _heatbeater)| client)
        .and_then(|rabbit_client| rabbit_client.create_channel());
    let ch = core.run(ch).unwrap();
    // panic occures immediately when call `ch.queue_declare`
    let f = ch.queue_declare("hello", &QueueDeclareOptions::default(), &FieldTable::new())
        .and_then(move |_| {
            ch.basic_publish(
                "",
                "hello",
                b"hello from tokio",
                &BasicPublishOptions::default(),
                BasicProperties::default(),
            )
        });

    core.run(f).unwrap();
}

and when I ran this, I got this error:

thread 'main' panicked at 'no Task is currently running', /home/kim/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.14/src/task_impl/mod.rs:44
note: Run with `RUST_BACKTRACE=1` for a backtrace.

I think this is the same issue as that described in here

Can this be fixed (is this a bug)? Or is this a restriction of client usage?

Probably should do done in futures with tokio-timer

Not all consumers are created, and messages aren't received

Hello!

I'm trying to write a small piece of code that will delete the queue and close the channel we have got an error on the linking with exchange point stage. For example, with the following code:

.and_then(move |channel| {
    channel.queue_bind(
        &queue_name_bind,
        &endpoint_link.get_response_exchange(),
        &endpoint_link.get_microservice(),
        &QueueBindOptions::default(),
        &FieldTable::new()
    )
        // Process the errors only when exchange point wasn't created yet
        .map_err(|err| {
            // Delete the recently created queue
            channel.queue_delete(&queue_name_bind, &QueueDeleteOptions::default()).map(move |_| {
                // Close the connection
                channel.close(INTERNAL_ERROR, "Connection closed: Invalid response exchange.").map(move |_| channel)
            })
        })
       // All is ready, go to the next step and re-use the channel with the next `.and_then` block 
        .map(|_| channel)
})
.map_err(|err| {
    let message = format!("Error during linking the response queue with exchange. Reason -> {}", err);
    error!("{}", message);
    err
})

I'm getting an error on the compile stage:

error[E0271]: type mismatch resolving `<[closure@src/engine/mod.rs:111:30: 117:22 channel:_, queue_name_bind:_] as std::ops::FnOnce<(std::io::Error,)>>::Output == std::io::Error`
   --> src/engine/mod.rs:102:14
    |
102 |             .and_then(move |channel| {
    |              ^^^^^^^^ expected struct `futures::Map`, found struct `std::io::Error`
    |
    = note: expected type `futures::Map<std::boxed::Box<futures::Future<Item=(), Error=std::io::Error>>, [closure@src/engine/mod.rs:113:100: 116:26 channel:_, err:_]>`
               found type `std::io::Error`
    = note: required because of the requirements on the impl of `futures::Future` for `futures::MapErr<std::boxed::Box<futures::Future<Item=(), Error=std::io::Error>>, [closure@src/engine/mod.rs:111:30: 117:22 channel:_, queue_name_bind:_]>`

How can I solve this issue, so that the error handling aren't expanding? Does the lapin provides any options for customizing behaviour (for example, client) so that it could close and delete everything if we've got a failure?

Error { repr: Custom(Custom { kind: Other, error: StringError("failed to handle frame: NotConnected") }) }

^ this is the error I see when I bind to a queue which does not exist. I see that there are error messages inside the library that would provide more information which don't seem to come back, such as here. This seems like a bug?

The "consumer" example code in lapin-futures gets to "got consumer stream", but then fails to get to the "got message" portion. There are messages in the queue and I see the consumer is connected in Rabbit's web interface. It does move the message from "ready" to "unacked" but never actually gets the message. I also see that that doc test is also ignoreed.

I am using master.

Any assistance is appreciated. It also could be something with my setup.

Examples and docs suggest to spawn a thread for amqp heartbeat like this:

      let heartbeat_client = client.clone();
      thread::Builder::new().name("heartbeat thread".to_string()).spawn(move || {
        Core::new().unwrap().run(heartbeat_future_fn(&heartbeat_client)).unwrap();
      }).unwrap();

Here is a snippet from Client::connect docs:

  /// The heartbeat future should be run in a dedicated thread so that nothing can prevent it from
  /// dispatching events on time.
  /// If we ran it as part of the "main" chain of futures, we might end up not sending
  /// some heartbeats if we don't poll often enough (because of some blocking task or such).

However it seems wrong. As I can see, there is no special synchronising there on the protocol level. Under the load on main connection thread (for example, consume / ack loop) heartbeat from the other thread causes amqp server to drop connection because of frame decoding errors.

Here is a sample RabbitMQ error:

=CRASH REPORT==== 24-Jul-2017::20:46:00 ===
  crasher:
    initial call: rabbit_reader:init/4
    pid: <0.22706.3455>
    registered_name: []
    exception exit: {unknown_method_id,{0,13}}
      in function  rabbit_framing_amqp_0_9_1:lookup_method_name/1 (src/rabbit_framing_amqp_0_9_1.erl, line 332)
      in call from rabbit_command_assembler:analyze_frame/3 (src/rabbit_command_assembler.erl, line 64)
      in call from rabbit_reader:handle_frame/4 (src/rabbit_reader.erl, line 988)
      in call from rabbit_reader:handle_input/3 (src/rabbit_reader.erl, line 1054)
      in call from rabbit_reader:recvloop/4 (src/rabbit_reader.erl, line 472)
      in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 454)
      in call from rabbit_reader:start_connection/4 (src/rabbit_reader.erl, line 390)

And this is the error originated from lapin-futures:

ERROR:lapin_futures::client: Failed to send heartbeat: Error { repr: Os { code: 104, message: "Connection reset by peer" } }
ERROR:lapin_futures::transport: upstream poll gave error: Error { repr: Os { code: 104, message: "Connection reset by peer" } }
ERROR:lapin_futures::transport: upstream poll gave error: Error { repr: Custom(Custom { kind: Other, error: StringError("bytes remaining on stream") }) }
ERROR:lapin_futures::transport: upstream poll gave error: Error { repr: Os { code: 104, message: "Connection reset by peer" } }

This happens only in dedicated heartbeat thread scenario. Evaluating heartbeat future on the same reactor with main connection works fine.

Digging more into the code, I'm not sure the this situation is being handled. Publishing a big message (one which the payload's length - or actually the body frame - is higher than the max frame size) requires splitting. Same thing for message delivery / consuming.

Is this supported?

Thanks

RabbitMQ Version: 3.6 & 3.7
Lapin version: a8ac2b2

When creating a consumer with an empty consumer tag, RabbitMQ generates one and returns it to the client: lapin-async gets it and create its consumer appropriately however lapin-futures just uses the consumer tag given in basic_consume which means that messages aren't received by the consumer.

Code sample: https://gist.github.com/kureuil/c0914bbe0c4463acd655bfa3872b09c1

Change the consumer_tag at line 46 from "consumer" to "" and it won't receive any messages.

Looking at the current design, seems a bit challenging to implement it.

You can't use a queue in a channel that didn't declare it. We should ensure the queue has been declared in the channel somehow, maybe abstracting the queue name in a struct?