capnproto / capnproto-rust Goto Github PK
View Code? Open in Web Editor NEWCap'n Proto for Rust
License: MIT License
Cap'n Proto for Rust
License: MIT License
So that it can be used in rust-lang/rustc-serialize
rust-lang-deprecated/rustc-serialize#35
I came across this protocol and it looks absolutely perfect for microcontrollers! I understand some features support dynamically sized types, so cannot be supported on the stack, but was wondering if a subset of this library could support no_std?
If it is possible, but would have to be a separate crate then that would also be useful to know.
Thanks!
[Let me know if there's a better place for these questions.]
What's the best way to handle different capnp message structs coming in from the same zmq socket? There doesn't seem to be a way to differentiate them before decoding. I see there's a type_id
on the reader and there's also capnp generics, maybe a third approach?
In my previous implementation with json I had a top-level "request" object which contained a type id and a string blob. Then I would use a match on the type id to decode the inner string blob into a struct based on the type of request. (This was a hack to work around rustc-serialize limitations at the time.)
EDIT: Maybe unions? http://stackoverflow.com/questions/28149139/serializing-mutable-state-and-sending-it-asynchronously-over-the-network-with-ne#comment44738481_28156323
I think this could be great for python (and other) FFI so I have been trying to figure out how to load &[u8]
into an actual object. This is what I have so far but I think it's probably misguided.
#[no_mangle]
pub extern fn test_capnproto(thing: *const u8, length: size_t) {
let array: &[u8] = unsafe{ slice::from_raw_parts(thing, length as usize) };
let mut message = MallocMessageBuilder::new_default();
let mut test_date = message.init_root::<date::Builder>();
{ // scope for builder
let b = test_date.borrow();
for (i, elem) in array.iter().enumerate() {
b.set_data_field(i, *elem);
}
}
println!("{:?}", test_date);
}
Any tips? Perhaps something to do with read_message
? I couldn't figure it out from the examples
capnproto-rust
is currently impenetrable besides the examples. If one is not already familiar with using it in C++, it's very hard to "dive in".
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:38:5: 38:70 warning: the trait `core::marker::Sized` is not impl[788/1285]
r the type `Self` [E0277]
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:38 fn get_from_pointer(reader : &PointerReader<'a>) -> Result<Self>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:38:5: 38:70 help: run `rustc --explain E0277` to see a detailed explanatio
n
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:38:5: 38:70 note: `Self` does not have a constant size known at compile-ti
me
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:38 fn get_from_pointer(reader : &PointerReader<'a>) -> Result<Self>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:38:5: 38:70 note: this warning results from recent bug fixes and clarifica
tions; it will become a HARD ERROR in the next release. See RFC 1214 for details.
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:38 fn get_from_pointer(reader : &PointerReader<'a>) -> Result<Self>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:38:5: 38:70 note: required by `core::result::Result`
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:38 fn get_from_pointer(reader : &PointerReader<'a>) -> Result<Self>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:70:5: 70:71 warning: the trait `core::marker::Sized` is not implemented fo
r the type `Self` [E0277]
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:70 fn get_from_pointer(builder : PointerBuilder<'a>) -> Result<Self>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:70:5: 70:71 help: run `rustc --explain E0277` to see a detailed explanatio
n
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:70:5: 70:71 note: `Self` does not have a constant size known at compile-ti
me
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:70 fn get_from_pointer(builder : PointerBuilder<'a>) -> Result<Self>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:70:5: 70:71 note: this warning results from recent bug fixes and clarifica
tions; it will become a HARD ERROR in the next release. See RFC 1214 for details.
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:70 fn get_from_pointer(builder : PointerBuilder<'a>) -> Result<Self>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:70:5: 70:71 note: required by `core::result::Result`
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:70 fn get_from_pointer(builder : PointerBuilder<'a>) -> Result<Self>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:90:5: 90:76 warning: the trait `core::marker::Sized` is not implemented fo
r the type `Self` [E0277]
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:90 fn from_u16(value : u16) -> ::std::result::Result<Self, ::NotInSche
ma>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:90:5: 90:76 help: run rustc --explain E0277
to see a detailed explanatio
n
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:90:5: 90:76 note: Self
does not have a constant size known at compile-ti
me
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:90 fn from_u16(value : u16) -> ::std::result::Result<Self, ::NotInSche
ma>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:90:5: 90:76 note: this warning results from recent bug fixes and clarifica
tions; it will become a HARD ERROR in the next release. See RFC 1214 for details.
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:90 fn from_u16(value : u16) -> ::std::result::Result<Self, ::NotInSche
ma>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:90:5: 90:76 note: required by core::result::Result
/home/cmr/.multirust/toolchains/nightly/cargo/registry/src/github.com-0a35038f75765ae4/capnp-0.4.3/src/traits.rs:90 fn from_u16(value : u16) -> ::std::result::Result<Self, ::NotInSche
ma>;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(I'll take a poke at a patch)
(oops, accidentally submitted early)
Here's an example:
test.rs:
extern crate capnp;
pub mod submod {
#[path = "../test_capnp.rs"]
pub mod test_capnp;
}
fn main() { }
test.capnp:
@0xb05abd72309922f1;
struct Test {
test @0 :Int64;
method @1 :Method;
enum Method {
unkown @0;
}
}
This errors with:
test_capnp.rs:36:40: 36:50 error: failed to resolve. Maybe a missing `extern crate test_capnp`?
test_capnp.rs:36 pub fn get_method(&self) -> Option<::test_capnp::test::method::Reader> {
^~~~~~~~~~
test_capnp.rs:36:40: 36:74 error: use of undeclared type name `test_capnp::test::method::Reader`
test_capnp.rs:36 pub fn get_method(&self) -> Option<::test_capnp::test::method::Reader> {
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_capnp.rs:64:40: 64:50 error: failed to resolve. Maybe a missing `extern crate test_capnp`?
test_capnp.rs:64 pub fn get_method(&self) -> Option<::test_capnp::test::method::Reader> {
^~~~~~~~~~
test_capnp.rs:64:40: 64:74 error: use of undeclared type name `test_capnp::test::method::Reader`
test_capnp.rs:64 pub fn get_method(&self) -> Option<::test_capnp::test::method::Reader> {
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_capnp.rs:68:38: 68:48 error: failed to resolve. Maybe a missing `extern crate test_capnp`?
test_capnp.rs:68 pub fn set_method(&self, value : ::test_capnp::test::method::Reader) {
^~~~~~~~~~
test_capnp.rs:68:38: 68:72 error: use of undeclared type name `test_capnp::test::method::Reader`
test_capnp.rs:68 pub fn set_method(&self, value : ::test_capnp::test::method::Reader) {
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: aborting due to 6 previous errors
The problem is that test_capnp.rs
is using ::
to try to get to the top of the file, but instead that is referring to the top of the crate. Unfortunately I don't think there's any shorthand to get to the top of the local file scope, so to do this, we would need to create a series of super::...
s to walk up the current module chain to get to the root.
Why does get_segments_for_output
take a function of the segments as an argument instead of just returning the segments?
It'd be good to be able to use this with rustpkg. The most notable change is that capnprust.rs should be renamed to lib.rs, but there may be other changes necessary; I haven't delved into this project's compiler stuff and do not know if it needs anything special. Tests also get done with test.rs being the root file in a crate.
It'd be great if the documentation for the project was available online. You can easily integrate this into your CI automatically in the .travis.yml
file if you are able to generate an application token and secure it.
I wrote about how to do this here, it should only take a few minutes if you already have the travis
gem installed.
The RPC repo could also benefit from this.
Currently, when you read a message and the traversal limit is exceeded, you get an error like "Message contained out-of-bounds list pointer", which is inaccurate. It should instead say something like "Exceeded message traversal limit".
Hello. I haven't real experience with capnproto or rust, except per two toy micro proyects. But i wish help to port capnproto to tokio and future-rs. I'll remove the dependencies on gj and gj-io, add tokio and futures-rs and fix compiler errors until the compiler dont scream. Is that good idea?
P.d. sorry by my english
Generated code produces a large amount of non_camel_case_types and dead_code warnings, it should be marked allow
so that rustc
doesn't complain.
Is there any reason IntoIterator
trait is not implemented anywhere in the code?
It would be useful at least for capnp::primitive_list::Reader
.
I'm getting this error:
failed to run custom build command for `capnp-rpc v0.5.0`
Process didn't exit successfully: `/opt/capnp-raft/target/debug/build/capnp-rpc-1baa9a0f1ce31119/build-script-build` (exit code: 101)
--- stderr
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:365
stack backtrace:
1: 0x7fb138e90620 - sys::backtrace::tracing::imp::write::he9cf93dccfa17cceNlt
2: 0x7fb138e931e5 - panicking::log_panic::closure.39173
3: 0x7fb138e92c31 - panicking::log_panic::hd9f4aa0cbdff587aZlx
4: 0x7fb138e79f93 - sys_common::unwind::begin_unwind_inner::h90fc1a96c34cff35eds
5: 0x7fb138e7a608 - sys_common::unwind::begin_unwind_fmt::hfca8a38c37457f6fkcs
6: 0x7fb138e8fcf1 - rust_begin_unwind
7: 0x7fb138ec06bf - panicking::panic_fmt::h19598b35d555b127wZK
8: 0x7fb138ebc7d8 - panicking::panic::hfefb33ce32c56da43XK
9: 0x7fb138de0f35 - option::Option<T>::unwrap::h1607710876796834354
at ../src/libcore/macros.rs:20
10: 0x7fb138dde1c0 - main::h7000649270783f3cfaa
at /root/.cargo/registry/src/github.com-0a35038f75765ae4/capnp-rpc-0.5.0/build.rs:11
11: 0x7fb138e92974 - sys_common::unwind::try::try_fn::h6140512138114007296
12: 0x7fb138e8fb58 - __rust_try
13: 0x7fb138e9260f - rt::lang_start::h21dafd243738f0d57ix
14: 0x7fb138deffe9 - main
15: 0x7fb137fcdaf4 - __libc_start_main
16: 0x7fb138ddde78 - <unknown>
17: 0x0 - <unknown>
I've tested both stable and nightly, on Windows and Linux.
Hello,
While playing with the lib, I opened a broken file and it crashed with an assert:
---- tests::broken_file stdout ----
thread 'tests::broken_file' panicked at 'assertion failed: size > 0', /Users/octplane/.multirust/toolchains/nightly/cargo/registry/src/github.com-88ac128001ac3a9a/capnp-0.7.0/src/serialize_packed.rs:101
stack backtrace:
1: 0x104de2598 - std::sys::backtrace::tracing::imp::write::h9fb600083204ae7f
2: 0x104de4835 - std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::hca543c34f11229ac
3: 0x104de43bf - std::panicking::default_hook::hc2c969e7453d080c
4: 0x104dd9466 - std::sys_common::unwind::begin_unwind_inner::h30e12d15ce2b2e25
5: 0x104d7b6d7 - std::sys_common::unwind::begin_unwind::h65390c63b501d6ed
6: 0x104d80bb9 - _<capnp..serialize_packed..PackedRead<R> as std..io..Read>::read::h225fe399cdb1fa2c
You can checkout https://github.com/octplane/kiosque
at tag ISSUE_CAPNPROTORUST_BROKEN_FILE
and run cargo test broken_file
.
Schema is at https://github.com/octplane/kiosque/blob/ISSUE_CAPNPROTORUST_BROKEN_FILE/logformat/src/schema.capnp and guilty file is https://github.com/octplane/kiosque/blob/ISSUE_CAPNPROTORUST_BROKEN_FILE/broken/sample148.capnp
I suppose this should be recoverable in someway at the consumer level, hence this issue.
Thanks!
The crates documentation discusses build scripts where you would have a git submodule in tree which is built. Usually this is done in a separate -sys
crate but there's no requirement. Building the underlying library is done because there is a desire to reduce the dependencies on system wide installed libraries since it results in inconsistent behaviour. It also means that when cargo has built a package, it's ready to go and doesn't result in errors like these.
capnproto-rust
redefines some traits / structures that seem to exist in the standard library, such as BufferedReader and BufferedWriter and the corresponding wrappers, but with a few changes.
Is there a particular reason to redefine those traits and structures, such as an insufficent API or too much API instability? Or is it there more for legacy reasons?
I'm writing a program which wants to read data from files using mmap and capnp. This is not quite memory safe, for the subtle reason[1] that capnp-rust's pointer validation is potentially not TOCTOU safe; if another process makes a stray write into the file (unlikely, but I'd like to cover all valid error situations), it could cause a reader to validate a valid pointer, then use an invalid one resulting in random memory accesses.
If capnp-rust were modified to be TOCTOU-safe when validating pointers, and all reads from the secoment were made into unordered atomic loads[2] to prevent the optimizer from introducing new TOCTOUs, then capnp-rust would be able to provide memory safety even when reading from raw pointers that might be concurrently modified.
[1] There is also the not-subtle reason that I/O errors result in SIGBUS, but I'm tackling that as a separate problem. There are several possible approaches.
[2] std::intrinsics::atomic_load_unordered() exists; unfortunately there's no stable wrapper for this in libstd (you can transmute to AtomicUsize, but that doesn't work for smaller types and only provides Monotonic, not Unordered).
The c++ api has orphan support are there plans to adding it to the rust api? It seems like the only way to do a bottom-up building of a capn'proto message.
The Rust team yesterday accepted this RFC on a standard Allocator interface. It hasn't been implemented yet, but I was taking a look and it seems like it would cover the needs of the current Allocator
trait in capnproto-rust
- we can use the Layout
struct to provide the same size + alignment properties that Word
currently has.
When this becomes implemented in the stdlib, would you be interested in using it instead of/to back the current Allocator
scheme in this crate? I'd be interested in working on a PR for this in the future.
The InputStream
and OutputStream
traits closely mirror the functionality of the standard library's Read
and Write
traits, but they are less flexible. In addition to having to wrap Read
and Write
instances, it makes it impossible to use a type that implements Read
and Write
(for example TcpStream
) in separate InputStreamWrapper
and OutputStreamWrapper
instances.
I looked at replacing the uses of these traits with their standard library equivalent, or just providing default implementations of them for the standard library types, but it requires some changes to the deepest darkest parts of serialize_packed
, so I got a bit scared off.
As per this issue: NixOS/nixpkgs#8186
It would seem Cargo.lock is needed to ensure NixOS builds are deterministic. Would you consider adding Cargo.lock to git repo?
It's prohibitively difficult to use capnproto-rust with non-blocking input/output streams (Read/Write instances that can return WouldBlock
. I've been looking at the codebase the last few days to see what it would take to add support for resumable serialization/deserialization. I believe by adding a WouldBlock
variant to the capnproto Error
type which holds intermediate state it would be possible. So for instance:
enum WouldBlockState {
ReadPacked(..),
Read(..),
WritePacked(..),
Write(..),
}
enum Error {
Decode { .. },
WouldBlock(WouldBlockState),
Io(io::Error),
}
with additional read/write methods to resume reading a message after a WouldBlock
error:
pub fn resume_read_message(WouldBlockState, &mut InputStream, ReaderOptions) -> Result<..>;
pub fn resume_read_message_packed(WouldBlockState, &mut BufferedInputStream, ReaderOptions) -> Result<..>;
pub fn resume_write_message(WouldBlockState, &mut OutputStream, ReaderOptions) -> Result<..>;
pub fn resume_write_message_packed(WouldBlockState, &mut BufferedOutputStream, ReaderOptions) -> Result<..>;
I want to get your thoughts on this before diving too far into the implementation.
Hi! In http://youtu.be/A65w-qoyTYg?t=14m57s you say that reading from a Cap'n Proto message is comparable to a struct field access, or at least I've got that impression from your talk.
Now, testing it with a simple benchmark
#[bench] fn capnp_unpack (bencher: &mut Bencher) {
bencher.bytes = 3;
let encoded: Vec<u8> = {
let mut builder = MallocMessageBuilder::new_default();
{ let mut cache_value = builder.init_root::<cache_value::Builder>();
cache_value.set_value (b"foo"); }
let mut buf = Vec::with_capacity (128);
capnp::serialize::write_message (&mut buf, &builder) .unwrap();
buf};
bencher.iter (|&:| {
let mut bytes = encoded.as_slice();
let reader = capnp::serialize::new_reader (&mut bytes, capnp::ReaderOptions::new()) .unwrap();
let value = reader.get_root::<cache_value::Reader>();
assert! (value.get_value() == b"foo");
});}
which uses the following schema
struct CacheValue {
expires @0 :UInt64;
value @1 :Data;
tags @2 :List(Text);
}
shows that reading from a Cap'n Proto message is much more expensive than a struct access.
I get 3931 ns/iter
from it.
(In fact, the LMDB database where I keep this value is faster at returning it from the database (616 ns/iter
!) than Cap'n Proto is at decoding it).
Theoretically I'd imagine that the decoding should involve just some memory accesses, but examining the new_reader
code I see a couple of Vec allocations there.
So,
Am I doing something wrong?
Is it a state of the art Cap'n Proto performance or will it be improved?
As I could see to get a memory aligned [u64] in capnproto you must use a primitive list with T = u64/UInt64 but them it is unclear how to get access to a slice of it
Especially with cargo, generating code using tools can be annoying. Add a syntax extension that will generate the code into a module. With include_str
etc, no flexibility is lost.
When attempting to build from on Mac OS X 10.9.2, rustc 0.10 (x86_64-apple-darwin) against 2021bf0, build fails with:
rustc -O capnp/lib.rs --out-dir capnp
capnp/lib.rs:16:1: 16:19 error: can't find crate for `libc`
capnp/lib.rs:16 extern crate libc;
^~~~~~~~~~~~~~~~~~
error: aborting due to previous error
make: *** [capnp/compilation-marker] Error 101
The commit which causes this problem is in fact 2021bf0 as 999c7c2 builds perfectly fine.
However, from rust master, this is not an issue. This issue is mostly for others who hit the same problem, as I suspect it won't be fixed.
I have a schema containing a struct field named data
that's a union. That causes a build failure:
error[E0255]: a module named `data` has already been imported in this module
--> /home/roc/pernosco/code/target/debug/build/rr-trace-3d4d5d2e72ce7323/out/rr_trace_capnp.rs:3240:7
|
3064 | use capnp::{text, data, Result};
| ---- previous import of `data` here
...
3240 | / pub mod data {
3241 | | #![allow(unused_imports)]
3242 | | use capnp::capability::{FromClientHook, FromTypelessPipeline};
3243 | | use capnp::{text, data, Result};
... |
3458 | | pub type WhichBuilder<'a,> = Which<Result<primitive_list::Builder<'a,i32>>,Result<struct_list::Builder<'a,::rr_trace_capnp::opened_fd::Owned>>>;
3459 | | }
| |_______^ `data` already imported
Alright, I think I finally know enough to properly phrase this question.
Right now it seems that the only way to create a TypedReader
is to build a message using the standard pattern, serialize, deserialize (which then owns the segments), and stuff into the TypedReader
.
Is there a way to build the message using OwnedSegments
to start with? Or get at the OwnedSegments
from the constructed message? If we're using HeapAllocator
, the builder presumably owns the data and it could be safe to move from builder into TypedReader
.
EDIT: I suspect it has something to do with Builder::get_segments_for_output()
, but I'm not sure how to get the OutputSegments
-> OwnedSegments
Generic types currently return AnyPointers but should return the actual type. By generic I mean:
struct Container(Value) {
value @0 :Value;
}
So, with a bit of hackery using git submodules, capnp-rpc can be made to work as a cargo package as it stands now, without duplicating all the code. I've managed it with brandonson/capnproto-rust-rpc. Unfortunately, ugly hackery is all it is, and I suspect it wouldn't take much to make it topple over and stop working.
The capnp crate works great with cargo, so it's a shame that capnp-rpc doesn't. I'd love to see capnp-rpc moved into its own repo, as that would let it exist as its own cargo package properly, instead of the weird things I've done to sort of get that working.
could you please add capnproto-rust to http://hiho.io/rust-ci/, thanks
I have a without @1 :Text;
field in a message which is (intentionally) not initialised. The result of calling get_without()
on the reader for this message varies based on compile settings. This is not ideal.
With --release
and lto = true
, it returns an error:
Error { kind: Failed, description: "Message contains non-list pointer where text was expected." }
On all other configurations, including --release
without lto = true
, and non-release, it returns the empty string.
This is, um, not expected.
capnproto-rust
, or is this arguably a compiler bug?get_
error for missing fields?I've made a sample project, which will demonstrate the problem with cargo run --release
. The other cases can be tested with cargo run
and by #commenting
lto = true
in Cargo.toml
, and cargo run --release
. It's a binary (which does nothing on success) as it doesn't seem to fail in cargo test --release
; didn't look why.
git clone https://github.com/FauxFaux/capnp-get-null
cd capnp-get-null
RUST_BACKTRACE=1 cargo run --release
Finished release [optimized + debuginfo] target(s) in 0.0 secs
Running `target/release/capnp-get-null`
thread 'main' panicked at 'without: Error { kind: Failed, description: "Message contains non-list pointer where text was expected." }', /checkout/src/libcore/result.rs:906:4
stack backtrace:
0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
...
6: core::result::unwrap_failed
at /checkout/src/libcore/macros.rs:41
7: capnp_get_null::main
at /checkout/src/libcore/result.rs:799
at src/main.rs:28
at src/main.rs:35
8: main
9: __libc_start_main
10: _start
The code which errors is at: main.rs:28
The schema is totally trivial:
@0xbc035dbe0d877227;
struct Foo {
with @0 :Text;
without @1 :Text;
}
Tested on Ubuntu 17.10 (Artful) amd64 with rustc 1.21.0 (3b72af97e 2017-10-09)
(stable) and rustc 1.23.0-nightly (8b22e70b2 2017-10-31)
.
I admit defeat :) Continuing this question from IRC...I've been unable piece together how to appropriately use Owned. I'm not familiar enough with capnp or Rust lifetimes to work out how the various pieces are used, and there aren't many examples in the three code repositories that use Owned.
My question on IRC:
any capnp tips to dealing with ownership? E.g. i want to serialize a timestamp, but to "rebuild" the timestamp on the other end, Chrono needs to take ownership of the u64 from capnp. I keep running into this with anything more advanced than simple primitives
And @dwrensha's response:
capnp::traits::Owned might help: https://github.com/dwrensha/capnproto-rust/blob/f24ab9e95f1467ff023dd3b8a51495aa1637fa4d/src/traits.rs#L41-L56
I think I eventually want to switch message::Reader and message::Builder to be parameterized by a
T where T : for<'a> Owned<'a>
.imagine that message::Reader is paramterized by foo::Owned. Then we would have
got_root<'a>(&'a self) -> foo::Reader<'a>
. The point is that the container doesn't need to worry about lifetimes; we can wait until a self-borrowing accessor method to commit to a lifetime.er, it would be
pub struct Reader<S, T> where S: ReaderSegments, T: for<'a> Owned<'a> {...}
andget_root<'a>(&'a self) -> <T as capnp::traits::Owned<'a>>::Reader
Do you happen to have a small example handy? This seems like a critical part of using Capnp correctly. I'd be happy to contribute back documentation once I figure out how the various pieces work together :)
Hello,
I'd like to:
Any tips / pseudo-code on how to do this would be much appreciated.
Thanks!
Edit: I note the C++ code has capnp::FlatArrayMessageReader. Something like this seems to be workable for reading a message from a mmap'd memory segment.
Please, please add a basic example somewhere.
At the moment I have absolutely no idea how to use any of your stuff.
Hi,
I am able to write a message to a vec, but I am unable to read it back.
This is how I write to vec:
let mut message1 = capnp::message::Builder::new_default();
let mut message2 = capnp::message::Builder::new_default();
{
let mut transfer = message1.init_root::<Transfer::Builder>();
// Setters
transfer.set_from("fromAddress");
transfer.set_to("toAddress");
let mut op_code = message2.init_root::<OpCode::Builder>();
op_code.set_transfer(transfer.borrow_as_reader());
}
let mut out:Vec<u8> = Vec::new();
capnp::serialize_packed::write_message(&mut out, &message1);
Now I am sending 'out' over the network to another computer (this is why I convert to Vec), how can I deserialize it back to 'op_code' on the other side ?
Currently only struct_list
implements iter
. For data_list
, text_list
, list_list
, enum_list
and primitive_list
it seems to be missing.
In trying to structure an app so that messages are read from a (TCP) stream, and then passed off to another thread for further processing, I'm running into issues because the message reader is not Send
. Is this intended, or perhaps fallout from the opt-in built-in changes? For reference these are the errors I'm seeing when trying to move a message into a new thread:
error: the trait `core::marker::Send` is not implemented for the type `*const capnp::arena::SegmentReader`
error: the trait `core::marker::Send` is not implemented for the type `*const u8`
error: the trait `core::marker::Send` is not implemented for the type `*const capnp::layout::WirePointer`
As I see capnproto is all about non copy/ in-place casting so If at any point you need to deep copy all effort is lost. https://github.com/nikomatsakis/rayon is a parallel crate with easy parallel processing but to use it with capnproto without support you need to copy everything losing capnproto purpose.
As I could see there is no way to set elements using a mutable iterator.
You can follow a null
capnp pointer. But if you try to access some fields in it (in my case a text field) it will trigger a segmentation fault. I'm not sure if the bug is accessing the field, or being able to follow the pointer in the first place.
Example of the bug: https://github.com/csherratt/capnp_crash
Backtrace:
#0 0x000055555557cf84 in capnp::private::endian::u32.Endian::get (self=0x0) at /home/colin/Source/capnproto-rust/src/private/endian.rs:52
#1 0x000055555557d679 in capnp::private::endian::WireValue<T>::get (self=0x0) at /home/colin/Source/capnproto-rust/src/private/endian.rs:29
#2 0x000055555557db39 in capnp::private::layout::WirePointer::is_null (self=0x0) at /home/colin/Source/capnproto-rust/src/private/layout.rs:322
#3 0x0000555555581048 in capnp::private::layout::wire_helpers::read_text_pointer (segment=0x0, reff=0x0, default_value=0x0, default_size=0)
at /home/colin/Source/capnproto-rust/src/private/layout.rs:1773
#4 0x0000555555581eb2 in capnp::private::layout::PointerReader<'a>::get_text (self=0x7fffffffd5d8, default_value=0x0, default_size=0)
at /home/colin/Source/capnproto-rust/src/private/layout.rs:1915
#5 0x000055555555c56d in capnp_crash::test_capnp::foo::Reader<'a>::get_txt (self=...) at target/debug/build/capnp_crash-d343ef91834373e5/out/test_capnp.rs:59
#6 0x000055555555cc0a in capnp_crash::main () at src/main.rs:26
#7 0x000055555558c7f5 in rt::unwind::try::try_fn::h9959899096596013704 ()
#8 0x000055555558a479 in __rust_try ()
#9 0x000055555558c4e0 in rt::lang_start::h46248b0221ecaad5blx ()
#10 0x000055555556ba67 in main ()
If I have a schema like this
struct Foo {
bar :union {
// omitted
}
}
the autogenerated code for Foo::Builder
includes a method called init_bar
, which consumes Foo::Builder
. I'm new to rust, but this seems unideal to me for a couple reasons:
init_bar
or I have to create another Foo::Builder
from the containing buffer after setting up bar.Foo::Builder
is borrowed in the current scope I can't call init_bar
Was this intentional or is it a bug?
At present this project is in at least one thing deviating from the standard naming conventions encouraged by the Rust developers and used in most Rust projects; method names are camelCase
rather than snake_case
. (There may be more; I haven't delved.) It would be good to get this following Rust conventions rather than C++ conventions, at least before v1.0.
Hi,
This might be more of a rust question, but how do I stop EzRpcServer? I have a test that starts the server in a child task, and I'd like to stop it when the test finishes. The test below passes, but then it hangs presumably because the server is still running.
#[test]
fn my_test() {
spawn(proc() {
let rpc_server = EzRpcServer::new("127.0.0.1:12345").unwrap();
});
// Do things.
}
Thanks!
A 0 is hardcoded in the compiler but used with floating point types as in https://github.com/millere/osiris/blob/3cc9adc1d5fb698b8e1aa6b8dbba07b471395303/net/src/proto.capn and https://github.com/millere/osiris/blob/3cc9adc1d5fb698b8e1aa6b8dbba07b471395303/net/src/proto_capnp.rs#L80
Currently, the code (especially layout.rs) is horribly unsafe. In fact, it appears less safe than even the C++ version, which uses template metaprogramming to catch overflow at compile time.
While the interface claims to be safe, the amount of unsafe code in the implementation is worrying when security is a concern. Most of the unsafety seems to appear as a result of accessing fields essentially by raw pointer arithmetic. One solution might be to generate a #[repr(packed)]
struct type that represents the actual data off of the wire. Accessing this struct is then safe, at least once one has a reference (by necessity shared) to it. Rust provides #[inline]
to ensure that the accessors inline away and have zero overhead.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.