stellar / rs-soroban-env Goto Github PK

We need a way for a contract to exit the subtransaction / call that it's currently running immediately, without popping / returning all the way back out to its outermost frame. The moral equivalent of the exit(int) standard library function in C.

Pass void for success, or a status for failure -- or just a status if we're going to keep the "ok" / "success" status code (this might be the exception to the "never pass a status" rule).

We should allow aborting the current subtransaction with some details -- eg. calling env.abort(status) -- rather than just calling panic!(). This might include revising the status type to include a variant that denotes its file name / line number / a user-provided message as in #205.

Include new types since round 1 was done.

• Bignum
• Map
• Metering

Under new infrastructure (#211) that splits the diagnostic details and error code.

As described in stellar/rs-soroban-sdk#52 (comment) there are some testing scenarios where taking a snapshot of the host Env and copy it and replay from a point is helpful, and not having this can make some testing less ergonomic/easy.

Probably something like a deep_clone fn like how @graydon describes in stellar/rs-soroban-sdk#52 (comment).

I think the Eng interface should use RawObj for return values that are known to be definitely objs. It would make the SDK more efficient since the SDK would not need to try into or abort those conversions. Alternatively we could expose an unchecked conversion function but it'd be great to keep unsafe conversations inside the env crates and not expose them at the interface.

@graydon wdyt?

Test contract machinery uses RefCell and right now is not sufficiently careful to avoid immutable borrows while mutably borrowing. This needs to be fixed for this to be useful.

To support converting an iterator into a Map, we need to be able to get a slice of a map in a similar way that we can get a slice of a Vec. The host fn should accept two keys, a low and a high, and return a map that has the elements between those two keys.

For Vecs range selection is start inclusive and end exclusive, but I assume for a Map it will need to be start inclusive and end inclusive.

@jonjove reported in Discord (ref) that we had some extern host fns overlapping which he fixed in #185.

I double checked and tried changing functions to use the same link names and the Rust compiler doesn't error at all.

This seems like quite a footgun. Of course if this happens with functions that behave wildly differently we're likely to notice. But for functions that behave somewhat similarly this could go unnoticed but have an enormous impact.

The host functions are generated by some macro rules in the common crate. We should find some way to ensure that their names are all unique.

cc @jayz22 @graydon @jonjove

What

Add const fns for safe fast comparisons of statics.

Why

There are a few places in the SDK I noticed decent code size jumps just for checking if a RawVal was a true/false static. Statics are convenient in that the entire RawVal is static for that value, and so it is reasonably wasteful if we're doing tag shifting, then value extracting, then Result creating, then Result comparing, when in most cases we would be better served by each static having a corresponding RawVal::is_[static](&self) const function.

https://github.com/stellar/rs-stellar-contract-env/blob/633ff92861920ca98d45262e541d3dfeb2bf8611/stellar-contract-env-host/src/host.rs#L820-L824

Symbol, and any other type we define as a TaggedVal or with a RawVal cannot be used in a match statement.

They can't be used in match patterns because the RawVal type is not #[derive(Eq, PartialEq)], and types must derive those traits to be used in pattern matching. It's not sufficient that we implement the traits manually, they must be derived.

This has an unfortunate affect that mapping we need to do on Symbol's and other primitive non-object RawVal/TaggedVal types must occur within a set of if blocks, which Rust does not optimize as well as it does match blocks.

For a small match block of only 3 conditions this causes a Symbol mapping to be an extra 37 bytes in compiled WASM. Given that we use Symbols as constants in enum encoding, this problem seems to be worth solving.

Related discussion: https://discord.com/channels/897514728459468821/996186142007369778

We discussed this previously and I think it fell through teh cracks. We'll be using Vecs in situations where the size of the vec will be known ahead of time and probably very small. It will be ideal if the SDK can inform the host of what that size will be so it can be used as the capacity.

The vec_new fn should be changed to accept a capacity as a u32. If the STATIC_VOID value is passed the host should assume no value has been passed.

It'll be easier to make this change now.

cc @graydon @jayz22

Some tests like invoke_single_contract_function have instructions on where to find the source code how to regenerate the WASM contract, but some other tests don't. This information should be filled in for every contract used in the test cases.

• JS
• Java
• Go
  And add necessary host functions to facilitate that.
  Ref to discord discussion.

Currently the im library does a small-vector optimization, making small vecs up to 8 Vals use a single small 128 byte allocation with no substructure-sharing. It does not do a similar optimization for small maps, so an empty map uses about 2700 bytes. We need to put in a small-map optimization too, probably using something simple like https://lib.rs/crates/vector-map or https://lib.rs/crates/sorted_vector_map or https://lib.rs/crates/litemap or whatever

What

Host should reject ScMaps that contain duplicate keys.

Why

Maps with duplicate keys are an indication of bad/wrong input. They're a footgun because they are unlikely to have been signed correctly. They will also not round-trip serialize, so the input and output of a contract fn that passes the structure straight through will not be identical XDR bytes.

Related to #221 stellar/rs-stellar-xdr#98

cc @graydon @jonjove @jayz22

Supersedes #221 #222

Dependent on stellar/rs-stellar-xdr#106

cc @graydon @jonjove @jayz22

stellar/stellar-protocol#1257 (comment)

#147

cc @jayz22

What problem does your feature solve?

Unary tuples cannot be converted to Vec right now. See https://discord.com/channels/897514728459468821/994655698967339029.

What would you like to see?

Support this operation.

What alternatives are there?

Use the new vec! macro instead.

We should add a impl TryFrom<ScVal> for RawVal that can convert an ScVal to a RawVal. It wouldn't be able to support all types but it could support all but the SCV_OBJECT arm.

For the XDR types that parallel types that are RawValConvertible we should also add a impl From<_> for RawVal. This would be for any arm that has a non-builtin type. i.e. SCV_STATIC, SCV_SYMBOL, and SCV_STATUS.

We can also implement conversions in the opposite direction and most will be symmetrical, except the SCV_POS_I64 arm that would not be able to convert the other way for negative values of int64.

@jonjove asked for how to do these conversions in a meeting today. @graydon suggested we should have this in the common package.

cc @jonjove @graydon

What

Change the arguments for the vec_slice fn from start,length to start,end.

Why

In many programming languages slicing a vector takes a start and end index, not a start and length. This is true in Rust too. Because the fn accepts a start and length, the guest code has to take an end and compute the length, and then the host is taking the length and computing the end:
https://github.com/stellar/rs-stellar-contract-env/blob/566bb4a833fd675cb640cdf3ecfab4abad726b8b/stellar-contract-env-host/src/host.rs#L1194-L1200
The conversion back and forth is unnecessary.

Related to stellar/security#229

nonce is included but not actually a parameter

The current proposal in CAP-51 having both of them as object types, which provides guarantees on the correct length and provides conversions to and from binary. They exist in the xdr and in the code, but they aren't used and instead custom conversion code like this exists.

There have been discussion of alternatives (e.g. see https://discord.com/channels/897514728459468821/979885313377845278/989628715417882654).

The use of hash and public key should be migrated to the host object type or the alternative.

What did you expect to see?

serialize_to_binary and deserialize_from_binary should operate on values. Their implementations should match CAP-0051.

What did you see instead?

They operate on objects, and deserialize_from_binary does not have the signature specified in CAP-0051 (the CAP is wrong).

#142 (comment)

What problem does your feature solve?

Host function exports are split into many small, similar modules.

What would you like to see?

Similar modules merged to make more efficient use of the available symbol space.

What alternatives are there?

N/A

stellar/rs-soroban-sdk#187 (comment)

Also, map functions currently are missing tests.

Contract code is just contract data with a LedgerKeyContractCodeWasm static key, so it can be used along with put_contract_data and del_contract_data to update and delete contract code. We should update those host functions so they trap if they are called with LedgerKeyContractCodeWasm.

Given that any &'static str in guest linear memory has static address, we can write a variant of the log host function that is typed as &'static str on the SDK side and passes a (u32,u32) pair denoting a linear memory span to the host. The host can then store that pair plus a refcount to the wasm module / vm in the debug log, and materialize the string when dumping the log just by fishing it out of the wasm module. This should let users log arbitrary static strings, including such delights as file names, module names, function names. Should be able to build a useful contract_trace!() macro out of it.

This information is currently spread between cap46/cap51/code
The intent is to enable contract development SDKs in non-rust programming language.

Add STATIC_NONE static type for use with Option<T>.

The static type needs extending to include a STATIC_NONE value.

The value will be used by the Option<T> when a type can be stored as a type or a STATIC_NONE.

Related discussion: https://discord.com/channels/897514728459468821/982314893237714967

The Host VM exposes a list of exports which the CLI uses in its inspect sub-command. It would be useful if the Host VM could expose a list of imports which the CLI would use in its inspect sub-command.

Started in #207 (and overlaps/supports #205 a bit) but there's still a lot to do:

• Add mechanism to get diagnostics out of the system at useful times (testing, deployment)
• Revise the HostError type to just wrap an ScStatus (maybe with an optional backtrace on platforms that support it?)
• Revise all the code that generates HostErrors to emit debug events and generate simpler HostErrors

Currently the Status type reuses a major/minor structure that we also use for object index/type pairs: a 28 bit status-type and a 32 bit status-code.

This is plausible, but it's also not necessarily the best representation of Status values, and we've been feeling a lot of pressure to pack more information into Status values in at least a few places:

• Conversion errors want to pack at least a from-type and to-type into a status value.
• Debug / trace logging would benefit from packing some or all of a file name, line number and user-provided static string into a status.
• User-specified abort Statuses (as in #146) could also benefit from these things.

I'm thinking it might be nice to decompose the 60 bits available to Status values a little differently. For example: assuming WASM binaries in the current design are capped at 64KB, we can assume any linear-memory address (such as those for string constants) fits in 16 bits. Even if we are being more future-proof-y and saying 1MB / 20 bits, we could easily do a 4-field split: 4 bit type, 16 bit code-or-line-number, and 2 20-bit string constants (for filename and user message). Or keep the type space a bit bigger, stick with 64k for addresses, and go 12-16-16-16. Or go with 256KB / 18 bit addresses, and do 8-16-18-18.

We might even be able to figure out a way to build a 1-level indirection table for string constants -- we really only care about the start positions, not all the sub-positions within them -- so we can denote the start positions by table index rather than resolved address, and drop to (say) 10 bit indexes -- not likely anyone will have a contract with more than 1024 files.

(We could also do something even more elaborate and add a status object type, maybe that we overflow to, but I don't know if that'd be any better.)

We need four new host fns that supports the guest asking if:

• all elements in the vec match a specific type
• all keys in the map match a specific type
• all values in the map match a specific type
• all keys+values in the map match a specific pair of types

There could arise interesting bugs, maybe even an exploit, with the fact that vec and map's elements from the host perspective are untyped, but from the contract developers, they are, and the fact that in the SDK we are incentivized to not iterate an entire Vec/Map to check each elements types.

For example:

A developer writes a contract that accepts as an input an SCO_VEC. The contract converts the RawVal into a Vec, reads the first element successfully, then stores the remainder for later use. A later contract call reads the remainder, attempts to read out the next element, but the contract panics because the second element is actually an i32. The contract might be in a broken state and unusable from this point. A classic case of surprising behavior.

The panic and surprise occurs because we lazily inspect elements of a Vec. We don't inspect all the elements of the Vec when it is first converted from the RawVal, because that could be a waste of resources, especially with a huge Vec.

Same issues arise with a Map.

Related discussion:
https://discord.com/channels/897514728459468821/935999748840751125/982174401783029780

We should add comments to the Env trait fns as we implement them because some of their behaviors are ambiguous knowing only the fn name.

What problem does your feature solve?

The name here confuses me. I don't understand the intention. Based on the code, it looks like we are only using Frame::HostFunction as the initial frame when calling in via invoke_function (this behavior is fine). But the name suggests that we would push a Frame::HostFunction every time a host function is called, which we are not doing (this behavior is also fine). Perhaps we can come up with a name that indicates the actual usage?

What would you like to see?

A better name that indicates the actual usage.

What alternatives are there?

Leave the name as is, with some comments about what it actually means. I don't feel great about this though.

Some cost models make more sense when viewed in terms of more than one independent variable (such as different sizes of separate inputs to a function that uses them both in a subtle way). Currently we funnel all inputs to cost models into a single input variable, and in many cases that might work (sum all input sizes, take the max of all input sizes, multiply one input size by another, etc.) but in other cases it might make more sense to treat the cost model as a function of separate input dimensions.

I've noticed a couple of issues with the binary host functions.

1. Strange implementations of binary_front and binary_back https://github.com/stellar/rs-stellar-contract-env/blob/b15d15c447eb61ed06fa0e76cac2fd31608bdeea/stellar-contract-env-host/src/host.rs#L1551-L1575
   a. These should use Vec::front and Vec::back instead of indexing, and this avoids the need for explicit empty checks
   b. The error messages indicate "u32 overflow" which is not correct

2. Following up on the comment #147 (comment), it turns out that this same block of code actually appears many times across other functions too. Let's factor that out.

Let's try to clean this up in general.

What problem does your feature solve?

CAP-54 describes a native contract, but we don't have any support for this yet.

What would you like to see?

Support for native contracts.

What alternatives are there?

None.

The current budget.rs cost model is inadequate for serious use. It needs:

• To be able to load a batch of parameters from CONFIG_SETTING LEs
• To use a more uniform and lightweight interface for defining multiple cost centers (currently one has to define a number of new variables and functions in a number of structures and traits)
• To handle cost functions that are parameterized by constants but have nonlinear relationships to inputs (eg. charging log_16(sizeof(map)) CPU for a map lookup)
• To handle systematic/automatic calibration of costs by measurement -- both taking measurements over a range of input values and then fitting cost-function parameters to those measurements

What

Host should reject ScMaps that are out-of-order.

Why

Out-of-order maps are an indication of bad/wrong input. They're a footgun because they are unlikely to have been signed correctly. They will also not round-trip serialize, so the input and output of a contract fn that passes the structure straight through will not be identical XDR bytes.

Related to #222 stellar/rs-stellar-xdr#98

cc @graydon @jonjove @jayz22