neo-project / neo-vm Goto Github PK

When syncing from the MainNet chain file (as obtained from http://sync.ngd.network/) on multiple occasions, I randomly get exceptions to occur when deducting NEO balances (see neo-project/neo#391).

I thought it was gone after moving to the latest VM since I was able to sync once without an issue after moving to latest commit 94a91ae6 . I was originally on commit ab74c361. However, upon syncing the chain again, it happened again.

I added a richer log to see a bit more information about what was happeining. Here is an example of what I got on two separate occasions syncing the chain:

2018-09-25 19:14:41.118 F  Neo.Ledger.Blockchain: Account (ANcEb4Xw8ouJnk5e5Q1z6o92Z67pNrydQ2 = 0x1e9ca763049717f04952a1c0118678098a33f64a) doesn't have a balance for asset 0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b 1 Block: 1858860
System.Collections.Generic.KeyNotFoundException: The given key '0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Neo.Ledger.Blockchain.Persist(Block block) in [redacted]/neo/Ledger/Blockchain.cs:line 532
...
...

2018-09-26 02:41:58.910 F  Neo.Ledger.Blockchain: Account (AWnA7E3tNpc3vtYRWdxkHYiRBBu2mSsAYR = 0xf23af39524ce9e711eaac979a015e384a7a097a4) doesn't have a balance for asset 0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b 1 Block: 2582510
System.Collections.Generic.KeyNotFoundException: The given key '0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Neo.Ledger.Blockchain.Persist(Block block) in /home/solinsky/repos/aph-server/neo/Ledger/Blockchain.cs:line 532
[ERROR][9/26/18 2:41:58 AM][Thread 0010][akka://NeoSystem/user/$a] The given key '0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b' was not present in the dictionary.
Cause: System.Collections.Generic.KeyNotFoundException: The given key '0xc56f33fc6ecfcd0c225c4ab356fee59390af8560be0e930faebe74a6daff7c9b' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Neo.Ledger.Blockchain.Persist(Block block) in [redacted]/neo/Ledger/Blockchain.cs:line 532
   at Neo.Ledger.Blockchain.OnImport(IEnumerable`1 blocks) in [redacted]/neo/Ledger/Blockchain.cs:line 250
   at Neo.Ledger.Blockchain.OnReceive(Object message) in [redacted]/neo/Ledger/Blockchain.cs:line 413

My best theory so far is that something happening during execution of a script was non-deterministic leaving the balance of an account in a state different from what it was at the time the consensus node forged the block using a previous version of the VM. This is a theory that needs to be confirmed. It has happened on different hardware now, so I don't think it is faulty hardware, and I'm using the same version of LevelDB I have been using for a while before having encountered this.
This is running Neo 2.9.0, maybe I should try against 2.8.0 to see if it ever happens with latest VM to see if my assumption that it is VM related is correct.

Anyway, the only difference to my PersistBlock function that threw the above exceptions is the exception handling. I added comments to the PersistBlock function also to aid in thinking through what might be happening. I'll include the top part of the function that I modified to give more info here in case someone else has more time to look into it.

This code starts on line 439 of Blockchain.cs:

        private void Persist(Block block)
        {
            using (Snapshot snapshot = GetSnapshot())
            {
                snapshot.PersistingBlock = block;
                snapshot.Blocks.Add(block.Hash, new BlockState
                {
                    SystemFeeAmount = snapshot.GetSysFeeAmount(block.PrevHash) + (long)block.Transactions.Sum(p => p.SystemFee),
                    TrimmedBlock = block.Trim()
                });
                foreach (Transaction tx in block.Transactions)
                {
                    snapshot.Transactions.Add(tx.Hash, new TransactionState
                    {
                        BlockIndex = block.Index,
                        Transaction = tx
                    });
                    snapshot.UnspentCoins.Add(tx.Hash, new UnspentCoinState
                    {
                        Items = Enumerable.Repeat(CoinState.Confirmed, tx.Outputs.Length).ToArray()
                    });
                    foreach (TransactionOutput output in tx.Outputs)
                    {
                        AccountState account = snapshot.Accounts.GetAndChange(output.ScriptHash, () => new AccountState(output.ScriptHash));
                        if (account.Balances.ContainsKey(output.AssetId))
                            account.Balances[output.AssetId] += output.Value;
                        else
                            account.Balances[output.AssetId] = output.Value;
                        if (output.AssetId.Equals(GoverningToken.Hash) && account.Votes.Length > 0)
                        {
                            foreach (ECPoint pubkey in account.Votes)
                                snapshot.Validators.GetAndChange(pubkey, () => new ValidatorState(pubkey)).Votes += output.Value;
                            snapshot.ValidatorsCount.GetAndChange().Votes[account.Votes.Length - 1] += output.Value;
                        }
                    }
                    
                    // Iterate all tx inputs grouped together by common input transactions.
                    foreach (var group in tx.Inputs.GroupBy(p => p.PrevHash))
                    {
                        // Get the transaction state in common for the group of transactions to process.
                        TransactionState tx_prev = snapshot.Transactions[group.Key];
                        
                        // For each input from this transaction.
                        foreach (CoinReference input in group)
                        {
                            // Mark the UTXO for the input as spent
                            snapshot.UnspentCoins.GetAndChange(input.PrevHash).Items[input.PrevIndex] |= CoinState.Spent;
                            TransactionOutput out_prev;
                            try
                            {
                                // Get the output from the the previous transaction that is now being spent.
                                out_prev = tx_prev.Transaction.Outputs[input.PrevIndex];
                            }
                            catch (Exception ex)
                            {
                                logger.Fatal($"Can't find the output being spent from tx hash: {input.PrevHash} index: {input.PrevIndex}{Environment.NewLine}{ex}");
                                // This really should never happen or something is bad wrong!
                                throw;
                            }

                            // Get the account for the address that owned the output being spent.   
                            AccountState account = snapshot.Accounts.GetAndChange(out_prev.ScriptHash);
                            
                            // If governing token was spent
                            if (out_prev.AssetId.Equals(GoverningToken.Hash))
                            {
                                // Keep track when this transaction input was spent to be able to calculate claims.
                                snapshot.SpentCoins.GetAndChange(input.PrevHash, () => new SpentCoinState
                                {
                                    TransactionHash = input.PrevHash,
                                    TransactionHeight = tx_prev.BlockIndex,
                                    Items = new Dictionary<ushort, uint>()
                                }).Items.Add(input.PrevIndex, block.Index);
                                
                                if (account.Votes.Length > 0)
                                {
                                    // Reduce number of votes selected validators have due to the voter's reduced
                                    // governing token blance.
                                    foreach (ECPoint pubkey in account.Votes)
                                    {
                                        ValidatorState validator = snapshot.Validators.GetAndChange(pubkey);
                                        validator.Votes -= out_prev.Value;
                                        if (!validator.Registered && validator.Votes.Equals(Fixed8.Zero))
                                            snapshot.Validators.Delete(pubkey);
                                    }
                                    snapshot.ValidatorsCount.GetAndChange().Votes[account.Votes.Length - 1] -= out_prev.Value;
                                }
                            }

                            try
                            {
                                // Reduce the balance of the asset by the account that owned the asset being spent
                                // by the the number of units of the asset that were spent. 
                                account.Balances[out_prev.AssetId] -= out_prev.Value;
                            }
                            catch (Exception ex)
                            {
                                logger.Fatal($"Account ({account.ScriptHash.ToAddress()} = {account.ScriptHash}) doesn't have a balance for asset {out_prev.AssetId} {out_prev.Value} Block: {block.Index}{Environment.NewLine}{ex}" );
                                // This really should never happen or something is bad wrong!
                                throw;
                            }
                        }
                    }
...

I propose that we adopt an extended multisig on neo 2.x and 3, that allow verifying witnesses as an alternative to pubkeys. We can validate size, if 20 bytes then getmessage and verify, otherwise must be pubkey.
This simple change allows many.important elaborate changes, for example, consensus nodes having surrogate backups that interoperate on network.
I think this is simple to change, but opinions are very appreciated.

Example:
push2 (limit)
push3 (count)
push20 ...
push33 ...
push20 ...

This is a minimal example to show the issue of inconsistent returns for BigIntegers.

I came across this while making a modification to a NEP5 'balanceOf' implementation where wallets are expecting a ByteArray return for the value.

/*
test1: Evaluation Stack: [{"type":"Array","value":[]},{"type":"ByteArray","value":"40420f"}]
test2: Evaluation Stack: [{"type":"Array","value":[]},{"type":"Integer","value":"999900"}]
test3: Evaluation Stack: [{"type":"Array","value":[]},{"type":"Integer","value":"999900"}]
test4: Evaluation Stack: [{"type":"Array","value":[]},{"type":"ByteArray","value":"dc410f"}]
*/
using Neo.SmartContract.Framework;
using Neo.SmartContract.Framework.Services.Neo;
using System.Numerics;

namespace Neo.SmartContract
{
    public class Contract1 : Framework.SmartContract
    {
        public static object Main(string operation, params object[] args)
        {

            switch (operation)
            {
                case "test1":
                    return Test1();
                case "test2":
                    return Test2();
                case "test3":
                    return Test3();
                case "test4":
                    return Test4();
            }
            return false;
        }

        public static BigInteger Test1()
        {
            BigInteger blah = 1000000;
            return blah;
        }

        public static BigInteger Test2()
        {
            BigInteger blah = 1000000;
            BigInteger blah2 = blah - 100;
            return blah2;
        }

        public static byte[] Test3()
        {
            BigInteger blah = 1000000;
            BigInteger blah2 = blah - 100;
            return blah2.AsByteArray();
        }

        public static BigInteger Test4()
        {
            BigInteger blah = 1000000;
            BigInteger blah2 = blah - 100;
            Storage.Put(Storage.CurrentContext, "asdf", blah2);
            return Storage.Get(Storage.CurrentContext, "asdf").AsBigInteger();
        }
    }
}

I think that is not necessary allow execute ' RET' command while only push is allowed

What do you think @erikzhang ?

We propose the implementation of a modular exponentiation opcode for NeoVM.
The reason is that some dapp may require implementing RSA cryptography (which is our case on SciChain) and this is also fundamental for other number theory implementations, including zero-knowledge.

Currently, BigIntegers are limited to 32-bytes, but modular exponentiation may require a bit more than this, so one solution is to threat inputs and outputs as bytearrays (currently limited to 1024 bytes).

The initial proposal for MODPOW is to calculate: x = mul*base^{exponent} mod p
Values base, exponent, p and mul could be possibly passed in two options: (i) as an array [base, exponent, p, mul]; (ii) four independent elements taken from stack.
I prefer option (i), and it would be nice to allow mul to be optional and naturally assume value 1.
If exponent is negative, it would be nice to have the calculation of the modular inverse.
All elements would be converted to BigInteger for the calculation (from bytearray), and the final result would be returned as bytearray (thus, not strictly violating BigInteger limits on NeoVM).
On C#, BigInteger ModPow could do the exponentiation and ModularInverse function could do the inverse.

The mul could be eliminated by replacing it by two quadratic exponents with a subtraction and division (as Ethereum did on EIP 198), however that would require us to allow subtraction and division of huge numbers too (over BigInteger limit). We need to see if this mul already solves most general problems involving modular exp applications.

To establish a fair price for the operation it would be nice to have some time experiments and see the actual impact, but it won't be more costly than VERIFY itself.

MODPOW opcode could possibly have opcode 0xb0, although this is a discussion for later.
MODPOW could be acompanied later with another interesting operation which is scalar multiplication for elliptic curve points (possibly opcode 0xb1). This would allow us to implement CHECKSIG, CHECKMULTISIG and VERIFY directly on NeoVM, allowing users to switch cryptographic curves if they desire.

It would be a pleasure to have some voices of the crypto specialists in the community, in order to ensure a nice proposal (we can also create a NEP for this, if necessary).

Why is this not in ByteArray?

The VM is entering a FAULT state when a SETITEM is used on my byte array. See below:

case OpCode.SETITEM:
   {
         StackItem newItem = EvaluationStack.Pop();
         if (newItem.IsStruct)
         {
            newItem = (newItem as Types.Struct).Clone();
         }
         int index = (int)EvaluationStack.Pop().GetBigInteger();
         StackItem arrItem = EvaluationStack.Pop();
         if (!arrItem.IsArray)   // <------ the isArray check
         {
            State |= VMState.FAULT;  // <-- :(
            return;
         }

@erikzhang

I think that if you call this SYSCALL in the first script, you put a null value into the stack, maybe this is the unique point when you can add a null value to the stack, maybe we can fix it? if don't other VM like c++ VM should create a new StackItem (NullStackItem) because could produce different behaviours

I want to move the limits (stack,BigInteger..) and Gas costs from ApplicationEngine to this project, NeoVM.

Benficts:

• This could help with unit testing in this project
• Speed up the VM because we need to check twice now
• Centralize the VM procedures
• Help projects that want to use the official neoVM with the current configuration.

If someone want to use the NeoVM, with other limits, they could configure before use it according to his necessities.

I am interested in exploring using this VM in an internal project at Google, but I know from experience that our lawyers will push back pretty hard if I try to import MIT licensed code, especially code in a hot space where there are lots of patents being filed. The MIT license does not grant us any license to patents that might be filed by the authors of this code, and I've been told by lawyers that this creates an unacceptable risk before. However, IANAL, so I could be wrong...

Any chance of re-licensing or dual-licensing to something that will give potential users more confidence that they can use this code without getting sued?

We need to have some sort of interface for ExecutionEngine, in order to better test features inside Neo without needing to have a full system set-up. If we have that interface, we can use mock and test only the desired features when necessary (example, verify if a given Interop actually pushes on stack the expected value).

Perhaps I'm unclear on exactly how HASKEY should work with Array StackItems, but I do not think this line is correct:

https://github.com/neo-project/neo-vm/blob/master/src/neo-vm/ExecutionEngine.cs#L890

Should it be something like this instead:

                             case VMArray array:
                                    int index = array.IndexOf(key);
                                    if (index < 0)
                                    {
                                        State |= VMState.FAULT;
                                        return;
                                    }
                                    context.EvaluationStack.Push(index < array.Count);
                                    break;

I think that codecov is computing wrong the coverage on ExecutionEngine class.

Opcodes like CHECKSIG or CHECKMULTISIG are not tested, you can verify that Crypto methods are without unit tests https://codecov.io/gh/neo-project/neo-vm/src/master/tests/neo-vm.Tests/Types/Crypto.cs#L41 but in the switch https://codecov.io/gh/neo-project/neo-vm/src/master/src/neo-vm/ExecutionEngine.cs#L860 it appears as tested

Also in the readme, the badge appears without percentage

A StackItem of type Boolean with a false state returns 0 when calling GetByteLength() on it. This seems counter intuitive as false is still a value and takes up space in memory. Should the Boolean StackItem type not always return a length of 1?

My rationale is; I expect GetByteArray() to return 1 byte set to 0 for false and set to 1 for true, this would then result in GetByteLength() to always be 1. Neither of these assumptions is true (meaning; GetByteArray() on a Boolean with a false state, returns an empty bytearray). If this is intentional, why is it like that?

I'm currently working on a TypeScript/JavaScript -> NEO VM compiler and one thing that would be really handy is a native hash table type. While it's certainly possible to implement a hash table on top of the existing array functionality, it would be very script-space inefficient and GAS inefficient, requiring many op codes to properly access and manipulate the hash table. To that end, I propose we add 2 new types to the VM:

• MapStackItem which functions similarly to the ArrayStackItem
• DictStackItem which functions similarly to the StructStackItem (that is, cloning in the same places StructStackItem is cloned).

Spec

Terms:

• array-like: Either a ArrayStackItem or StructStackItem
• map-like: Either a MapStackItem or a DictStackItem
• stack item: An arbitrary stack item
• key: A stack item that can be used as a key. Only allows non-complex stack items, i.e. not an ArrayStackItem, StructStackItem, MapStackItem, or DictStackItem.

New

• OpCode.NEWMAP: Pushes a new MapStackItem onto the stack. Note that unlike NEWARRAY, an initial size is not set because it does not make sense in this context.
• OpCode.NEWDICT: Pushes a new DictStackItem onto the stack. Similarly, does not have an initial size.
• OpCode.PICKMAPITEM: Pops a key item key off the stack. Pops a map-like item map off the stack. Pushes map[key] onto the stack. If key does not exist, sets state to FAULT.
• OpCode.SETMAPITEM: Pops a stack item value off the stack. Clones the value if it is a StructStackItem or DictStackItem. Pops a key key off the stack. Pops a map-like item map off the stack. Sets map[key] to value.
• OpCode.REMOVEMAPITEM: Pops a key item key off the stack. Pops a map-like item map off the stack. Deletes map[key]. If key does not exist, does nothing.
• OpCode.HASMAPKEY: Pops a key item key off the stack. Pops a map-like item map off the stack. Pushes true onto the stack if key is present in map, otherwise pushes false.
• OpCode.PACKMAP: Pops an integer item length off the stack. Creates a MapStackItem and sets the items by popping a key item key off the stack and a stack item value off the stack length times.
• OpCode.UNPACKMAP: Pops a map-like item map off the stack. Unpacks map onto the stack in the same fashion that OpCode.PACKMAP expects.

Changed

• OpCode.ARRAYSIZE: modified to get the size of a map-like item.
• OpCode.APPEND, OpCode.SETITEM: modified to additionally clone DictStackItems.

Optional

These are nice to have, but not necessary given the above operations. However, they would help make certain operations more efficient and easier to manage, in particular OpCode.MAPENTRIES and OpCode.ZIPMAP.

• OpCode.MAPENTRIES: Pops a map-like item map off the stack. Pushes an ArrayStackItem that is an array of tuple ArrayStackItems with [key, value] onto the stack. I.e. Array<[key, value].
• OpCode.MAPKEYS: Pops a map-like item map off the stack. Pushes an ArrayStackItem of the keys of map.
• OpCode.MAPVALUES: Pops a map-like item map off the stack. Pushes an ArrayStackItem of the values of map.
• OpCode.ZIPMAP: Pops an array-like item keys off the stack. Pops an array-like item values off the stack. Zips them together and pushes a MapStackItem onto the stack.
• OpCode.UNZIPMAP. Pops a map-like item map off the stack. Unzips the map in the same fashion that OpCode.ZIPMAP expects.

Details

One minor detail is the implementation will need to add hashCode functionality and a custom comparator in order to have StackItem keys. equals, which would also be required, is already implemented.

We propose an operation for Elliptic curve point scalar multiplication and (possibly) addition. The idea is to allow onchain implementation of zero knowledge and other cryptography techniques, allowing users to switch prefered curves.
This is related to MODPOW issue, and also could be interop, if community prefers.
The intented syntax is to receive curve parameters on a input array on stack, followed by point (array of two items) and scalar (another bytearray).output will be a point (array of two elements).

It will be immensely powerful if it supports even this most basic GPGPU op codes.

The ScriptBuilder class uses System.BitConverter to convert integer types to bytes. The output of the BitConverter.GetBytes method will be different depending on the CPU architecture (little endian vs big endian).

else if (data.Length < 0x10000)
{
    Emit(OpCode.PUSHDATA2);
    ms.Write(BitConverter.GetBytes((ushort)data.Length), 0, 2);
    ms.Write(data, 0, data.Length);
}
else// if (data.Length < 0x100000000L)
{
    Emit(OpCode.PUSHDATA4);
    ms.Write(BitConverter.GetBytes((uint)data.Length), 0, 4);
    ms.Write(data, 0, data.Length);
}

Like PACK but for structs.

As discussed with community, it would be useful to have standard test format to guarantee cross-compatible integration tests between projects.
I believe current JSON tests are very nice, and I have started to build a validation JSON schema for it. By doing that, some small ideas emerged to fix and clarify details.

Right now, the following draft can already be validated:

{
  "category": "Arrays",
  "name": "ARRAYSIZE",
  "tests": [
    {
      "name": "Wrong type SYSCALL[System.ExecutionEngine.GetScriptContainer]+ARRAYSIZE",
      "script": "0x682953797374656D2E457865637574696F6E456E67696E652E476574536372697074436F6E7461696E6572C0",
      "message": "0x00"
    }
  ]
}

The proposed draft schema is:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "http://neo.org/neovm.json",
  "type": "object",
  "title": "NeoVM Tests",
  "required": [
    "category",
    "name",
    "tests"
  ],
  "properties": {
    "category": {
      "$id": "#/properties/category",
      "type": "string",
      "title": "test category (rename to 'suite' name?)",
      "default": "",
      "examples": [
        "Arrays"
      ],
      "pattern": "^(.*)$"
    },
    "name": {
      "$id": "#/properties/name",
      "type": "string",
      "title": "Name for test suite",
      "default": "",
      "examples": [
        "ARRAYSIZE"
      ],
      "pattern": "^(.*)$"
    },
    "tests": {
      "$id": "#/properties/tests",
      "type": "array",
      "title": "Tests for specific suite",
      "items": {
        "$id": "#/properties/tests/items",
        "type": "object",
        "title": "Each test case",
        "required": [
          "name",
          "script",
          "message"
        ],
        "properties": {
          "name": {
            "$id": "#/properties/tests/items/name",
            "type": "string",
            "title": "specific test name (rename to description?)",
            "default": "",
            "examples": [
              "Wrong type SYSCALL[System.ExecutionEngine.GetScriptContainer]+ARRAYSIZE"
            ],
            "pattern": "^(.*)$"
          },
          "script": {
            "$id": "#/properties/tests/items/script",
            "type": "string",
            "title": "specific test script",
            "default": "",
            "examples": [
              "0x682953797374656D2E457865637574696F6E456E67696E652E476574536372697074436F6E7461696E6572C0"
            ],
            "pattern": "^(0x)?([0-9A-F][0-9A-F])*$"
          },
          "message": {
            "$id": "#/properties/tests/items/message",
            "type": "string",
            "title": "specific test message",
            "default": "",
            "examples": [
              "0x00"
            ],
            "pattern": "^(0x)?([0-9A-F][0-9A-F])*$"
          }
        }
      }
    }
  }
}

Material on JSON schema may be found here: https://json-schema.org/understanding-json-schema
Validation can be experimented here: https://www.jsonschemavalidator.net

Right now, my only suggestions are to rename "category" to "suite" (or "testsuite"), and "name" to "description" (inside tests). I also suggest the inclusion of a field "version", to evolve this standard in the future. The part of "steps" is still unfinished, it will take a little work, but I think it can be finished very soon.

I'm porting the recent VM changes to neo-python's VM and noticed the following

This line is supposed to calculate the stack item count

Assume a setup as follows:

• 1 InvocationStack with 1 ExecutionContext
  • The ExecutionContext has 1 EvaluationStack with 0 items
  • The ExecutionContext has 1 AltStack with 1 item of type Array
    • The Array holds 3 StackItems.

I would think stackitem_count to be 3 (the 3 items in the array), but it counts Array as another item so it returns 4. Is this intended or should we ignore the actual Array and Map types in the count and only count their elements?

Erik, I think it's time to come back to this issue: #54 (comment)

Unified entry point for all verification methods. Together with this, we can abolish CHECKSIG and MULTISIG, for Neo 3.0, and implement them as Native contracts using VERIFY.

I'm here to propose an NVM Extension Proposals (like a NEP in fact), that will allow us to introduce new features, in the future NVM3, in a very controlled way.

I think that after NeoVM 3 is released, not much changes will happen on its opcodes in a near future, it should be very stable. However, there will always appear new ideas and extensions that would better fit here on NVM, than on the ApplicationEngine.

I propose a single new opcode EXTENSIONS , that receives two bytes as parameter, allowing up to 65k possible extensions, grouped by category (first byte), and another one for specifics (otherwise 0x00).

This will work like SYSCALLS, but in a very lightweight manner, and constrained to NeoVM project itself (not mentioning specific Neo Blockchain capabilities). In order to create an EXTENSION, a new NEP must be created declaring it, and after that, it could be implemented on NVM project.
Versioning is not proposed here, meaning that it is the responsibility of upper layer and dependent projects to care how they manage their NVM usage. The only thing NVM will provide is disable specific extensions, in a map format.

I think that NVM 3 fulfilled it's intention of being a lightweight virtual machine, however I think it should embed capabilities related to a verifiable execution engine, meaning that it should be able to embed some elaborate verifiable data structures, hashing and cryptography.

This should allow NVM to become a standard lightweight vm for blockchain and verifiable computation, as there's no such existing vm nowadays.

Therefore, my first proposal would be to embed the following extensions:

• Category 0x00: reserved (not used)
• Category 0x01: hashing
• Category 0x02: digital signatures

0x0101 -> SHA256
0x0201 -> secp256r1 verification

We could start a NEP with that.

[EDIT] The initial ideia was to create a new opcode called CONVERT. Then, after discussions with @shargon , I changed to a new direction, without creating any new opcode, just updating PACK and UNPACK.

=====

I think three different problems can be solved by having an opcode capable of converting byte[] to Array (of byte) and back.

1. REVERT only works on Array types: #17
2. SETITEM only works on Array types: #58
3. SIZE and ARRAYSIZE with be redundant: https://github.com/neo-project/neo-vm/blob/master/src/neo-vm/OpCode.cs

The idea of this opcode CONVERT is to allow a convertion between a reference type Array to a static type bytearray, and to allow byte[] to have easy use of operations that are available for arrays. All these operations could be done in high level, however, at a big cost of opcodes, and higher code complexity. Using this opcode, REVERSE could be easily done:

PUSH10 ababababab...
CONVERT # converts byte[] to Array (of byte) on stack
DUP # duplicates Array on stack
REVERSE  # reverse and drop array
CONVERT  # convert Array back to byte[] on stack

SETITEM will work in the same way.

Opcode SIZE could be easily replaced by ARRAYSIZE (as proposed by Erik on #57).

PUSH10 ababababab...
CONVERT # converts byte[] to Array (of byte) on stack
DUP # duplicates Array on stack
ARRAYSIZE # removes Array and puts its size on stack

/// <summary>
/// A bytearray or array (of byte) is taken from the stack. If bytearray is taken, an array with the same bytes is put on stack. If Array is taken, a bytearray is created with the exact size, having each element transferred from one to another.
/// </summary>
CONVERT = 0xCE,

The following code was successfully executed in the old version of Neo-gui and failed in the new Neo-gui.
Smart Contract Code

public static bool Main(string method)
{
    return method == "a";
}

invoke:
Function: Main
Parameters: a

Result:

This would allow ArrayStackItems, MapStackItems, StructStackItems and InteropInterfaces to be used as keys in MapStackItems, which would be very convenient for modeling smart contract language maps. E.g. in C# or JavaScript, MapStackItem would map 1:1 to builtin dictionaries and maps respectively.

I have a main method like this:

public static object Main(string operation, params byte[][] args)

and would like to prime args in a unit test. I've tried something like this:

byte[] program = // script is loaded here
var engine = new ExecutionEngine(null, Crypto.Default);
engine.LoadScript(program);

using (ScriptBuilder sb = new ScriptBuilder()) {
    sb.EmitPush(new byte[] { 1, 2, 3, 4, 5 });  // args[0]
    sb.EmitPush(1);
    sb.EmitPush(OpCode.PACK);
    sb.EmitPush("operation");  // operation

    engine.LoadScript(sb.ToArray());
}

engine.Execute();

This doesn't work but seems like it should.

The latest VM does not seem to be working properly with certain contracts. I traced this to the CALL_I instruction.

I believe that since the recent introduction of the Instruction class, the CALL_I jump target's offset is being calculated from the instruction pointer value immediately after the CALL_I is read.

But as I read it, in the previous implementation, the CALL_I jump target's offset would be calculated from the instruction pointer value after the rvcount/pcount operand bytes are read, i.e. two bytes further into the script, which is what the existing compiler expects.

So as a result, in the new VM, all CALL_I jumps are landing two bytes short of where they should be, which would make most contracts compiled using NEP8 malfunction on most operations.

@erikzhang This instruction shouldn't be a Peek? if you pop the key, value and array, where goes the result?

Is necessary to clone first the array for call this opCode?

Still remains

We can't perform SETITEM on ByteArray, but maybe we can perform PICKITEM on it.

Do you think so? @igormcoelho @shargon @lightszero @dicarlo2

Issue

An Address is converted to a ScriptHash when using neo-python, which means that the Address cannot be stored as a string within Storage.

Solution

Add ability to convert ScriptHash back to an Address.

A more uniform view of the number zero can be achieved by always represent it in its most compressed format. 0x0000 is zero, 0x00 is also zero, but 0x'' (empty array) is also zero. So we should adopt empty array as our new standard. Currently, the standard (by C# BigInteger) is "0x00" I think (although 0x"" is also recognized as zero...)
This helps in many things:

• PUSH0 now may push VM_Integer zero (as it is an empty array now, not 0x00, this eventually causes confusion during compiling...)
• In general, PUSHX should push a number (VM_Int), and PUSHF (which is currently equals to PUSH0), should push a VM_Boolean False.
  So, Number 0 as bytearray, will actually be an empty bytearray, as expected. Before this, I used to make some strange computations, like PUSH0 PUSH0 ADD, not being equals to PUSH0, for example. Now this wont happen anymore.

If someone wants the hexstring zero, it can do PUSHBYTES1 00, as always.

The NUMEQUAL instruction in the new 2.4.1 version of NEOVM adds a check for size, which causes problems with some contracts that have been deployed on the MainNet.
For example, the NNC contract (0xfc732edee1efdf968c23c20a9628eaa5a6ccb934). Because the NNC contract had an upgrade operation, in which there was a comparison of new and old contract scripts. this comparison was added to indicate that the upgraded contract is a different contract. Because the content of the comparison exceeds the size limit, it causes the 2.10.1 version of the node to return fault when it executes the contract. The upgraded NNC contract can not be found in the 2.10.1 version of the node that re-synchronized data.

We use structs heavily in NEO-ONE TypeScript smart contracts and we depend on the fact that structs are cloned from PICKITEM. So, we'd like to implement #49 again, however this time with a simple logic switch such that old contracts use the old buggy behavior and new contracts use the corrected behavior.

Ideally, we could just turn this feature on using the height the contract was created at, but we don't store this information anywhere. So we'll need an alternate way to determine if a given contract should use the new behavior or old. I suggest that we add a new contract property similar to payable, dynamic invoke, etc. which when present means that the contract uses the new behavior.

This is blocking for NEO-ONE TypeScript smart contracts.

Can we start adding release tags for this repo as well? It's becoming hard to know which commits work with which neo-cli version. Like now I want to crosscheck RPC output against neo-cli 2.10.0.0 after all the VM updates but that still returns multiple engine states as if it still were flags.

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
        "script": "00c1046e616d6567c2563cdd3312230722820b1681d30fe5f6cffbb9",
        "state": "HALT, BREAK",
        "gas_consumed": "0.128",
        "stack": [
            {
                "type": "ByteArray",
                "value": "4e45582054656d706c617465205632"
            }
        ]
    }
}

At now InteropInterface just accept IInteropInterface objects, but this cause a problem if you want to change the VM, because you can't share the same InteropService . This interface is defined inside the VM package, so we can't reuse the same InteropService for different implementations of the VM.

If we change this to a native object, or just accept any object, all should works for both scenarios

Guys, do you think that's ok to change line

https://github.com/neo-project/neo-vm/blob/master/src/neo-vm/ExecutionEngine.cs#L618-L619

I can make the PR, just wondering if that's too crazy :)

Many applications (so many!) could benefit of a jump-free execution mode, where execution is bound to the number of opcodes, at most. Could we provide such a way now? It could be a flag passed to the execution engine, e.g. nojumps. This would also require disabling recursive calls and perhaps all calls.

Hello,

I am using string.Concat in my smart contract.

internal static string Generate(string prefix, string entityName, byte[] publicKey)
        {
            return string.Concat(prefix, entityName, publicKey);
        }

        internal static string Generate(string prefix, string entityName, byte[] publicKey, BigInteger id)
        {
            return string.Concat(prefix, entityName, publicKey, id);
        }

Here the second method fails as BigInteger is used.

internal static bool Create(byte[] publicKey, string entityName, string name, BigInteger dd = 2)
In the above line I was not able assign default parameter to BigInteger, in this case compiler shows error.

Later:

internal static string Generate(string prefix, string entityName, byte[] publicKey)
        {
            return string.Concat(prefix, entityName, publicKey);
        }

        internal static string Generate(string entityName, byte[] publicKey, BigInteger id)
        {
            return string.Concat(entityName, publicKey, id);
        }

Removed one parameter from the method... Works good

Moved from this issue: neo-project/neo#140

Add ability to remove item from list.

Let me start by explaining my interpretation of the function named GetCallingScriptHash and perhaps we can already stop there if that's wrong.

When I think of GetCallingScriptHash then I believe that it will return the script hash of whoever is calling the smart contract this function is used in. 2 examples of that:

1. wallet 1 calls Contract A, and Contract A uses GetCallingScriptHash -> I expect the script hash of Wallet 1.

2. Contract A DynamicAppCalls Contract B, and Contract B uses GetCallingScriptHash -> I expect it to return the script_hash of Contract A.

If that is the expected behaviour then GetCallingScriptHash is currently limited to be called from the Main function of the dynamically called contract. The question is; is this intentional, a bug or something that needs enhancement?

Let me elaborate on that with an example

Contract A:
def Main():
   dynamicappcall -> contract B

Contract B:
def Main()
     function_x()

def function_x:
    GetCallingScriptHash()

The invocation stack frames are as follows;
0 -> the argument list provided on the console to the contract
1 -> Contract_A.Main
2 -> Contract_B.Main
3 -> Contract_B.function_x

Now; GetCallingScriptHash does InvocationStack.peek(1) (or basically takes the second last item of the invocation stack). That means it will return the script_hash of frame [2] which is Contract_B, not Contract_A (frame [1]) as the function name suggests.

Dear all, I've been thinking on this for a while, and I think now that a single new opcode can help solving many different issues.

I propose the creation of a new opcode, CONVERT <type>, that receives a byte <type> indicating the conversion type and converts element that is on top stack (putting the converted element on stack). I initially propose the following types:

0x00: ByteArray
0x01: Array
0x02: Integer
0x03: Struct
0x04: Map
0x05: Boolean
...
0x80: Clone

Applications:

1. we could REVERSE for bytearrays (opened at #17 and currently solved on PR neo-project/neo-devpack-dotnet#37, but could be much more efficient):

PUSH_bytearray
CONVERT 0x01 # convert to array
DUP
REVERSE
CONVERT 0x00 # convert to bytearray

2. We could perhaps transform array into structs and solve this: #51
   CONVERT 0x03 # convert stack array to struct

3. We could transform bytearrays to integers and guarantee that they would follow BigInteger standard (by summing zero for example).
   CONVERT 0x02 # convert to biginteger

4. We could convert maps and arrays, so repeated opcodes could be eliminated (including bytearray/array sizes, appends, ...)
   CONVERT 0x01 # convert to array and get length
ARRAYSIZE

5. We could do an efficient concatenation of array elements by transforming array to bytearray
   PUSH ...
PUSH ...
CAT
PUSH ...
CAT
####
PUSH ...
NEWARRAY
CONVERT 0x00 # convert to bytearray

6. We could perhaps add a CLONE type in conversion(0x80 + 0x03), so we could choose to convert into a new one or an actual clone (and help solving this #63)
   CONVERT 0x83 # convert to cloned struct

7. We could avoid this PACK/UNPACK workaround: #60
   CONVERT 0x01 # convert to array and assign value
... assign ...
CONVERT 0x00 # convert back to bytearray

8. We can use it to enforce alternative types such as Boolean and Byte (even if stored as bytearray).
   PUSH5
CONVERT 0x05 # to boolean (true if not zero)

Perhaps this feature could antecipate issues and bring many future possibilities on Neo 3.0 (and removing many unused opcodes), just by having a direct conversion tool for StackItems.

we should add this functions to opcode.

    //Syscall("System.ExecutionEngine.GetEntryScriptHash")
    ENTRYSCRIPT = 0xD0,
    //Syscall("System.ExecutionEngine.GetExecutingScriptHash")]
    CALLINGSCRIPT = 0xD1,
    //[Syscall("System.ExecutionEngine.GetCallingScriptHash")]
    EXECUTINGSCRIPT = 0xD2,

this is base function,like implement static variable.
static variable is shared by same script,so we need to compare executingscript

we can store static vars like this:
map<scripthash, static vars>

so when we use a static var,opcode maybe like:

DUPFROMSTATICSTACK，//get map
EXECUTINGSCRIPT, //get key :scripthash
PICKITEM //get vars=map[key]
PUSH var id //get key:var id
PICKITEM //get var=vars[key]

I think neocompiler use syscall to get executingscript for static variable is a bad idea.
syscall is more slower than opcode.
and this will make neocompiler to write a fix syscall.

Problem

The instructions DUPFROMALTSTACK and DUP are currently returning a reference of a StackItem. This is not how stack machines should work at their core, hence leaving some security vectors open as we are directly modifying a StackItem on another stack. Which breaks all rules of stack isolation.

We can see in the instruction APPEND that the VM is not pushing the array back to the ALT stack. It is modifying the memory on the ALT stack directly. reference append

Solution

Both DUPFROMALTSTACK and DUP should return a clone by popping the item of the ALT stack. Instead of creating a referenced copy of it.

Psuedo code

Currently 
dup => modify reference directly

Proposel
pop => modify clone => push back to ALT stack

If we take the APPEND instruction for example, after appending the item to the collection we need to push the collection back to the ALT stack. there are a couple other instructions that will need some changes to make this work. However, those changes are very minor for a major improvement.

This issue occurs also in other VM implementations (py, go, ..)

It seems to me that the description of the following opcodes does not match the code:

/// Reads a 2-byte value n and a jump is performed to relative position n-3.
        /// </summary>
        JMP = 0x62,
        /// <summary>
        /// A boolean value b is taken from main stack and reads a 2-byte value n, if b is True then a jump is performed to relative position n-3.
        /// </summary>
        JMPIF = 0x63,
        /// <summary>
        /// A boolean value b is taken from main stack and reads a 2-byte value n, if b is False then a jump is performed to relative position n-3.
        /// </summary>
        JMPIFNOT = 0x64,

Could you please clarify?

same remark for CALL:

/// Current context is copied to the invocation stack. Reads a 2-byte value n and a jump is performed to relative position n-3.
        /// </summary>
        CALL = 0x65,

we don't see any relative jump of n-3:

case OpCode.CALL:
                        {
                            if (!CheckMaxInvocationStack()) return false;
                            ExecutionContext context_call = LoadScript(context.Script);
                            context_call.InstructionPointer = context.InstructionPointer + instruction.TokenI16;
                            if (context_call.InstructionPointer < 0 || context_call.InstructionPointer > context_call.Script.Length) return false;
                            context.EvaluationStack.CopyTo(context_call.EvaluationStack);
                            context.EvaluationStack.Clear();
                            break;
                        }

Hello,
Is there any chance someone could document the OpCodes?

I have an additional question, if someone could answer, that would be great.
Why does the GoverningToken uses a PUSHT instruction and the UtilityToken uses a PUSHR instruction?
These are the transactions in the genesis block.

 public static readonly RegisterTransaction GoverningToken = new RegisterTransaction
        {
            AssetType = AssetType.GoverningToken,
            Name = "[{\"lang\":\"zh-CN\",\"name\":\"小蚁股\"},{\"lang\":\"en\",\"name\":\"AntShare\"}]",
            Amount = Fixed8.FromDecimal(100000000),
            Precision = 0,
            Owner = ECCurve.Secp256r1.Infinity,
            Admin = (new[] { (byte)OpCode.**PUSHT** }).ToScriptHash(),
            Attributes = new TransactionAttribute[0],
            Inputs = new CoinReference[0],
            Outputs = new TransactionOutput[0],
            Scripts = new Witness[0]
        };

        public static readonly RegisterTransaction UtilityToken = new RegisterTransaction
        {
            AssetType = AssetType.UtilityToken,
            Name = "[{\"lang\":\"zh-CN\",\"name\":\"小蚁币\"},{\"lang\":\"en\",\"name\":\"AntCoin\"}]",
            Amount = Fixed8.FromDecimal(GenerationAmount.Sum(p => p) * DecrementInterval),
            Precision = 8,
            Owner = ECCurve.Secp256r1.Infinity,
            Admin = (new[] { (byte)OpCode.**PUSHF** }).ToScriptHash(),
            Attributes = new TransactionAttribute[0],
            Inputs = new CoinReference[0],
            Outputs = new TransactionOutput[0],
            Scripts = new Witness[0]
        };

thanks

how to use it in vm ware project?

The source code for Neo-VM and Neo is not consistent.
For example:
1, EvaluationStack is defined in the ExecutionContext class of Neo-VM, but it is used in the EvaluationStack class of Neo.
2, the loadScript function prototype defined in Neo-VM is “public ExecutionContext LoadScript (byte[] script, int rvcount = -1)”, and the call in Neo is engine.LoadScript(script, false)
3， and there are many other errors.
Please upload the latest stable version of NEO's source code to Github, thank you!

People, I don't know if it's something we can improve on neo-vm, or here at the compiler, but I really think this code should work:

using Neo.SmartContract.Framework;
using Neo.SmartContract.Framework.Services.Neo;
using System;
using System.Numerics;
namespace NeoContract1   {
    public class Contract1 : SmartContract  {
        public static byte[] Main()    {
            byte[] b = new byte[]{0x01, 0x02, 0x03, 0x04};
            b[2] = 0x05;
            return b;
        }
    }
}

I generates the following AVM 00c56b0401020304765255c4616c7566, that means:

00 PUSH0  #An empty array of bytes is pushed onto the stack
c5 NEWARRAY  #
6b TOALTSTACK  # Puts the input onto the top of the alt stack. Removes it from the main stack.
04 PUSHBYTES4 01020304 # 
76 DUP  # Duplicates the top stack item.
52 PUSH2  # The number 2 is pushed onto the stack.
55 PUSH5  # The number 5 is pushed onto the stack.
c4 SETITEM  # bytearray[2] = 5
61 NOP  # Does nothing.
6c FROMALTSTACK  # Puts the input onto the top of the main stack. Removes it from the alt stack.
75 DROP  # Removes the top stack item.
66 RET  #

Operation SETITEM naturally fails, because it's only currently allowed for arrays or maps (not bytearray). I think it should be allowed on bytearrays, since the alternative to this is to do lots of substrings and concats (like this ugly solution: https://github.com/NeoResearch/learning-examples/blob/master/NonTrivialFunctions.md).

The problem I see is that it is applying a DUP because it's guessing ByteArray is a reference type, so collateral effects should apply "remotely", but that won't work on copy type bytearray. So, how can we fix this in a easier way? Allow SETITEM to update bytearray too (on neo-vm project) and make the compiler avoid the DUP in these cases (in this project)?

========

Just to mention that this code works:

            object[] obj = new object[]{0x01, 0x02, 0x03, 0x04};
            obj[2] = 0x05;

But since it works as native array, it pushes elements one by one, and it's quite hard to convert it to byte[] (requires a full loop and many concats). So, a direct solution for byte[] would be nice, perhaps avoiding generating the DUP and allowing SETITEM to push byte[] back to stack after updating it.

Moved from: https://github.com/neo-project/neo-compiler/issues/133

Like CAT, SUBSTR and LEFT, it would be nice to have an opcode for reversing a byte array. Useful for script hashes.