fudong1127 / pybtcengine Goto Github PK

################################################################################
#                                                                              #
#  Copyright (C) 2011, Alan C. Reiner    <[email protected]>               #
#  Distributed under the GNU Affero General Public License (AGPL v3)           #
#  See LICENSE or http://www.gnu.org/licenses/agpl.html                        #
#                                                                              #
################################################################################


********************************************************************************
*
* Project:  PyBtcEngine
* Author:   Alan Reiner
* Date:     11 July, 2011
* Donate:   1Gffm7LKXcNFPrtxy6yF4JBoe5rVka4sn1
*
* Descr:   PyBtcEngine is a full computational backend for Bitcoin software.
*          It combines C++ and Python to enable *all* the functionality 
*          necessary to write Bitcoin software, *except for networking*.
*          The networking is more application-oriented, whereas this library
*          is more focused on enabling you to read/interpret/construct the
*          packets you would be exchanging on the Bitcoin network.
*  
*          For a demo of all the features of the library, you can see the 
*          PyQt block-explorer I made in the pyqt directory.  If it doesn't
           autodetect your blkfile, you can provide it as a CLI argument
*
*          This library was originally based on Sam Rushing "caesure" code,
*          but is a complete rewrite and no longer resembles or includes any
*          of the code he wrote.  So, I have removed his license/copyright.  
*          Of course, he still gets a ton of credit for providing such good 
*          reference code.  "Lis" from the Bitcoin forums also deserves a ton 
*          of credit for writing a fantastic, dependency-less version of ECDSA.  
*          This may enable more platforms where this code can be used, as they 
*          don't need to have any particular crypto libraries installed
*
*          Note:  I am not a "real" programmer, I'm an engineer that writes 
*                 algorithms to solve problems really quickly.  There's a few 
*                 places I should've used inheritance but didn't.  And although
*                 I usually am good about const correctness, it turned into a 
*                 nightmare here, so I've mostly removed it
*
********************************************************************************

********************************************************************************
* Please take a moment to donate 1.0 BTC!    1Gffm7LKXcNFPrtxy6yF4JBoe5rVka4sn1
********************************************************************************

********************************************************************************
*  STATUS:   Last Updated - 28 Oct, 2011
*            Legend:   
*                      _    not implemented
*                      .    implemented but not tested
*                      +    implemented and partially tested
*                      X    implemented and tested
*  
*                                          C++       Python     SWIG
*     ---------------------------------------------------------------
*      (01)  Ser/Unser Block Objects        X          X         X
*      (02)  Hash160/Hash256                X          X         X
*      (03)  Difficulty calcs               X          X         X
*      (04)  Address Generation                        X         X
*      (05)  Address Verify/Manip                      X         X
*     ---------------------------------------------------------------
*      (06)  BlkHeaders read/scan/org       X          X         X
*      (07)  BlkHeaders reorgs              X                    X
*      (08)  Blockchain read/scan/org       X                    X
*      (09)  Blockchain reorgs              X                    X
*      (10)  Blockchain verify integrity    X                    X
*     ---------------------------------------------------------------
*      (11)  NonStd Tx Detection            +          +         +
*      (12)  Arbitrary script parsing       X          X         X
*      (13)  OP_CHECKSIG/CHECKMULTISIG      X          X         X
*      (14*) Arbitrary script eval                     X         X
*      (15)  ECDSA Sign/Verify              X          X         X
*     ---------------------------------------------------------------
*      (16)  Address/Wallet tracking        X          X         X
*      (17)  Scan blkchain for Tx           X                    X
*      (18)  Scan blkchain for NonStd       X                    X
*      (19)  Reorg w/ double-spend          X                    X
*      (20)  Add new blockdata real-time    X                    X
*     ---------------------------------------------------------------
       (20)  SelectCoins for tx                        X         X
*      (21)  Tx construct given inputs                 X         X
*      (22*) Distr Proposals/multisig                  
*      (23)  Tx broadcast
*      (24)  Tx fee calculations                       X         X
*      (25)  Blockchain download
*
*
*
*
* Notes:
*     (09)  I have THOROUGHLY tested the blockchain reorg code.  and making 
*           sure everything looks exactly right at each step.  After a reorg,
*           invalidated Tx's show up in the address ledger with a flag
*           that identifies it has been removed.  This is really important in
*           the event of oddities in the blockchain -- normally the transaction
*           would just disappear without evidence it ever existed, which can 
*           cause a lot of user confusion.
*           
*     (11)  Is "mostly" tested because there are a few TxIn scripts that I fail
*           to identify as "standard."  This is because it appears that a few 
*           TxIns (less than 50 of 1,500,000) have a couple extra bytes in the 
*           DER signature that throws me off.  I'm not sure whether I need to
*           turn around and reimplement the signauture reading based on a real
*           DER library, or just hack it to accept extra 0x00 bytes at the end.
*
*     (14*) I have tested a variety of very complicated scripts, mostly from
*           the testnet, including OP_CHECKSIG and OP_CHECKMULTISIG.  Both the
*           C++ and python code can parse all these scripts, and the python 
*           code can execute these scripts successfully.  HOWEVER, I have 
*           NOT implemented the OP_IF/OP_NOTIF/OP_ELSE/OP_ENDIF op-codes. 
*           These codes require a bit of work to get right, and not currently
*           part of any near-term isStandard() transaction types.  
*        
*
*     (22*) In anticipation of supporting multi-signature transactions, I have
*           created a specification for handling multi-sig transactions, which
*           also happens to work well for offline wallets/transactions.  I 
*           have called these "Distribution Proposals".  See the following BIP:
*              https://gist.github.com/1321518
*
*
* More Notes:
*      FullRAM implementation only in C++ code.  However,
*      modern computers can handle the full blockchain in RAM right now, and 
*      probably for the next year, so there's no hurry to do this.

*      Short/Medium-term future:
*            *** NETWORKING & BLOCKCHAIN DOWNLOAD ***
*            Deterministic address generation
*            Wallet serialization/storage to file
*            Convert wallet files to/from official BTC wallets (pywallet)
*  
********************************************************************************

********************************************************************************

   For information on dependencies and compiling, see Using_PyBtcEngine.README

********************************************************************************
#####
28 Oct, 2011

   *** Implemented SelectCoins and started "Tx Distribution Proposals."  

   Actually spent a bit of time writing the following specification instead
   of coding:
               https://gist.github.com/1321518

   I figured that ironing out a good way to implement it before I actually do,
   would be a good investment of time.

   Fantastically, this method of dealing with multi-signature transactions, 
   also works surprisingly well for offline transactions:  you can use the 
   exact same mechanism, as a 1-of-1 tx that needs to be signed.  The way 
   the TxDP works, this can be done by an offline computer without access to
   the blockchain, and thus only needs the ECDSA private keys.

   The only problem is, this complicates my TxDP code, because most TxDPs will
   only involve signing a single TxIn, whereas this is a "regular" transaction
   which will potentially have multiple TxIns.  It's going to get done, but it
   is extra complexity...


18 Oct, 2011

   *** Working blockchain reorgs with double-spend detection, GUI works in 10.04

   I added a createTestChain.py script that can be modified to create any kind
   of difficulty-1 blockchain, starting with the actual genesis block.  I have
   used it to create a reorg-double-spend test, and crushed some pretty heavy 
   bugs in that code.  Next I will be working on the Tx/Ledger handling, so
   that any transactions that were, at any point, marked valid, will remain in
   the DB/memorypool for some amount of time, to let the user know what 
   happened.

   Along the way, also had to iron out my tx-construction code in python.  It's
   a complete pain in the ass (endianness is a bitch), but it's finally done
   now and the python code can construct Txs from a list of Addr-Txout-pairs
   and a list of recipient-value-pairs.  (I do not have TxOut selection 
   implemented, but it should be straightforward compared to the ECDSA and
   OP_CHECKSIG stuff).

#####
06 Oct, 2011

   *** A working block-explorer demo in PyQt4!  (Linux/Ubuntu, only, right now)

   Got enough of the interface working in SWIG to be able to create a full 
   blockchain explorer.  It does not include anything to do with addresses,
   besides identifying the addresses involved in the particular block you
   are browsing.  In the immediate future I plan to include a separate
   tab/view for doing address searches and display.

   I had implemented a feature to auto-update the GUI in realtime if you
   are simultaneously running the BTC client and it updates blk0001.dat. 
   Unfortunately, there's a bug in the addBlockData() method which causes
   a segfault.  This feature will be back soon, but I think I need the help
   of MSVS to thoroughly test/debug addBlockData().

#####
25 Sep, 2011:   

   *** Address scanning, Tx ledgers, wallet balance, etc, ADDED
   
   The directory cppForSwig contains a C++ implementation of the block objects
   with everything that doesn't require a bignum library.  The goal was to 
   create C++ code that could be accessed via SWIG/python to do the heavy 
   lifting, since python isn't very good at doing it quickly/efficiently.
   
   At the moment, the BlockDataManager_FullRAM will store the entire blockchain
   in RAM, and do all calculations on it there.  I anticipate in the future, to
   convert this to a file-based approach with an initial scan.

   Below is a list of the operations the C++ code supports, as well as timings
   based on my decent AMD Phenom II X4 840 CPU and WD 1TB HDD, using a file
   containing the blockchain up to about block 145,000:  600 MB.

      -- Reading blockchain from disk:    5.4s
      -- Populating header and tx maps:  10.2s
      -- Organizing headers into chain:   0.6s
      -- Scan blkchain for 3 addrs' tx:   2.9s

   Note that these operations are all from scratch, meaning you don't need to
   keep any other files on your HDD except for your address list, and the 
   client could start up, read the entire blkchain file, and find all of your
   transactions and compute your balances from scratch, in under 30s.  I'm 
   very pleased with these results!  (NOTE:  This does not involve any kind
   of verification, or crypto operations).

   Some additional features:
      -- Will maintain a tx ledger for each address individually, as well
         as for the entire wallet as a whole (this should aggregate multiple
         TxIn objects in a single Tx into a single ledger entry, though I
         haven't tested this specific feature yet).  
      -- The ledgers can be sorted by order in the blockchain
      -- A separate list of your TxOut-TxIn pairs provides a simple way
         to access your unspent TxOuts, and prune off TxOuts that have 
         already been spent
      -- The organize-blockchain should detect reorganizations and invalid
         blocks, but does not yet do anything other than return a flag
         I will eventually go through and populate the previouslyValid list.
      -- The basics of the SWIG interface are in place:  I've made sure that
         std::strings and std::vectors are handled properly and that the 
         blockchain reading works from python.  But that was a few commits
         ago and the interface needs to be updated. 
   

#####
20 Aug, 2011:
   
   PyBtcEngine is a finally a fairly complete set of tools that addresses most
   of the calculations, manipulations and crypto that is involved in analyzing
   the blockchain.  It's not particularly fast, which is why I started creating
   a C++ backend to handle all the block data, to be imported via SWIG.  This
   is not yet complete, but believe will be the most efficient method possible
   to process the entire blockchain at once.


   Due to many requests on the forum, as well as personal desire to identify 
   large chunks of binary/hex data, I have created a separate script called
   mysteryHex.py which will try to completely break down a chunk of data and
   find BTC-related strings in it, using PyBtcEngine.  It should be a fairly
   versatile tool, allowing for quick ID of headers, transactions, public keys,
   etc.  Also allows the user to read their blockfile store to construct a 
   dictionary of 1-2 million hashes to include in the block file identification
   (use -s to enable hash searching from knownHashes.bin, and -u to create or
   update your knownHashes.bin file from blk0001.dat in your BTC directory).
   
   In fact, here's the output of ./mysteryHex.py --help:

      Try to identify Bitcoin-related strings in a block of data
      Usage: mysteryHex.py [--binary|-b] -f FILE 
         or: mysteryHex.py unidentifiedHex

      Options:
        -h, --help            show this help message and exit
        -f FILENAME, --file=FILENAME
                              Get unidentified data from this file
        -k BLK0001FILE, --blkfile=BLK0001FILE
                              Update hashlist from this file (default
                              ~/.bitcoin/blk0001.dat)
        -g HASHFILE, --hashfile=HASHFILE
                              The file to store and retrieve header/tx hashes
        -b, --binary          Specified file is in binary
        -s, --usehashes       Import header/tx hashes to be used in searching
        -u, --updatehashes    Search blk0001.dat to update hashlist
        -r, --rescanhashes    Rescan blkfile for header/tx hashes
 


#####
16 July, 2011:

   After a ton of work, I have finally completed much of the calculation engine!
   All number/binary conversions work, with easy endianness swapping.
   Complete, recursive serialization and unserialization of all block/tx data
   MerkleTree calculation and verification done
   Cryptography library from "Lis" wrapped and tested on arbitrary data
   Most script OP_CODEs implemented, and OP_CHECKSIG works on example tx data!

   Basically, everything else that I need to do (besides networking), should 
   be very easy to implement.  With all the endianness, serialization and 
   crypto methods tested, packet construction, block verification, data
   structures, and crypto anything should be a breeze.

   Of course, this does not include networking.  The networking engine is going
   to be a whole different beast... but luckily Sam's code already has a lot of 
   that written.  When it comes to networking, I might just stick to using other
   people's code, instead of re-writing it in my own way.

   Some immeidate things that I need to do:
      SIGHASH_ALL/NONE/SINGLE/ANYONCANPAY
      Wallet/Account file formats and implementation

   Long term:
      Networking engine (ugh)