Flying with emscripten about td

[arseny30]

@jwktje Ouch. Seems like we already added cwrap into CMakeLists.txt, but only in our private repository. Will be fixed with the next release of tdlib.

For now you may try to manually apply this diff to CMakeList.txt and then rebuild tdlib from scratch.

-  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Os -s ALLOW_MEMORY_GROWTH=1 -s USE_ZLIB=1 -s MODULARIZE=1 -s WEBSOCKET_URL=\"'wss:#'\" -s EXTRA_EXPORTED_RUNTIME_METHODS=\"['FS']\"")
-  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Os -s ALLOW_MEMORY_GROWTH=1 -s USE_ZLIB=1 -s MODULARIZE=1 -s WEBSOCKET_URL=\"'wss:#'\" -s EXTRA_EXPORTED_RUNTIME_METHODS=\"['FS']\"")
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Os -s ALLOW_MEMORY_GROWTH=1 -s MEMFS_APPEND_TO_TYPED_ARRAYS=1 -s USE_ZLIB=1 -s MODULARIZE=1 -s WEBSOCKET_URL=\"'wss:#'\" -s EXTRA_EXPORTED_RUNTIME_METHODS=\"['FS','cwrap']\"")
+  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Os -s ALLOW_MEMORY_GROWTH=1 -s MEMFS_APPEND_TO_TYPED_ARRAYS=1 -s USE_ZLIB=1 -s MODULARIZE=1 -s WEBSOCKET_URL=\"'wss:#'\" -s EXTRA_EXPORTED_RUNTIME_METHODS=\"['FS','cwrap']\"")

[isopen]

@levlam Thanks
TDLib works in the browser.

[Bannerets]

linux-generic64 to linux-generic32 changes nothing, but change of emmake ./Configure to emconfigure ./Configure helped, it works. Big thanks, @levlam.

TDLib works in browser correctly, I was able to sign in.

Final dockerfile
https://github.com/Bannerets/tdlib-binaries/blob/bb632d800f2c19930ceadef74b1333417eba9eac/misc/docker/emscripten/Dockerfile

[levlam]

We will publish build instructions for Emscripten, when they will be production ready. It shouldn't be hard to compile TDLib for Emscripten, but to actually use TDLib in a browser, additional Websocket to TCP proxies are needed. They IPs need to be hard coded to TDLib source code, which is acceptable only for testing purposes.

[jwktje]

@isopen It looks like you're doing some solid research. Just starting out myself and would love to use that build script. Do you have a quick tip on how to use that dockerfile?

Update: I think i'm figuring it out. Never mind for now :)
Update 2: It's failing for me on:
[48%] Building CXX object CMakeFiles/tdcore.dir/td/telegram/AuthManager.cpp.o
The command '/bin/bash -c make -j 4 && rm -rf *' returned a non-zero code: 137
Any specifics I'd need to know?
Update 3: If anyone has the same 137 error as me, I solved it by increasing Docker's resources under Preferences -> Advanced.

[jwktje]

@arseny30 Thanks for helping. Sadly no luck yet. cwrap is also undefined for me and not exported.
The docs also mention this i think. Check this page

It also mentions needing to use the flag to include the function. But this doesn't seem to work for me in the dockerfile by @isopen

Did you build your wasm file differently @arseny30 ?

[Bannerets]

I get [Error : 0 : RSA_size != 256] error after checkDatabaseEncryptionKey. Any suggestions?

I used this tdlibParameters:

    api_id: ...,
    api_hash: '...',
    database_directory: 'db',
    use_message_database: false,
    use_secret_chats: false,
    system_language_code: 'en',
    application_version: '1.0',
    device_model: 'Browser',
    system_version: 'Unknown'

Tested in both Chrome and Firefox.
TDLib compiled with patch by Arseny and dockerfile by @isopen.

[Bannerets]

Unfortunately, with your prebuilt openssl I got this error:

[ 98%] Linking CXX executable td_wasm.js
ERROR: linking module flags 'PIC Level': IDs have conflicting behaviors

I built openssl 1.1.0f myself but faced with the same RSA_size != 256 error.

Added some debugging prints:

#include <openssl/err.h>
//...
Result<RSA> RSA::from_pem(Slice pem) {
  // ...
  // ...
  if (!PEM_read_bio_RSAPublicKey(bio, &rsa, nullptr, nullptr)) {
    return Status::Error("Error while reading rsa pubkey");
  }

  LOG(INFO) << "RSA_size: " << RSA_size(rsa); //---> 252
  LOG(INFO) << "RSA_check_key: " << RSA_check_key(rsa); //---> 0

  unsigned long errcode = ERR_get_error();
  LOG(INFO) << "ERR_get_error: " << errcode; //---> 67764371
  char errstring[256];
  ERR_error_string(errcode, errstring);
  LOG(INFO) << errstring; //---> error:040A0093:rsa routines:RSA_check_key_ex:value missing

  RSA_print_fp(stdout, rsa, 0);
  /*--->
    Public-Key: (2014 bit)
    Modulus:
        2f:70:db:79:75:9c:fe:cf:75:9c:fe:cf:9a:41:da:
        f4:9a:41:da:f4:81:35:a6:f9:81:35:a6:f9:0e:c9:
        ba:97:0e:c9:ba:97:fc:f6:c5:68:fc:f6:c5:68:d6:
        34:86:49:d6:34:86:49:d6:d4:9f:4e:d6:d4:9f:4e:
        ae:97:d9:5c:ae:97:d9:5c:3c:0a:10:d8:3c:0a:10:
        d8:ed:d6:92:6a:ed:d6:92:6a:db:7d:45:7f:db:7d:
        45:7f:65:fc:d6:ff:65:fc:d6:ff:11:df:91:c0:11:
        df:91:c0:97:62:5f:6c:97:62:5f:6c:25:55:69:34:
        25:55:69:34:6b:34:f0:11:6b:34:f0:11:a0:90:19:
        6e:a0:90:19:6e:49:af:7e:b6:49:af:7e:b6:64:94:
        ca:5f:64:94:ca:5f:5b:6d:d2:76:5b:6d:d2:76:b6:
        0e:5d:f6:b6:0e:5d:f6:10:60:7e:f2:10:60:7e:f2:
        23:6c:25:5f:23:6c:25:5f:40:83:a9:67:40:83:a9:
        67:f6:4c:1d:f4:f6:4c:1d:f4:4f:b2:6a:2a:4f:b2:
        6a:2a:ee:e6:4a:d1:ee:e6:4a:d1:85:cd:e7:4a:85:
        cd:e7:4a:6a:bf:59:ba:6a:bf:59:ba:73:e7:13:0f:
        73:e7:13:0f:21:79:25:1f:21:79:25:1f
    Exponent: 65537 (0x10001)
  */

  if (RSA_size(rsa) != 256) {
    return Status::Error("RSA_size != 256");
  }

Correct modulus is:

C150023E2F70DB7985DED064759CFECF0AF328E69A41DAF4D6F01B538135A6F91F8F8B2A0EC9BA9720CE352EFCF6C5680FFC424BD634864902DE0B4BD6D49F4E580230E3AE97D95C8B19442B3C0A10D8F5633FECEDD6926A7F6DAB0DDB7D457F9EA81B8465FCD6FFFEED114011DF91C059CAEDAF97625F6C96ECC74725556934EF781D866B34F011FCE4D835A090196E9A5F0E4449AF7EB697DDB9076494CA5F81104A305B6DD27665722C46B60E5DF680FB16B210607EF217652E60236C255F6A28315F4083A96791D7214BF64C1DF4FD0DB1944FB26A2A57031B32EEE64AD15A8BA68885CDE74A5BFC920F6ABF59BA5C75506373E7130F9042DA922179251F

So, first 4 bytes (C1 50 02 3E) are lost.
But RSA::from_pem(Slice pem) called with correct pem.
It seems, PEM_read_bio_RSAPublicKey(bio, &rsa, nullptr, nullptr) works incorrectly.

I checked this code:

char n_str[] = "AEEC36C8FFC109CB099624685B97815415657BD76D8C9C3E398103D7AD16C9BBA6F525ED0412D7AE2C2DE2B44E77D72CBF4B7438709A4E646A05C43427C7F184DEBF72947519680E651500890C6832796DD11F772C25FF8F576755AFE055B0A3752C696EB7D8DA0D8BE1FAF38C9BDD97CE0A77D3916230C4032167100EDD0F9E7A3A9B602D04367B689536AF0D64B613CCBA7962939D3B57682BEB6DAE5B608130B2E52ACA78BA023CF6CE806B1DC49C72CF928A7199D22E3D7AC84E47BC9427D0236945D10DBD15177BAB413FBF0EDFDA09F014C7A7DA088DDE9759702CA760AF2B8E4E97CC055C617BD74C3D97008635B98DC4D621B4891DA9FB0473047927";

BIGNUM *n_new = BN_new();
BN_hex2bn(&n_new, n_str);

char* number_str = BN_bn2hex(n_new);
LOG(INFO) << "bn2hex: " << number_str;

It outputs FFC109CBFFC109CB5B9781545B9781546D8C9C3E6D8C9C3EAD16C9BBAD16C9BB0412D7AE0412D7AE4E77D72C4E77D72C709A4E64709A4E6427C7F18427C7F1847519680E7519680E0C6832790C6832792C25FF8F2C25FF8FE055B0A3E055B0A3B7D8DA0DB7D8DA0D8C9BDD978C9BDD97916230C4916230C40EDD0F9E0EDD0F9E2D04367B2D04367B0D64B6130D64B613939D3B57939D3B57AE5B6081AE5B6081CA78BA02CA78BA026B1DC49C6B1DC49C7199D22E7199D22E47BC942747BC9427D10DBD15D10DBD153FBF0EDF3FBF0EDFC7A7DA08C7A7DA08702CA760702CA76097CC055C97CC055C3D9700863D970086D621B489D621B4897304792773047927 (first 4 bytes are lost), so, problem is definitely somewhere in openssl BigNum.

[isopen]

@Bannerets Building openssl emscription is still rather unstable.

[levlam]

TDLib 1.4.0 includes an example of TDLib usage in a browser. You can also use NPM package tdweb if you don't want to build TDLib yourself.

[UnblockerUsername]

I can not collect it either too.
Need explanations for professionals.:0
@levlam help please

[UnblockerUsername]

@isopen Collect openssl via emscripten. readme

@levlam Thank you. You can close ticket. I was able to compile td_wasm.js, td_wasm.wasm

[levlam]

I prefer the issue to stay open until we publish our instructions on building TDLib with Emscripten and its usage from browser.

[Bannerets]

@isopen How did you build tdlib? I can't link openssl properly.

I used this dockerfile https://github.com/Bannerets/tdlib-binaries/blob/07ee8ac58bd65952935458b51553be300953dea5/misc/docker/emscripten/Dockerfile
Any suggestions?

I got these errors:

WARNING:root:emcc: cannot find library "ssl"
WARNING:root:emcc: cannot find library "crypto"
warning: unresolved symbol: AES_cbc_encrypt
warning: unresolved symbol: AES_encrypt
warning: unresolved symbol: AES_ige_encrypt
...

Full log

[isopen]

@Bannerets Build script Dockerfile.wasm

[jwktje]

@isopen Would love your help on this. I need to specify to Emscripten to take the following option when building:
-s EXTRA_EXPORTED_RUNTIME_METHODS='["cwrap", "allocate", "intArrayFromString"]'

Where in the dockerfile would I specify this? I tried adding it to the last line like so:
&& emmake make -s EXTRA_EXPORTED_RUNTIME_METHODS='["cwrap", "allocate", "intArrayFromString"]' -j 4

[isopen]

Try it export EMMAKEN_CFLAGS = "-s ..."

[jwktje]

No luck sadly. When console.logging the tdlib JS glue it is not including the extra functions. Any other way you could think of making this work? I'd wanna eventually use this code:

function execute(query) {
        // Create a pointer using the 'Glue' method and the String value
        var ptr  = allocate(intArrayFromString(JSON.stringify(query)), 'i8', ALLOC_NORMAL);

        // Call the method passing the pointer
        var retPtr = tdlib._td_execute(client, ptr);

        // Retransform back your pointer to string using 'Glue' method
        var resValue = Pointer_stringify(retPtr);

        // Free the memory allocated by 'allocate' 
        _free(ptr);  
        
        return JSON.parse(resValue);
    }

Update: @isopen tried adding the flag in all sorts of variations and locations in the file but can't get emscripten to include the runtime methods. What do you think?

[arseny30]

@jwktje You don't need to explicitly call intArrayFromString. It is much simpler to use cwrap, which is already exported by default.

In our code it looks like this:

    this.td_functions = { 
      td_create: this.TdModule.cwrap('td_create', 'number', []), 
      td_destroy: this.TdModule.cwrap('td_destroy', null, ['number']), 
      td_send: this.TdModule.cwrap('td_send', null, ['number', 'string']), 
      td_execute: this.TdModule.cwrap('td_execute', null, ['number', 'string']),
      td_receive: this.TdModule.cwrap('td_receive', 'string', ['number']), 
      td_set_verbosity: this.TdModule.cwrap('td_set_verbosity', null, [ 
        'number' 
      ]), 
      td_get_timeout: this.TdModule.cwrap('td_get_timeout', 'number', []) 
    };
// ...
   this.td_functions.td_send(this.client, JSON.stringify(query));   

Also in your example it probably should be tdlib_.intArrayFromString not just intArrayFromString.

[levlam]

Something is completely broken. The error means that built-in public RSA keys of Telegram servers are invalid, which isn't the case. Rebuilding OpenSSL from scratch may help.

[isopen]

When using openssl1.0.2p all oki doki
[ 0][t 0][1541869303.285000086][PublicRsaKeyShared.cpp:23][!Td][&r_rsa.is_ok()] [Error : 0 : RSA_size != 256] -----BEGIN RSA PUBLIC KEY-----
RSA_size == 360
@levlam RSA_new?

[Bannerets]

Found that RSA_size(rsa) equals to 252.

If I comment these lines,

  if (RSA_size(rsa) != 256) {
    return Status::Error("RSA_size != 256");
  }

I receive authorizationStateWaitPhoneNumber update and see websocket connections in console, but get [Session.cpp:1106][!HandshakeActor] Failed to open connection: [Error : 0 : Unknown fingerprints : [172.29.2.0:443] to DcId{2} from [0.0.0.0:0]] warnings, as I expected.

@isopen I can't build tdlib with openssl-1.0.2p, can only with openssl-1.0.2a (seems that it requires many changes). Do you have dockerfile with openssl 1.0.2p? Or even prebuilt binaries?

[jwktje]

I got to the exact error that @Bannerets got stuck on. But I just concluded that I was dumb and doing stuff wrong and gave up haha. But this makes me motivated to keep trying.

[isopen]

@Bannerets openssl-1.0.2p #18 (comment)
For some versions, you can ignore fatal errors while building. Because only .a static files are important.

[levlam]

@Bannerets Yes, it will be unable to connect to Telegram DCs without correct RSA keys.

Between, we use OpenSSL 1.1.0 in our build for Emscripten. As a fast test, you can try this prebuilt OpenSSL 1.1.0f:
crypto.zip

[levlam]

@Bannerets, could you also output original string:

char n_str[] = "AEEC36C8FFC109CB099624685B97815415657BD76D8C9C3E398103D7AD16C9BBA6F525ED0412D7AE2C2DE2B44E77D72CBF4B7438709A4E646A05C43427C7F184DEBF72947519680E651500890C6832796DD11F772C25FF8F576755AFE055B0A3752C696EB7D8DA0D8BE1FAF38C9BDD97CE0A77D3916230C4032167100EDD0F9E7A3A9B602D04367B689536AF0D64B613CCBA7962939D3B57682BEB6DAE5B608130B2E52ACA78BA023CF6CE806B1DC49C72CF928A7199D22E3D7AC84E47BC9427D0236945D10DBD15177BAB413FBF0EDFDA09F014C7A7DA088DDE9759702CA760AF2B8E4E97CC055C617BD74C3D97008635B98DC4D621B4891DA9FB0473047927";

LOG(INFO) << "Original: " << sizeof(n_str) << ' ' strlen(n_str) << ' ' << n_str;

BIGNUM *n_new = BN_new();
BN_hex2bn(&n_new, n_str);

char* number_str = BN_bn2hex(n_new);
LOG(INFO) << "bn2hex: " << strlen(number_str) << ' ' << number_str;

[Bannerets]

@levlam

Original: 513 512 AEEC36C8FFC109CB099624685B97815415657BD76D8C9C3E398103D7AD16C9BBA6F525ED0412D7AE2C2DE2B44E77D72CBF4B7438709A4E646A05C43427C7F184DEBF72947519680E651500890C6832796DD11F772C25FF8F576755AFE055B0A3752C696EB7D8DA0D8BE1FAF38C9BDD97CE0A77D3916230C4032167100EDD0F9E7A3A9B602D04367B689536AF0D64B613CCBA7962939D3B57682BEB6DAE5B608130B2E52ACA78BA023CF6CE806B1DC49C72CF928A7199D22E3D7AC84E47BC9427D0236945D10DBD15177BAB413FBF0EDFDA09F014C7A7DA088DDE9759702CA760AF2B8E4E97CC055C617BD74C3D97008635B98DC4D621B4891DA9FB0473047927

bn2hex: 512 FFC109CBFFC109CB5B9781545B9781546D8C9C3E6D8C9C3EAD16C9BBAD16C9BB0412D7AE0412D7AE4E77D72C4E77D72C709A4E64709A4E6427C7F18427C7F1847519680E7519680E0C6832790C6832792C25FF8F2C25FF8FE055B0A3E055B0A3B7D8DA0DB7D8DA0D8C9BDD978C9BDD97916230C4916230C40EDD0F9E0EDD0F9E2D04367B2D04367B0D64B6130D64B613939D3B57939D3B57AE5B6081AE5B6081CA78BA02CA78BA026B1DC49C6B1DC49C7199D22E7199D22E47BC942747BC9427D10DBD15D10DBD153FBF0EDF3FBF0EDFC7A7DA08C7A7DA08702CA760702CA76097CC055C97CC055C3D9700863D970086D621B489D621B4897304792773047927

Original is ok. But, hm, length is the same. I found that there are much more differences. Every four bytes are deleted. diff.

[levlam]

@Bannerets Yes, bytes 8k...8k+3 are always identical to bytes 8k+4...8k+7, which is clearly wrong. It is a clear sign that something is wrong with OpenSSL build.

[levlam]

@Bannerets
It seems that we have used emconfigure ./Configure linux-generic32 or emconfigure ./Configure linux-generic32 no-shared for OpenSSL configuration. Could you try them?

[levlam]

@Bannerets Maybe just change of linux-generic64 to linux-generic32 will resolve the problem.