web-devkits / webinizer

Webinizer is a collection of tools to convert native applications, such as these programmed by C/C++/Rust, to Web applications which are constructed by HTML/CSS/JavaScript and WebAssembly.

Webinizer

Important

Webinizer is now in Beta trial. We'd greatly appreciate your feedback!

About

Webinizer is a collection of tools to convert native applications, such as these programmed by C/C++/Rust, to Web applications which are constructed by HTML/CSS/JavaScript and WebAssembly.

Webinizer consists of two parts; a core engine that analyzes the code and then either fixes or highlights issues, and a web frontend to configure projects and display results. This repo consists of the core engine, for the web frontend see the webinizer-webclient repo.

We also provide the webinizer-demo repo that holds the demo projects and build scripts to setup Webinizer.

Setting up Webinizer

Run with Docker (recommended)

Webinizer provides scripts to build a Docker image that will setup everything for you and make Webinizer ready to use out of the box.

Please follow the Docker installation and setup guide for the detailed instructions.

Run locally

The Webinizer also supports running locally without Docker. Currently this is validated on Linux (Ubuntu 20.04).

Please follow the Run locally guide for the detailed instructions.

Documentation

Webinizer User Manual is available under documentation/ folder with below structure.

Using Webinizer

• Installation and setup
• Build a simple project
• Build and define a library
• Build a project with dependencies

Extending Webinizer

Webinizer provides an extension mechanism for developers to extend the capabilities of the tool.

• Extension Overview
• How to develop an Extension
• How to use an Extension
• How to install an Extension
• How to test an Extension

Areas under development

We're considering how to create a package format and repository hosting for libraries and applications ported to Wasm. There are proposals for module format and registry hosting in our wiki.

Setting up the development environment

The steps to setup the development environment are similar to those described in Run Webinizer locally.

Basic commands

• npm run serve to launch the core engine server and start at port 16666.
• npm run test to run the unit tests, see tests/README.md for details.
• npm run lint-fix to format code with ESLint.
• npm run doc to generate the Webinizer Extension API document into ./docs.
• npm run swagger-autogen to generate the swagger specification for server RESTful APIs , see generate swagger specification for details.

Recommended IDE setup

It's preferred to use VS Code for development.

It's preferred to use Prettier formatter, along with Format on Save. Please setup the Prettier and ESLint extensions and configure them accordingly.

NOTE that we set printWidth as 100 and tab width as 2.

Contributing

We welcome contributions to Webinizer. You can find more details in CONTRIBUTING.md .

webinizer's Issues

[Github Action] Encounter error during install system dependencies

dpkg: error processing package grub-efi-amd64-signed (--configure):
 installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32

Reference:
https://github.com/orgs/community/discussions/47863
actions/runner-images#7192

Initial memory error wasn't reported when using wasm-ld 17.0.0 to build FFmpeg

The initial memory error wasn't reported by wasm-ld during the build time, but reported at runtime.

• Emscripten: 3.1.31
• wasm-ld: LLD 17.0.0

This error used to be reported with Emscripten 3.1.23 and LLD 16.0.0 during the build time.

Enhancements for `Dependent packages check` advisor

Currently, dep_check advisor will just remove the find_package(XXX) statement but ignore other operations related to this emscripten ported library , which may cause the errors block the building process. We should make enhancements to avoid this problem.

• For emscripten ported libraries (as shown in the above pictures), webinizer will add -sUSE_LIBXXX into the compiler & linker flags, and emscripten will take responsibilities of linking libraries and including header files directories, so except for find_package statement, the include_directories(${XX_INCLUDE_DIRS}), target_link_libraries(target ${XX_LIBRARIES}) and other operations related to this library are not necessary any more.
• For the dependencies added from webinizer UI, should also remove the find_package(DEP) statement from the main project, besides, should adjust the statement pattern to target_link_libraries(target -ldepName)

The migration of Snake Game project

The Snake Game is classic game using SDL2, we are trying to successfully build and add it into the demos pool. Moreover, we can enhance the current advisors' implementation during porting. This issue will track the problems encountered and the corresponding solutions during the migration

The source code link of Snake Game: https://github.com/udacity/CppND-Capstone-Snake-Game

Support more project metadata

We should support more project metadata and enhance the Publish page accordingly.

• author
• keywords
• repository
• homepage
• bugs
• license

Detect if a line is within a multi-line comment block

The detection is used multiple times in our codebase, and we could improve the detection for lines within a multi-line comment block. See examples below:

https://github.com/intel/webinizer/blob/f6c375223d2e80fff7e6bd8191fb0d2871ef31ef/src/advisors/cpp_exception.ts#L52-L57

Encapsulate options and configs from ProjectConfig class

Refactor to make the options and configs operations decoupled from the ProjectConfig class.

Handle buildSteps metadata for builders registered from extensions

Handle project paths / file names with spaces

Overall environment variables should be updated if a dependent project's related configs are updated

Currently, target and pkgConfig update should be propagated to all requiredBy projects.

Handle duplicated preload files

Need to check duplicated preload files and remove the possible duplicates when updating the configs.

Config field classes should be created regardless of the data value is null/empty or not

Move getFileLocation function to CMakeBuilder class

Currently, the getFileLocation function is used by CMakeBuilder only and the logic was implemented for cmake errors logs as well. We should probably move it to the CMakeBuilder class, and implement separate methods for other builders if needed in the future.

Cleanup SUPPORT.md

Remove direct email address and other tweaks.

The content of error not handled by webinizer is not well displayed

The reason is that the raw data of desc in actions is

"desc": "This error isnot handled by Webinizer, please try to resolve it manually:\n\n```emcc: error: no input files\n```",

and the emcc in content will be recognized and converted to code-block: language and the content will be ignored and not displayed.

Resolution:

https://github.com/intel/webinizer/blob/cf8459f2c83fa3b642155a671c3151dfa6d11fbd/src/advisors/errors_not_handled.ts#L39C52-L39C59
->

Develop `ModularizeJS` advisor

• Modularize JS output configuration option is enabled by default, generally the building process goes well but errors will occur if we want to output html. Developing ModularizeJS advisor is to automatically detect the output target suffix and disabled modularize js out option if the output suffix is set as html.

feature proposal: generate script for building a C/C++ code base to WebAssembly

after everything is tuned and a project can be successfully built by Webinizer, it would be great to export a build script to do compilation offline.

Enhancements summary

As more and more projects are tested, more and more improvements will be found based on the current implementation. We open this issue to save and track these enhancements.

• #54
• #56
• #58

new feature: support wasi-sdk as the compiler toolchain

Add webinizer as a default keyword

Enhance builder creation from the factory

Pass an options object instead of args string to make the creation more flexible.

https://github.com/intel/webinizer/blob/f6c375223d2e80fff7e6bd8191fb0d2871ef31ef/typings/webinizer.d.ts#L659-L665

Add explanation for builder/advisor/action in documentation

Enhancements for `SIMD` advisor

According to the emscripten documentation, there are five different ways to leverage WebAssembly SIMD in your C/C++ programs:

• Enable LLVM/Clang SIMD autovectorizer to automatically target WebAssembly SIMD, without requiring changes to C/C++ source code.
• Write SIMD code using the GCC/Clang SIMD Vector Extensions (__attribute__((vector_size(16))))
• Write SIMD code using the WebAssembly SIMD intrinsics (#include <wasm_simd128.h>)
• Compile existing SIMD code that uses the x86 SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2 or 128-bit subset of the AVX intrinsics (#include <*mmintrin.h>)
• Compile existing SIMD code that uses the ARM NEON intrinsics (#include <arm_neon.h>)

x86intrin.h header file is usually used in C\C++ codebase, which contains the support of multiple instruction sets, commonly covering the one or several of MMX, SEE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX, AVX2, etc, but it is related to the what compiler you use. For the clang inside of the emsdk, it contains ia32intrin.h & mmintrin.h, which will not handled by emscripten during the building, so the better ways to port your project with SIMD support are:

• if there is no SIMD code used in the codebase and you want to enable SIMD option to let compiler autovector to target wasm-simd, add -msimd128 is enough.
• if your current codebase exists SIMD code, include <wasm_simd128.h> and then use the wasm SIMD intrinsics to replace your current code.
• if your current codebase include SIMD code targeting x86 SSE* instruction sets, add -msimd128 and additionally corresponding flag.
  • For example, SSE: pass -msse and #include <xmmintrin.h>. Use #ifdef __SSE__ to gate code.