K8 currently depends on a very old version of the V8 engine with known security issues.

Thanks,
CH

Is it possible to use 64 bit types in the byte array? The README does not give them as an example.

I got a segfault while compiling v8 according the the instructions in this repository's README.

I was able to fix this by targeting x64.debug instead of x64.release, as indicated in this v8 bug report: https://bugs.chromium.org/p/v8/issues/detail?id=2195

Since this fix is not yet documented in this repository, I thought I'd open and quickly close an issue to let others running into similar problems know: try cd v8-3.16.4 && make -j4 x64.debug, it might fix your segfault!

Hello @attractivechaos,

Could it be possible to provide a Linux ARM64 binary in the release too ?

I could provide a PR that adds a CI job on CircleCI that builds and uploads the binary.
I don't see any release related scripts in the repo, so I guess currently the Linux x86_64 and Mac ARM64 binaries are built manually and tar-ed into the release file. If this is the case then you would have to add one more step: download the Linux aarch64 binary from the last CircleCI workflow and put it next to the other binaries.

What do you think ?

Thank you!

Dear Heng,
I wanted to make a package for Gentoo Linux but I lack the above two files. Use

CXX ?= g++
CXXFLAGS ?= -O2
CPPFLAGS ?= -Iinclude

if possible in the Makefile.

Also, the docs say the only dependency is zlib but it turns out the main dependency is v8, whatever that is. ;-) There is a link to bugs opened for the v8 project but a link to their main homepage is missing (ideally a downlod page link).

Hope this helps,
Martin

v8 version 3.16.4 only supports ARCHES = ia32 x64 arm mipsel

On Aarch64, the compilation fails:

In file included from ../src/allocation.h:31,
                 from ../src/allocation.cc:28:
../src/globals.h:90:2: error: #error Host architecture was not detected as supported by v8
 #error Host architecture was not detected as supported by v8
  ^~~~~

Would it be possible to update to newer version of v8 to support ARM64 cpus?

Hello!

Cool work! Hacking with v8 is great :) And am glad you're making cool stuff. Though I couldn't help but cringe when I read:

Take the popular Node.js as an example. Node.js mixes file I/O and file system operations, two distinct concepts, in one File System module. In the module, we can either read an entire file or a fixed-length data blob, but are unable to read a line as is provided by most other programming languages. Many other JS shell implementations follow the CommonJS APIs, which have a similar problem: no usable APIs for general-purpose file I/O. After all these efforts, on file I/O, we even do not have a JS shell matching the usability of C, let alone high-level programming languages such as Perl and Python.

(bold mine)

This is an incorrect characterization of Node.js. This is simply not true. Please check out node Streams. This is how IO is done in node.

Let's take your example, line splitting. Here's how you do it:

var fs = require('fs')
var split = require('split')

var charStream = fs.createReadStream('foo')
var lineStream = charStream.pipe(split())
lineStream.on('data', function (line) {
    console.log('hey, look! a line: ' + line)
  })

// or: lineStream.pipe(process.stdout)

And it works beyond shells, you can use process to handle stdio:

var split = require('split')

process.stdin.
  pipe(split()).
  on('data', function (line) {
    console.log('hey, look! a line: ' + line)
  })

And you can use through to reverse them:

var split = require('split')
var through = require('through')

var reverse = through(function (line) {
  this.queue(line.split('').reverse().join('') + '\n')
})

process.stdin.
  pipe(split()).
  pipe(reverse).
  pipe(process.stdout)

Put that in a foo.js and run it:

cat foo.js | node foo.js

I recommend you learn more about node Streams (they're almost magical!), a great place to start is: https://github.com/substack/stream-handbook -- streams in node are much closer to UNIX pipes than C is. Yep, I'm well aware C was built for UNIX. Node's philosophy is really, really close to UNIX.

Oh I almost forgot, you'll definitely have to install split as it's not in core. How? you search npm for "stream line split" and find many modules, including split which has a ton of downloads. You install it with:

npm install split

I think maybe your real beef with node is that when you run node it doesn't come pre-loaded with lots of shiny things like split and so on. This is really a tradeoff between users, the idea is to keep a minimal core in Node and leave almost everything up to the modules.

This is the core philosophy of how node and npm do their magic. Having programmed in many environments for a while, I have to say, this is actually amazingly productive. It's The Right Thing To Do.

I agree that having to search and install a particular package is a bit hard to wrap your head around the first time you use node. It's a bit of friction when you first get started. Perhaps this will be fixed by better education: thankfully, core node folks have made nodeschool.io to help introduce the core conepts. Or by providing custom startup imports (i.e. a set of blessed modules you want with you on every shell execution). Or when we no longer have to install packages at all.

Cheers!
:)

Minor edits:

Download precomiplied binaries

wget -O- https://github.com/attractivechaos/k8/releases/download/1.0/k8-1.0.tar.bz2 | tar -jxf -

should be

Download precompiled binaries

wget -O- https://github.com/attractivechaos/k8/releases/download/v1.0/k8-1.0.tar.bz2 | tar -jxf -

Hi,

Could you please release 1.1 ?

We need the improvement added with 207544a#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R2 for Bioconda project

Thank you!

Hi,

While implementing the last line for compiling K8, I get the following error:

(py2) wg-dhcp174d160d214:v8-3.16.4 tarakshisode$ g++ -O2 -Wall -o k8 -Iinclude ../k8.cc -lpthread -lz `find out -name "libv8_base.a"` `find out -name "libv8_snapshot.a"`
clang: error: no such file or directory: '../k8.cc'

I looked up, there doesn't seem to be any k8.cc file?

May I know when I will release the new version? Can it be released recently? I saw that we will release the binary file for the arm version, which is so cool and I really want to use it

Here is a Dockerfile to reproduce how I attempted to build it:

FROM python:2.7

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y \
  wget \
  software-properties-common \
  curl \
  strace \
  sudo \
  time \
  make \
  g++ \
  rsync \
  && rm -rf /var/lib/apt/lists/*

RUN wget -O- https://github.com/attractivechaos/k8/archive/v0.2.3.tar.gz | tar zxf -
WORKDIR k8-0.2.3
RUN wget -O- https://github.com/attractivechaos/k8/releases/download/v0.2.1/v8-3.16.4.tar.bz2 | tar jxf -
WORKDIR v8-3.16.4
RUN make -j4 x64.release
RUN g++ -O2 -Wall -o k8 -Iinclude ../k8.cc -lpthread -lz `find out -name "libv8_*.a"|grep -v nosnap`

I also tried with v0.2.1 release of k8 - this failed in the same way.

docker build fails on the last step with:

/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/bootstrapper.o: In function v8::internal::Bootstrapper::NativesSourceLookup(int)': bootstrapper.cc:(.text._ZN2v88internal12Bootstrapper19NativesSourceLookupEi+0x9f): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetRawScriptSource(int)'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/bootstrapper.o: In function v8::internal::Genesis::CompileExperimentalBuiltin(v8::internal::Isolate*, int)': bootstrapper.cc:(.text._ZN2v88internal7Genesis26CompileExperimentalBuiltinEPNS0_7IsolateEi+0x12): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetScriptName(int)'
bootstrapper.cc:(.text._ZN2v88internal7Genesis26CompileExperimentalBuiltinEPNS0_7IsolateEi+0x1f): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetRawScriptSource(int)' /k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/bootstrapper.o: In function v8::internal::Genesis::CompileBuiltin(v8::internal::Isolate*, int)':
bootstrapper.cc:(.text._ZN2v88internal7Genesis14CompileBuiltinEPNS0_7IsolateEi+0x17): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetScriptName(int)' bootstrapper.cc:(.text._ZN2v88internal7Genesis14CompileBuiltinEPNS0_7IsolateEi+0xc5): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetRawScriptSource(int)'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/bootstrapper.o: In function v8::internal::Genesis::InstallExperimentalNatives()': bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x1c): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetDebuggerCount()'
bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x46): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetBuiltinsCount()' bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x5e): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetScriptName(int)'
bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x82): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetScriptName(int)' bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x93): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetRawScriptSource(int)'
bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0xe3): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetScriptName(int)' bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x10b): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetScriptName(int)'
bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x11c): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetRawScriptSource(int)' bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x161): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetScriptName(int)'
bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x189): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetScriptName(int)' bootstrapper.cc:(.text._ZN2v88internal7Genesis26InstallExperimentalNativesEv+0x19a): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)1>::GetRawScriptSource(int)'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/bootstrapper.o: In function v8::internal::Genesis::InstallNatives()': bootstrapper.cc:(.text._ZN2v88internal7Genesis14InstallNativesEv+0xf9b): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetDebuggerCount()'
bootstrapper.cc:(.text.ZN2v88internal7Genesis14InstallNativesEv+0xfb7): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetScriptName(int)' bootstrapper.cc:(.text._ZN2v88internal7Genesis14InstallNativesEv+0xffc): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetBuiltinsCount()'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/debug.o: In function v8::internal::Debug::CompileDebuggerScript(int)': debug.cc:(.text._ZN2v88internal5Debug21CompileDebuggerScriptEi+0x58): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetScriptName(int)'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/debug.o: In function v8::internal::Debug::Load() [clone .part.124]': debug.cc:(.text._ZN2v88internal5Debug4LoadEv.part.124+0x255): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetIndex(char const*)'
debug.cc:(.text.ZN2v88internal5Debug4LoadEv.part.124+0x356): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetIndex(char const*)' debug.cc:(.text._ZN2v88internal5Debug4LoadEv.part.124+0x37b): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetIndex(char const*)'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/debug.o: In function v8::internal::Debug::Load()': debug.cc:(.text._ZN2v88internal5Debug4LoadEv+0x263): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetIndex(char const*)'
debug.cc:(.text.ZN2v88internal5Debug4LoadEv+0x366): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetIndex(char const*)' /k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/debug.o:debug.cc:(.text._ZN2v88internal5Debug4LoadEv+0x38b): more undefined references to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetIndex(char const*)' follow
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/heap.o: In function v8::internal::Heap::CreateInitialObjects()': heap.cc:(.text._ZN2v88internal4Heap20CreateInitialObjectsEv+0x524): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetBuiltinsCount()'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/heap.o: In function v8::internal::Heap::ConfigureHeap(int, long, long)': heap.cc:(.text._ZN2v88internal4Heap13ConfigureHeapEill+0x8d): undefined reference to v8::internal::Snapshot::size'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/heap.o: In function v8::internal::Heap::ConfigureHeapDefault()': heap.cc:(.text._ZN2v88internal4Heap20ConfigureHeapDefaultEv+0xbc): undefined reference to v8::internal::Snapshot::size'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/serialize.o: In function v8::internal::Deserializer::Deserialize()': serialize.cc:(.text._ZN2v88internal12Deserializer11DeserializeEv+0xa4): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetBuiltinsCount()'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/serialize.o: In function v8::internal::Deserializer::ReadChunk(v8::internal::Object**, v8::internal::Object**, int, unsigned char*)': serialize.cc:(.text._ZN2v88internal12Deserializer9ReadChunkEPPNS0_6ObjectES4_iPh+0x853): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetRawScriptSource(int)'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/serialize.o: In function v8::internal::Serializer::ObjectSerializer::VisitExternalAsciiString(v8::String::ExternalAsciiStringResource**)': serialize.cc:(.text._ZN2v88internal10Serializer16ObjectSerializer24VisitExternalAsciiStringEPPNS_6String27ExternalAsciiStringResourceE+0x3a): undefined reference to v8::internal::NativesCollection<(v8::internal::NativeType)0>::GetBuiltinsCount()'
/k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/snapshot-common.o: In function v8::internal::Snapshot::ReserveSpaceForLinkedInSnapshot(v8::internal::Deserializer*)': snapshot-common.cc:(.text._ZN2v88internal8Snapshot31ReserveSpaceForLinkedInSnapshotEPNS0_12DeserializerE+0x2): undefined reference to v8::internal::Snapshot::new_space_used'
snapshot-common.cc:(.text.ZN2v88internal8Snapshot31ReserveSpaceForLinkedInSnapshotEPNS0_12DeserializerE+0xb): undefined reference to v8::internal::Snapshot::pointer_space_used_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot31ReserveSpaceForLinkedInSnapshotEPNS0_12DeserializerE+0x14): undefined reference to v8::internal::Snapshot::data_space_used'
snapshot-common.cc:(.text.ZN2v88internal8Snapshot31ReserveSpaceForLinkedInSnapshotEPNS0_12DeserializerE+0x1d): undefined reference to v8::internal::Snapshot::code_space_used_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot31ReserveSpaceForLinkedInSnapshotEPNS0_12DeserializerE+0x26): undefined reference to v8::internal::Snapshot::map_space_used'
snapshot-common.cc:(.text._ZN2v88internal8Snapshot31ReserveSpaceForLinkedInSnapshotEPNS0_12DeserializerE+0x2f): undefined reference to v8::internal::Snapshot::cell_space_used_' /k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/snapshot-common.o: In function v8::internal::Snapshot::HaveASnapshotToStartFrom()':
snapshot-common.cc:(.text._ZN2v88internal8Snapshot24HaveASnapshotToStartFromEv+0x2): undefined reference to v8::internal::Snapshot::size_' /k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/snapshot-common.o: In function v8::internal::Snapshot::Initialize(char const*)':
snapshot-common.cc:(.text.ZN2v88internal8Snapshot10InitializeEPKc+0x2d2): undefined reference to v8::internal::Snapshot::size_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot10InitializeEPKc+0x2df): undefined reference to v8::internal::Snapshot::raw_data'
snapshot-common.cc:(.text.ZN2v88internal8Snapshot10InitializeEPKc+0x2fc): undefined reference to v8::internal::Snapshot::raw_size_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot10InitializeEPKc+0x30b): undefined reference to v8::internal::Snapshot::new_space_used'
snapshot-common.cc:(.text.ZN2v88internal8Snapshot10InitializeEPKc+0x31d): undefined reference to v8::internal::Snapshot::pointer_space_used_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot10InitializeEPKc+0x32a): undefined reference to v8::internal::Snapshot::data_space_used'
snapshot-common.cc:(.text.ZN2v88internal8Snapshot10InitializeEPKc+0x337): undefined reference to v8::internal::Snapshot::code_space_used_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot10InitializeEPKc+0x344): undefined reference to v8::internal::Snapshot::map_space_used'
snapshot-common.cc:(.text._ZN2v88internal8Snapshot10InitializeEPKc+0x351): undefined reference to v8::internal::Snapshot::cell_space_used_' /k8-0.2.3/v8-3.16.4/out/x64.release/obj.target/v8_base/src/snapshot-common.o: In function v8::internal::Snapshot::NewContextFromSnapshot()':
snapshot-common.cc:(.text.ZN2v88internal8Snapshot22NewContextFromSnapshotEv+0xd): undefined reference to v8::internal::Snapshot::context_size_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot22NewContextFromSnapshotEv+0x2b): undefined reference to v8::internal::Snapshot::context_raw_data'
snapshot-common.cc:(.text.ZN2v88internal8Snapshot22NewContextFromSnapshotEv+0x48): undefined reference to v8::internal::Snapshot::context_raw_size_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot22NewContextFromSnapshotEv+0x57): undefined reference to v8::internal::Snapshot::context_new_space_used'
snapshot-common.cc:(.text.ZN2v88internal8Snapshot22NewContextFromSnapshotEv+0x6b): undefined reference to v8::internal::Snapshot::context_pointer_space_used_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot22NewContextFromSnapshotEv+0x75): undefined reference to v8::internal::Snapshot::context_data_space_used'
snapshot-common.cc:(.text.ZN2v88internal8Snapshot22NewContextFromSnapshotEv+0x7f): undefined reference to v8::internal::Snapshot::context_code_space_used_' snapshot-common.cc:(.text._ZN2v88internal8Snapshot22NewContextFromSnapshotEv+0x89): undefined reference to v8::internal::Snapshot::context_map_space_used'
snapshot-common.cc:(.text.ZN2v88internal8Snapshot22NewContextFromSnapshotEv+0x96): undefined reference to `v8::internal::Snapshot::context_cell_space_used'
collect2: error: ld returned 1 exit status