kscripting / kscript Goto Github PK

kscript should stop with an error and print some installation help instructions if is either is missing.

Thanks for putting this up... I've been playing with it to do quick kotlin prototyping.

Spent the morning getting it working, ended up being two small bugs:

1. using the github URL for resolving dependencies didn't seem to work on OSX; kept getting error: could not find or load main class Main_Scriptarg_cache_3335d424be6c36f7 when using anything with //DEPS. Putting resdeps.kts in my ~/bin solved the problem.

2. bug: in the example for Docopt, there's a line of doArgs.joinToString(). Should just be doArgs.

Fun stuff regardless, thanks!

2 Problems would need to be solved

1. How to overcome JVM startup delay? Either by using kotlin-native or by using a virtualizaiton model such as avian (see https://github.com/ReadyTalk/avian and https://blog.plan99.net/kotlin-native-310ffac94af2)
2. How to mimic process substitution? Currently by using exec we replace the kscript process with the actual jvm process at the end is really nice. Not sure how to do this from within the JVM.

Alternatively we could also reimplement in Go-lang

By rewriting it in Kotlin we pay with 120ms per launch but we gain a lot of stability, cleaner code and portablity.

potentially we should use ~/.kscript for all output

Simply grep the cache file for the dep-hash and use its value if present

See https://travis-ci.org/holgerbrandl/kscript

It seems related to the stream-redirection settings used in runProcess()

After following install guide as defined in root of README.md, was unable to run self update. Repro'ed on both Linux and OS X. This is not a blocker as I am able to run kscript and execute kscripts.

Error example:

$ kscript --self-update
Installing latest version of kscript...
~/.sdkman/candidates/kscript/current/bin/kscript: line 13: exec: sdkman_auto_answer=true: not found

I hope we can rewrite mnvcp with Kotlin. Because a lot machines (like default Ubuntu distribution) doesn't have ruby.

We can simply use and internal kscriptlet and BufferedReader to fetch scripts from the web.

Support for //MVN_REPO declaration.

This will keep kscripts maintainable and self-included. Also, it will allow to provide complex application demos with something like kscript http://short-url

Mockup example using scenery:
https://gist.github.com/holgerbrandl/e2e1b3d4c75f2d8248ea825a07fe26e6

This will make expandcp.kts more flexible (ie by supporting custom maven repositories)

In a project, I have multiple kotlin script files, and I have some methods I need to use in most, if not all script.

Is there an existing way to write these method in a single external file, and include it from the other scripts ?

eg :
utils.kts

fun foo(s : String) : String {
  //...
}

myscript.kts

#!/usr/bin/env kscript

//INCLUDE utils.kts

foo(args[0])

Example

kscript -i - <<"EOF"
//DEPS de.mpicbg.scicomp:kutils:0.8.1
//DEPS com.github.holgerbrandl:kscript:1.2.2
EOF

• unit test needed!

E.g. https://www.eclipse.org/proposals/technology.aether/

See

• https://stackoverflow.com/questions/4023830/how-compare-two-strings-in-dot-separated-version-format-in-bash
• https://stackoverflow.com/questions/4023830/how-compare-two-strings-in-dot-separated-version-format-in-bash

The initial failure looked like:

$ kscript 
which: no javac in (./C:\jdk8u131-64/bin) 
[ERROR] Could not locate 'C:\jdk8u131-64/bin/javac' 

There is, of course, a javac in that directory.

After reading the source, I tried setting JAVAC_CMD, but that produced the same error.
So I unset JAVA_HOME & re-ran and then got

$ kscript 
kscript - Enhanced scripting support for Kotlin on *nix-based systems. 

Usage     : kscript <scriptfile, script-url, - for stdin, some kotlin code> [<script_args>]* 
Usage     : kscript --clear-cache     # wipe cached script jars and urls 
Usage     : kscript -i                # interactive shell with DEPS as declared in script 

Copyright : 2017 Holger Brandl 
License   : MIT 
Version   : v1.6.0 
Website   : https://github.com/holgerbrandl/kscript 
md5sum: /dev/fd/63: No such file or directory 
error: source file or directory not found: /cygdrive/c/Users/ngalarneau/.kscript/scriptarg_cache_.kts 
[ERROR] compilation of '/cygdrive/c/Users/ngalarneau/.kscript/scriptarg_cache_.kts' failed 

Since users will need to do so to get proper IDE support, it would be great if kscript could provide methods to ease bi-directionaly dependency synchronization

E.g.

• print pom from script
• print gradle from script
• print //DEPS ... from pom.xml or build.gradle

$ ./kscript2 "1+1"
Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter
        at kscript.app.KscriptKt.md5(Kscript.kt:336)
        at kscript.app.KscriptKt.md5(Kscript.kt:341)
        at kscript.app.KscriptKt.prepareScript(Kscript.kt:306)
        at kscript.app.KscriptKt.main(Kscript.kt:94)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at org.jetbrains.kotlin.runner.AbstractRunner.run(runners.kt:61)
        at org.jetbrains.kotlin.runner.Main.run(Main.kt:109)
        at org.jetbrains.kotlin.runner.Main.main(Main.kt:119)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.DatatypeConverter
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:466)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:563)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
        ... 11 more

For solutions see https://stackoverflow.com/questions/43574426/how-to-resolve-java-lang-noclassdeffounderror-javax-xml-bind-jaxbexception-in-j

Using proper file annotations would be better design and also allow for better tooling

// DEPS log4j:log4j:1.2.14

// should rather be declared as

@file:DependsOn("log4j:log4j:1.2.14")

No such thing as DependsOn in kotlin stdlib. Custom project artifact dependency declaring those extra annotations would be needed.

Or support both dependening on user skill level

Once KEEP #75 Scripting Enhancements is implementsed, kscript should adopt that.

Or simply adopt from https://github.com/ligee/kotlin-jupyter#dependencies-resolving-annotations

I was playing around with spark to see if kscript + spark would be a nice replacement for a python + flask script I have, so I created a script file called spark-server.kts with the following contents:

#!/usr/bin/env kscript

//DEPS com.sparkjava:spark-kotlin:1.0.0-alpha

import spark.kotlin.*

val http: Http = ignite()

http.get("/hello") {
        "Hello Spark Kotlin!"
}

When I tried to run it with ./spark-server.kts I got the following error:

error: could not find or load main class Main_Spark-server

Then I renamed it to sparky.kts and everything worked ok.

BTW, kscript is really cool!

See
https://github.com/holgerbrandl/kscript-support-api/blob/1ecbdeecaa68d57a8b2365a42995d7ce3bee28a0/src/main/kotlin/kscript/text/Tables.kt#L111

Should be xor!?

It should support scripts as direct arguments

kscript "1+1"

This would be particulary useful for pipes:

grep foo.txt | kscript 'stdin.filter{ it.starts.with("hey") }'

e.g.

cat some_big_file | kscript 'stdin.filter { "^de0[-0]*".toRegex().matches(it) }.forEach{it::println}'

ie. by prefixing dependencies with a minus

//DEPS org.apache.commons:commons-csv:1.3 -log4j:log4j:1.2.14

First of all — it works!
It's possible to ship kscript as one file with both shell script and jar files inside. To do so, just cat kscript.jar into slightly modified kscript shell script. This is how united file may looks like:

#!/usr/bin/env bash

# version tag for backward compatibility with kscript v1.x
KSCRIPT_VERSION=2.0.0

exec $(kotlin -classpath $(dirname $0)/kscript kscript.app.KscriptKt "$@")
exit

<BINARY CONTENT OF JAR FILE>

It's known feature of tar/unzip to skip any content until their own header.
I've tested this hack on java versions "1.8.0_144" / "9" and it works. With java 9 I still faced issue #41 but I think it's a different story.

When I try the sample script, I get an error. But my script runs ok when I use:

kotlinc -script ./test.kts

Error:
Exception in thread "main" java.lang.UnsupportedClassVersionError: Main_Test : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.jetbrains.kotlin.runner.AbstractRunner.run(runners.kt:38)
at org.jetbrains.kotlin.runner.Main.run(Main.kt:99)
at org.jetbrains.kotlin.runner.Main.main(Main.kt:104)

E.g.

// DEPS

IF possible by also exposing given command line options.

I'm using IntelliJ to write kotlin script. In order to make my script valid with IntelliJ inspections, I'm defining a standard main method. The problem is that when running my file from kscript, my main method is not run at all.

Is there a special action to run the main method from the command line ?

Example

kscript - <<"EOF"
println("getting started")
generateSequence() { readLine() }.map {
    if (!it.startsWith(">")) it else
        "huhu" + it
}.forEach { println(it) }
EOF

This won't work at the moment because stidin is used for both script and data.

Possible workaround: add -s option to provide script as parameter and skip stdin for script provision.

streamline further by using simple code wrapper for

kscript -st '"huhu" + it'

maybe: trim string (for newline and whitespaces) and enable awk-mode if code starts with lines or stidin

To allow the user to tweak the used jvm.
Potentially also provide //KOTLIN_OPTS to configure kotlinc-jvm itself.

Not really needed for functioning but otherwise kscript will look unresponsive when downloading huge dependency trees.

• easier to read
• better if many dependencies are used

by default unless suppress with something like --quiet

• compile info
• dependency lookup info

debug:

• script and jar paths
• compile logs

E.g use https://gist.github.com/goodmami/6556701

This will also ease deployment.

Putting them next to the script is not always feasible/desired (e.g. because of lacking permissions)

I get errors trying inline example with kscript 2.1.0 and kotlin 1.1.51:

cat some_flights.tsv | kscript 'stdin.filter { "^de0[-0]*".toRegex().matches(it) }.map { it + "foo:" }.print()'

returns:

[ERROR] compilation of 'stdin.filter { "^de0[-0]".toRegex().matches(it) }.map { it + "foo:" }.print()' failed
/tmp/tmp6868296498685827648.tmp/scriptlet.b36b82442f201d8e.kts:1:1: error: cannot access 'stdin': it is private in file
stdin.filter { "^de0[-0]".toRegex().matches(it) }.map { it + "foo:" }.print()
^
/tmp/tmp6868296498685827648.tmp/scriptlet.b36b82442f201d8e.kts:1:7: error: unresolved reference. None of the following candidates is applicable because of receiver type mismatch:
public inline fun Array<out ???>.filter(predicate: (???) -> Boolean): List defined in kotlin.collections public inline fun BooleanArray.filter(predicate: (Boolean) -> Boolean): List defined in kotlin.collections public inline fun ByteArray.filter(predicate: (Byte) -> Boolean): List defined in kotlin.collections public inline fun CharArray.filter(predicate: (Char) -> Boolean): List defined in kotlin.collections public inline fun CharSequence.filter(predicate: (Char) -> Boolean): CharSequence defined in kotlin.text public inline fun DoubleArray.filter(predicate: (Double) -> Boolean): List defined in kotlin.collections public inline fun FloatArray.filter(predicate: (Float) -> Boolean): List defined in kotlin.collections public inline fun IntArray.filter(predicate: (Int) -> Boolean): List defined in kotlin.collections public inline fun LongArray.filter(predicate: (Long) -> Boolean): List defined in kotlin.collections public inline fun ShortArray.filter(predicate: (Short) -> Boolean): List defined in kotlin.collections public inline fun String.filter(predicate: (Char) -> Boolean): String defined in kotlin.text public inline fun Iterable.filter(predicate: (???) -> Boolean): List defined in kotlin.collections public inline fun Map.filter(predicate: (Map.Entry) -> Boolean): Map defined in kotlin.collections
public fun Sequence.filter(predicate: (???) -> Boolean): Sequence defined in kotlin.sequences
stdin.filter { "^de0[-0]".toRegex().matches(it) }.map { it + "foo:" }.print()
^
/tmp/tmp6868296498685827648.tmp/scriptlet.b36b82442f201d8e.kts:1:46: error: unresolved reference: it
stdin.filter { "^de0[-0]".toRegex().matches(it) }.map { it + "foo:" }.print()
^
/tmp/tmp6868296498685827648.tmp/scriptlet.b36b82442f201d8e.kts:1:58: error: unresolved reference: it
stdin.filter { "^de0[-0]*".toRegex().matches(it) }.map { it + "foo:" }.print()

Homebrew is the most used package manager on macOS: https://brew.sh/

This must not happen when the user has no internet connection

kscript --help
...
Copyright : 2017 Holger Brandl
License   : MIT
Version   : v2.1.0
Website   : https://github.com/holgerbrandl/kscript
Exception in thread "main" java.net.UnknownHostException: raw.githubusercontent.com
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
	at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
	at java.net.URL.openStream(URL.java:1045)
	at kscript.app.KscriptKt.versionCheck(Kscript.kt:280)
	at kscript.app.KscriptKt.main(Kscript.kt:54)

Hey, I installed it on mac exactly like in instructions. Not in /bin because it is not permitted in new mac, but in another folder and I added it to PATH, but when I start any script using kscript I get:

Exception in thread "main" java.lang.NoClassDefFoundError: kotlin/script/templates/standard/ScriptTemplateWithArgs
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at Main_Hello.main(Main_Hello.java:4)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jetbrains.kotlin.runner.AbstractRunner.run(runners.kt:60)
	at org.jetbrains.kotlin.runner.Main.run(Main.kt:104)
	at org.jetbrains.kotlin.runner.Main.main(Main.kt:110)
Caused by: java.lang.ClassNotFoundException: kotlin.script.templates.standard.ScriptTemplateWithArgs
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 19 more

Running kscript --help on my machine gives this :

kscript - Enhanced scripting support for Kotlin on *nix-based systems.

Usage : kscript <scriptfile, script-url, - for stdin, some kotlin code> [<script_args>]*
Usage : kscript --clear-cache # wipe cached script jars and urls
Usage : kscript -i # interactive shell with DEPS as declared in script

Copyright : 2017 Holger Brandl
License : MIT
Version : v1.5.0
Website : https://github.com/holgerbrandl/kscript

But when I try to run kscript --self-update (as described in the Readme) I get the following error :

/home/user/.kscript/scriptarg_cache_ad5dd5b728885618.kts:1:3: error: unresolved reference: self
--self-update
^
/home/user/.kscript/scriptarg_cache_ad5dd5b728885618.kts:1:3: error: variable expected
--self-update
^
/home/user/.kscript/scriptarg_cache_ad5dd5b728885618.kts:1:8: error: unresolved reference: update
--self-update
^
ERROR: compilation of '/home/user/.kscript/scriptarg_cache_ad5dd5b728885618.kts' failed

Which seems more natural than our current installation using curl.

Example

#!/usr/bin/env kscript
//DEPS de.mpicbg.scicomp.joblist:joblist-kotlin:1.1 de.mpicbg.scicomp:kutils:0.7

import de.mpicbg.scicomp.kutils.evalBash
evalBash("which STAR")

Feel free to steal code from

https://github.com/yschimke/oksocial-output/blob/master/src/main/java/com/baulsupp/oksocial/output/iterm/ItermOutputHandler.kt
https://github.com/yschimke/oksocial-output/blob/master/src/main/java/com/baulsupp/oksocial/output/OsxOutputHandler.kt

Features

• show previews inline
• open other files in preview etc