rec-framework / rec-core Goto Github PK

View Code? Open in Web Editor NEW

19.0 5.0 4.0 508 KB

Data pipelining service

Home Page: http://rec.kimleo.net

License: MIT License

Java 94.46% Shell 0.43% JavaScript 5.11%

modeling data-processing records stream

rec-core's People

Contributors

Stargazers

Watchers

Forkers

gitter-badger andy521 leexhwhy nnuujj

rec-core's Issues

[EPIC]Rec#reactive

Sub issues or PRs:

#44 require('rx') issue.

Replace hand-crafted logger with SLF4J

Currently rec-core using self-created logger util, and we need replace it with SLF4J which might be better integrated with other logger facility.

Migrate unit test to different modules and replacing with Java

Currently following test need to be migrated:

rec-accessor and record wrapper
rec-sepval parser
all rec-core test need to be migrated to Java

Add encoding convertor?

Issue found in #35 , when specified CSV file in utf-8 mode, it will cause potential encoding problem.

Suggest add a conversion method to convert file from one encoding to another.

Loader issue on Windows

Loader cannot get data file using rec file pattern

Fluquent API using operator overloading?

http://2ality.com/2011/12/fake-operator-overloading.html

Replace Kotlin script engine with JavaScript

Currently using kotlin has a lot of weird behavior, try switch JS instead.

As this is built on top of Java 8, an option is nashorn.

but not sure nashorn support for modern ES standard, still will using Rhino.

[EPIC] Rec-cloud requested features

Features:

CSV loader accepts Reader/InputStream instead of File.
Scripting.runScript accepts Reader/InputStream instead of file name.
ExecutionContext / 'rec' module should only shared by every scripting instance instead of static/global.
Add Resource interface which provides Source directly.

First line output as "." in version 1.3

RT.

Initializer bug on Windows

Reduce dependency by using Java instead of kotlin

RT.

Add specification document for DSLs: accessor, query selector, etc.

FlatFileTarget IndexOutOfBoundException

code:

const cnt = counter(function() true);
const file = flat('timestamps.csv');

csv('combined.csv', "...")
   .tee(cnt)
   .to(file);

file

blabal, ... ,,,

Caused by: java.lang.IndexOutOfBoundsException: Index: 13, Size: 13
        at java.util.ArrayList.rangeCheck(Unknown Source)
        at java.util.ArrayList.get(Unknown Source)
        at net.kimleo.rec.sepval.SepValEntry.get(SepValEntry.java:19)
        at net.kimleo.rec.sepval.SepValEntry.get(SepValEntry.java:7)
        at net.kimleo.rec.v2.accessor.RecordWrapper.getByIndex(RecordWrapper.java:41)
        at net.kimleo.rec.v2.accessor.RecordWrapper.get(RecordWrapper.java:34)
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.util.stream.ReferencePipeline.collect(Unknown Source)
        at net.kimleo.rec.v2.model.impl.FlatFileTarget.put(FlatFileTarget.java:30)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.util.Iterator.forEachRemaining(Unknown Source)
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.util.stream.ReferencePipeline.forEach(Unknown Source)
        at net.kimleo.rec.v2.model.impl.FlatFileTarget.putAll(FlatFileTarget.java:35)
        at net.kimleo.rec.v2.model.Source.to(Source.java:23)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
        ... 15 more

Travis CI cannot run checkstyle with Java 8 syntax by default

As showed in #39 .
Raised this issue to fix it.

Google DSPL support

https://developers.google.com/public-data/

Add loader and exporter for DSPL format.

Rec v2 API Specification & Guide

Discovered by #10 as we changed the API and architecture

[EPIC]Restartability

Fail fast
then restart.

Subissues and PR:

#45

MalformedInputException: Input length = 1

When access an ascii file.

Performance issue: Reduce memory cost for records

For release 0.1.0, it will cost more than 1 GB heap size when only 100,000 records.

need to be optimized.

Design Consideration: Splunk Search Processing Language

see: https://www.splunk.com/content/dam/splunk2/pdfs/solution-guides/splunk-quick-reference-guide.pdf

Enhancement: using Record#toString to get the original record.

This is useful when trying to keep original record without specify all the field names

Rec cannot use RxJS because cannot initialize global context

Rec get following error while loading and running rx.js:

=> Rec v2
Exception in thread "main" org.mozilla.javascript.JavaScriptException: Error: RxJS could not find any global context (window, self, global) (rx#34)
        at org.mozilla.javascript.gen.rx_3._c_anonymous_2(rx:34)
        at org.mozilla.javascript.gen.rx_3.call(rx)
        at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
        at org.mozilla.javascript.gen.rx_3._c_anonymous_1(rx:20)
        at org.mozilla.javascript.gen.rx_3.call(rx)
        at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
        at org.mozilla.javascript.gen.rx_3._c_script_0(rx:20)
        at org.mozilla.javascript.gen.rx_3.call(rx)
        at org.mozilla.javascript.gen.rx_3.exec(rx)
        at org.mozilla.javascript.commonjs.module.Require.executeModuleScript(Require.java:342)
        at org.mozilla.javascript.commonjs.module.Require.getExportedModuleInterface(Require.java:290)
        at org.mozilla.javascript.commonjs.module.Require.call(Require.java:219)
        at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63)
        at org.mozilla.javascript.gen.abc_js_2._c_script_0(abc.js:1)
        at org.mozilla.javascript.gen.abc_js_2.call(abc.js)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3282)
        at org.mozilla.javascript.gen.abc_js_2.call(abc.js)
        at org.mozilla.javascript.gen.abc_js_2.exec(abc.js)
        at org.mozilla.javascript.commonjs.module.Require.executeModuleScript(Require.java:342)
        at org.mozilla.javascript.commonjs.module.Require.getExportedModuleInterface(Require.java:290)
        at org.mozilla.javascript.commonjs.module.Require.requireMain(Require.java:137)
        at net.kimleo.rec.v2.scripting.Scripting.runReader(Scripting.java:34)
        at net.kimleo.rec.v2.scripting.Scripting.runfile(Scripting.java:22)
        at net.kimleo.rec.App.execute(App.java:66)
        at net.kimleo.rec.App.main(App.java:28)

Discovered when spike #34

Performance enhancement by add indecies for records

Performance issue found when using JS scripting tools:

RecSet#contains takes several seconds to check A contains B, when A has about 200,000 records.

Proposed using indices

rec-framework / rec-core Goto Github PK

rec-core's People

Contributors

Stargazers

Watchers

Forkers

rec-core's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs