cinchapi / concourse Goto Github PK
View Code? Open in Web Editor NEWDistributed database warehouse for transactions, search and analytics across time.
Home Page: http://concoursedb.com
License: Apache License 2.0
Distributed database warehouse for transactions, search and analytics across time.
Home Page: http://concoursedb.com
License: Apache License 2.0
Similar to the way the concourse-cli framework supports it
Add logic to validate each line of the import file for conformance with the spec as it is being processed.
For the get/select records (and maybe others) we should specify the size of the maps we're using where possible. This means replacing stuff like
Map<Long, Map<String, Set<TObject>>> result = Maps.newLinkedHashMap();
with
Map<Long, Map<String, Set<TObject>>> result = TMaps.newLinkedHashMapWithCapacity(records.size());
Same thing for Sets, if possible.
[default/cash]$ find_or_add "name", "jeff"
ERROR: No signature of method: org.cinchapi.concourse.Concourse$Client.find_or_add() is applicable for argument types: (java.lang.String, java.lang.String) values: [name, jeff]
Possible solutions: findOrAdd(java.lang.String, java.lang.Object), findOrAdd(java.lang.String, java.lang.Object)
[default/cash]$ findOrAdd "name", "jeff"
In CaSH, if compilation fails because of "No signature of method" and the method name has underscores, then convert the method name to camel case and try again
The Apache License from the root should be copied into each of the license dists. For java, this means putting it in the META-INF folder of the jar.
The NOTICE files for the drivers will probably be different than that which is included for the server, so each of the project directories should have a separate notice file, if necessary.
Though, it might not be necessary to include the NOTICE or the jars since dependencies are not being bundled.
For example, concourse-driver-php has an init-repo.sh script. This should be called from init.sh.
Similar to what was done in f369df1, but it needs to happen for Map and Timestamp.
BufferCapacityException
and throw that instead.I think we can do this for
But honestly, even for ones that do have state (i.e. a useful message and other attributes), we could use a singleton and just change the state whenever the exception needs to be thrown since CaSH is single threade
The Import framework allows people to specify a resolve key to find existing records where data should be placed. Now that the findOrInsert
method exists, extend the Import framework to take advantage by supporting resolve criteria.
[default/cash]$ add "baz", 10, 1
Returned 'true' in 0.011 sec
[default/cash]$ add "baz", 9, 1
Returned 'true' in 0.012 sec
[default/cash]$ add "baz", 9, 2
Returned 'true' in 0.01 sec
[default/cash]$ add "baz", 10, 2
Returned 'true' in 0.01 sec
[default/cash]$ select([1L, 2L])
Returned '
+------------------+
| Record | baz |
+------------------+
| 1 | [9, 10] |
| 2 | [9, 10] |
+------------------+
' in 0.027 sec
I thought the order of the values was supposed to be insertion order? but it seems to place them in sort order? Not sure if this is incorrect or not, but we should figure out.
Interesting enough, it always seems to display links in descending order:
+---------------------------------------------------------------------------------------------------------------+
| Record | bar | baz | foo | company | direct_reports | name | title | pi |
+---------------------------------------------------------------------------------------------------------------+
| 1447234653927000 | [1] | [true, @50] | [bar, baz] | null | null | null | null | null |
| 1447234653927006 | null | null | null | [Cinchapi] | [@2, @1] | [Jeff Nelson] | [CEO] | null |
| 1447234653928002 | null | null | null | null | null | null | null | [3] |
+---------------------------------------------------------------------------------------------------------------+
catch (Exception e) {
if(e.getCause() instanceof TTransportException) {
throw new ProgramCrash(e.getMessage());
}
else if(e.getCause() instanceof SecurityException) {
throw new ProgramCrash(
"A security change has occurred and your "
+ "session cannot continue");
}
else if(e instanceof CompilationFailedException) {
throw new MultiLineRequest(e.getMessage());
}
else if(e instanceof MissingMethodException
&& script != null
&& e.getMessage().contains(
"No signature of method: ConcourseShell.")) {
String method = e.getMessage().split("ConcourseShell.")[1]
.split("\\(")[0];
input = input.replaceAll(method, "ext." + method);
return evaluate(input);
}
else {
String message = e.getCause() instanceof ParseException ? e
.getCause().getMessage() : e.getMessage();
throw new EvaluationException("ERROR: " + message);
}
}
All the places where we do e instance of ...
should just be caught directly and handled in a separate block.
Use vagrant to setup an env with PHP 5.6 to checkout the code and run the unit tests. This is necessary to ensure that the logic for handling 64-bit pack/unpack works across versions.
TEST FAILURE in testMightContainLocatorKeyValue: null
---
locator = eqcicldw12dsowa7it4vi0pnqgewxci4c3ihyzf (org.cinchapi.concourse.server.model.Text)
value = w jvnwa8xofm6asavrgpyxpk1mbgah7slcaookolqo fpa3g5 5csjly (org.cinchapi.concourse.server.model.Value)
term = jvnwa8xofm6asavrgpyxpk1mbgah7slcaookolqo (org.cinchapi.concourse.server.model.Text)
record = 52259011321627880 (org.cinchapi.concourse.server.model.PrimaryKey)
[default/cash]$ concourse.getServerVersion
ERROR: null
Fatal error: Uncaught exception 'PharException' with message 'phar "/Users/jnelson/Dropbox/code/portland/concourse/concourse-driver-php/composer.phar" SHA1 signature could not be verified: broken signature' in /Users/jnelson/Dropbox/code/portland/concourse/concourse-driver-php/composer.phar:23
Stack trace:
#0 /Users/jnelson/Dropbox/code/portland/concourse/concourse-driver-php/composer.phar(23): Phar::mapPhar('composer.phar')
#1 {main}
thrown in /Users/jnelson/Dropbox/code/portland/concourse/concourse-driver-php/composer.phar on line 23
We had added timestamp with each TransactionToken precisely because we wanted to differentiate among different transactions that belonged to the same client session. However, it turns out that isn't really necessary because a client session can only have one transaction open at a time.
In actuality, we don't even need TransactionTokens anyway. We can just associate the AccessToken with the Transaction object and that should be sufficient. And, instead of sending over a possibly null TransactionToken for each RPC call to ConcourseServer, we could just send over a boolean that indicates whether the call should be routed to a transaction or not.
LineBasedImporter
to Importer
base class (e.g. transformValue()
, validateFileFormat()
, etc).transformValue
it'll have to be able to return something more generic than a JsonElementdef data = [
['name':'John Doe',
'department': 'Engineering',
'title': 'Senior Software Engineer',
'role': 'Software Engineer - Backend',
'manager': Link.toWhere('title = Director of Engineering'),
'salary': 10.00,
'location': 'Atlanta',
'exempt': true
]
]
def concourse = Concourse.connect();
println(concourse.select(concourse.insert(data)))
This yields
Caught: java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to com.google.common.collect.Multimap
java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to com.google.common.collect.Multimap
at com.cinchapi.concourse.util.Convert.mapsToJson(Convert.java:207)
at com.cinchapi.concourse.Concourse.insert(Concourse.java:1314)
at com.cinchapi.concourse.Concourse$insert$0.call(Unknown Source)
at quickstart.run(quickstart.groovy:20)
which happens because the Concourse API methods expect Multimap.
ConnectionPool
registers its own shutdown hook in order to ensure that the connections are released when the JVM dies. This behaviour isn't well documented and some well to do user might register their own shutdown hook to do the same.
Shutdown hooks in Java are run concurrently, so there may be a case of two threads trying to shutdown the pool at the same time.
The easiest fix for this should to change ConnectionPool#353
to
if(open.compareAndSet(true, false)){
in order to ensure that the check is atomic and two threads won't miss updates from each other.
Add an option for the user to specify the header on the command line. This will help in cases when the header is not provided in the file but the user doesn't want to write custom code to provide the header.
In the implementation, ImportCli
should set the header in each importer. Each importer should also check the first line of input against the header that was provided. If the first line of input is identical to the header, that line should be skipped during import. This is meant to help in cases where multiple files are being imported, but only one of them has the header. In that case, the user can simply specify the header on the command line.
Has a few errors right now:
Users/curtis/git/concourse/concourse-server/src/main/java/jsr166e/ConcurrentHashMapV8.java:4663: error: spliterator() in KeySetView cannot implement spliterator() in Set
public ConcurrentHashMapSpliterator<K> spliterator() {
^
return type ConcurrentHashMapSpliterator<K> is not compatible with Spliterator<K>
where K,E are type-variables:
K extends Object declared in class KeySetView
E extends Object declared in interface Set
/Users/curtis/git/concourse/concourse-server/src/main/java/jsr166e/ConcurrentHashMapV8.java:4729: error: spliterator() in ValuesView cannot implement spliterator() in Collection
public ConcurrentHashMapSpliterator<V> spliterator() {
^
return type ConcurrentHashMapSpliterator<V> is not compatible with Spliterator<V>
where V,E are type-variables:
V extends Object declared in class ValuesView
E extends Object declared in interface Collection
/Users/curtis/git/concourse/concourse-server/src/main/java/jsr166e/ConcurrentHashMapV8.java:4824: error: spliterator() in EntrySetView cannot implement spliterator() in Set
public ConcurrentHashMapSpliterator<Map.Entry<K, V>> spliterator() {
^
return type ConcurrentHashMapSpliterator<Entry<K,V>> is not compatible with Spliterator<Entry<K,V>>
where K,V,E are type-variables:
K extends Object declared in class EntrySetView
V extends Object declared in class EntrySetView
E extends Object declared in interface Set
It'd be nice to be able to query on the record id
where foo = bar and $id$ != 123
This should be allowable since there is a reserved identifier for referring to the id of a record.
username = foo
$ concourse shell
ERROR: Invalid username/password combination.
The issue here is that the logic for reading the prefs file always returns the default if its not present so it is returning "admin" as the password.
These tests should mimic the related ones in AtomicOperationWorkflowTest
http://refspecs.linuxfoundation.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html
The force-reload
action should be an alias for restart. In the future, it may actually reload the configuration.
When writing code that uses transactions, applications need to keep track of whether the client they are using is in a transaction so that they don't accidentally start a new one before committing.
public void methodA() {
concourse.stage();
try {
// do some work
concourse.commit();
}
catch(TransactionException e) {
concourse.abort();
}
}
public void methodB() {
concourse.stage();
try {
// do some work
methodA();
concourse.commit();
}
catch(TransactionException e) {
concourse.abort();
}
}
In the example above, the work that methodA() is done when called from methodB is an extension of the work methodB is doing, so the fact that methodB is already in a transaction should already be considered by the client when methodA is called
I can't think of a reason why the behaviour of calling stage when already in a transaction is to create a new transaction. I think its best to simply make it a no-op and continue with the current transaction, if it exists. This should be a small change.
Each client should just to see if the transaction token is not null, and if that is the case just return immediately.
We could implement this on the server if we edited the thrift API to take a transaction token in the stage
method.
[default/cash]$ select(["name", "salary", "title"], "role = Director and salary > 10")
Returned '
+--------------------------------------------------------------------+
| Record | title | name | salary |
+--------------------------------------------------------------------+
| 1448732216678007 | [Director of Engineering] | [Jane Doe] | [20.0] |
+--------------------------------------------------------------------+
' in 0.011 sec
The order of the columns should match the order of the input list. If that isn't possible, then it should be alphabetical.
concourse.add("foo", 1L, 1);
concourse.find(Criteria.where().key("foo").operator(Operator.EQUALS).value(1)); // empty but expect [1]
concourse.find(Criteria.where().key("foo").operator(Operator.EQUALS).value(1L)); // empty but expect [1]
The data is being force added as a Long, but the query interprets the value as an int, regardless of whether we use 1 or 1L
While executing following case :
callA(record); find_or_add('name', 'concourse'); callB(record"); add('name', 'jeff', 2);
handleShortSyntax() convert find_or_add() to find_or_concourse.add()
ER : concourse.find_or_add()
AR : find_or_concourse.add()
Gson does some special conversion for the built in java map that causes it to serialize Link
as a JSON object.
Repro:
@Test
public void testMapWithLinkObjectToJson(){
Map<String, Object> data = Maps.newHashMap();
data.put("foo", Link.to(1));
String json = Convert.mapToJson(data);
Assert.assertEquals("{\"foo\":\"@1\"}", json);
}
error:
org.junit.ComparisonFailure: expected:<{"foo":["@1"]}> but was:<{"foo":[{"record":1}]}>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at com.cinchapi.concourse.util.ConvertTest.testMapWithLinkObjectToJson(ConvertTest.java:637)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
https://github.com/cinchapi/concourse/blob/develop/update-fork.sh
Only perform git stash pop
if git stash
actually stores something. We'll know if something was stashed if the output from the command isn't equal to No local changes to save
The add
, set
and remove
methods have a signature with order key, value, record
The link
and unlink
methods have a signature with order key, source, destination
it would probably be more intuitive to have the link signature be key, destination, source
The ete test framework should use the skip-integration
flag when installing the managed concourse server. Right now, the framework doesn't use the flag and employs a hack where it kills the installer process after 1 second in order to avoid hanging on waiting for the admin password.
Its possible the logic will need to check if the version >= 0.5.0 before trying to use the flag.
The POM file that is generated for Java artifacts using the fully qualified version number ([X.Y.Z.B[-SNAPSHOT]) instead of the correct X.Y.Z[-SNAPSHOT] version number that conforms to what is used for the pom.version
In the doJsonify method, I didn't know if I should've used StringBuilder or StringBuffer.
[default/cash]$ find("location = Atlanta \\(HQ\\)")
ERROR: Syntax error in [location = Atlanta \( HQ\, )]: Mismatched parenthesis
and use G1GC..its time ๐
Right now the script is place in /usr/local/bin, which is cool, but a symlink should also be added to /etc/init.d. If we do add this symlink then we must be sure to remove it when the uninstall script is invoked.
Right now the initialization script for the upgrade framework runs as part of the server installation process. The purpose of this script is to place the schema version files in the data directories so it should not run before the user has had a chance to modify concourse.prefs to set the appropriate data directories.
The correct way to do this is for ConcourseServer to check the data directories for the schema files. If they don't exist, then ConcourseServer should spawn a separate JVM to run initialization.
Additionally, instead of putting this logic in ConcourseServer.java, we can add it to the concourse ctrl script to check for the presence of the schema files and run the initializer if necessary. I don't like this approach as much though because it requires the implementation of more one off pref parsing. Nevertheless it does have the benefit of making the JVM spawning/displaying console logging easier and more natural for the user.
We depend on the .maven
file to tell Gradle the correct repo for archive uploading. Why don't we just add logic that checks the correct conditions in code:
maven {
credentials {
username "admin"
password "password"
}
if(project.version.endsWith('-SNAPSHOT')) {
url "http://xxx"
} else {
url "http://xxx"
}
}
Instead of
Map<Long, Set<T>> pretty = PrettyLinkedHashMap
.newPrettyLinkedHashMap("Record", key);
for (Entry<Long, Set<TObject>> entry : raw.entrySet()) {
pretty.put(entry.getKey(), Transformers.transformSet(
entry.getValue(),
Conversions.<T> thriftToJavaCasted()));
}
should be doing something like
Map<Long, Set<T>> transformed = Maps.transformValues(raw,
TransformFunctions.set(Conversions
.<T> thriftToJavaCasted()));
assuming TransformFunctions.set looks like
public static <V1, V2> Function<Set<V1>, Set<V2>> set(
final Function<V1, V2> function) {
return new Function<Set<V1>, Set<V2>>() {
@Override
public Set<V2> apply(Set<V1> input) {
return Transformers.transformSet(input, function);
}
};
}
Only challenge here is how do we transformed
to be pretty lazily?
Allow CCL statements to contain ||
for OR and &
/&&
for AND
Similar to CON-237
Need to profile what slow downs may (or may not) be occurring with G1GC (in Java 8...not Java 7) and optimize where necessary.
This will make debugging A LOT easier. Right now those logs get deleted when the temporary server is wiped away
diff "1452270300424000", time("last week")
seems to cause an infinite loop
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.