deanhiller / playorm

ORM for nosql with SCALABLE SQL

All code will remain on github but documentation can be found at http://buffalosw.com/wiki/playorm-documentation/ and the feature list can be found at http://buffalosw.com/products/playorm/

playorm's Issues

javadoc all the api packages in playorm

Users LOVE the eclipse feature of javadoc popping up except we really lack javadoc info in the api packages so let's add all the javadoc info we need to so we can make the api even more usable.

thanks,
Dean

Need support for NoSqlEmbedded in command line tool

Command line tool is not supporting NoSqlEmbedded yet.

Support for simple NoSqlEmbeddable without NoSqlId

It would be nice to allow NoSqlEmbeddable classes that are just a simple container.

For example:

@NoSqlEntity
public class User {
@NoSqlId
private String id;
@NoSqlEmbedded
private Name name;
}
@NoSqlEmbeddable
public class Name {
private String firstName;
private String lastName;
}

A user (usually) just has one name. But "Name" could be easily reused in other entities.

So Name would not require an ID and all fields in Name could be persisted using column names like "name.firstname", "name.lastname" etc.

This implies that Name cannot be used in lists.

Astyanax retry policy is missing as configuration option

To make PlayOrm more robust it would be very useful to add configuration for Astyanax's retry policy:

CassandraSession.java:

ColumnList<String> result = keyspace.prepareQuery(CF_STANDARD1)
    .withRetryPolicy(new ExponentialBackoff(250, 5))
    .getKey(rowKey)
    .execute().getResult();

So for example in application.conf we could have:

nosql.cassandra.retryPolicy=none, ntimes, exponential, constant, boundedexponential

nosql.cassandra.retryPolicy.ntimes.maxAttempts=5

nosql.cassandra.retryPolicy.constant.sleepTime=250
nosql.cassandra.retryPolicy.constant.maxAttempts=5

nosql.cassandra.retryPolicy.exponential.baseSleepTime=250
nosql.cassandra.retryPolicy.exponential.maxAttempts=5

nosql.cassandra.retryPolicy.boundedexponential.baseSleepTime=250
nosql.cassandra.retryPolicy.boundedexponential.maxSleepTime=5000
nosql.cassandra.retryPolicy.boundedexponential.maxAttempts=5

Alternatively we can make it simpler in the config and then decide which mode to use in the code:

nosql.cassandra.retry=true
nosql.cassandra.retry.maxAttempts=10
nosql.cassandra.retry.useExponentialSleepTime=true
nosql.cassandra.retry.sleepTimeBetweenAttempts=250
nosql.cassandra.retry.maxSleepTimeBetweenAttempts=5000

See here:
https://github.com/Netflix/astyanax/wiki/Customizing-calls

Annotations e.g. NoSqlIndexed should have inherited set

I have two classes:

BaseModel {
@NoSqlIndexed
public String id;
}

@NoSqlQuery(name="findById" ....)
Car extends BaseModel {
}

Playorm will complain about @NoSqlIndexed being missing because it is set in the super class. The @inherited annotation should be set on all relevant annotations e.g. NoSqlOneToMany etc.

Hex code of a field remains in play during re-scan

During re-scan, the hex-code of the column name is still coming if we delete that field in PlayFramework without re-starting the server.

NPE when putting entity with null @NoSqlEmbedded/List

When calling EntityManager.put with an entity like this:

@NoSqlEntity
public class Breed {

@NoSqlId
private UUID id;

@NoSqlIndexed
private String officialName;

@NoSqlEmbedded
private List<LocalizedString> i18nNames;

}

where just "officialName" is not null, causes the following exception. I'd expect playorm to assume that the list is empty. I do not like to initialize lists in with "new ArrayList()", because these instances are usually nothing else than garbage.

java.lang.NullPointerException
at com.alvazan.orm.impl.meta.data.MetaEmbeddedEntity.translateToColumnImpl(MetaEmbeddedEntity.java:203)
at com.alvazan.orm.impl.meta.data.MetaEmbeddedEntity.translateToColumnList(MetaEmbeddedEntity.java:180)
at com.alvazan.orm.impl.meta.data.MetaEmbeddedEntity.translateToColumn(MetaEmbeddedEntity.java:154)
at com.alvazan.orm.impl.meta.data.MetaClassSingle.translateToRow(MetaClassSingle.java:87)
at com.alvazan.orm.layer0.base.BaseEntityManagerImpl.putImpl(BaseEntityManagerImpl.java:108)
at com.alvazan.orm.layer0.base.BaseEntityManagerImpl.put(BaseEntityManagerImpl.java:93)

Querying without flushing doesn't appear to work

Have the following situation (note that ids are manually generated in the constructor and I have the proper annotation):

1. Fetch User object with id. This works fine.
2. Create some new Car objects and then save using NoSql.em().put(car, true) as it is a new insert.
3. Do a named query to fetch these Car objects e.g. select * from Card as c where c.userId = :userId and c.carType = :carType
4. No cars are found.

However if I do a NoSql.em().flush() after the create then the named query finds the cars.

I have checked the hashcode for the session and both the creates/saves are done in the same session.

RFE/help: "sql log"

Hi,

is there some kind of log, that a developer can use to trace which operations are performed against cassandra?

Just like Hibernate's "show_sql" switch.

Would be cool to see find/findAll operations,
row creates, updates, deletes,
how SJQL queries are executed,
more??

Warning for missing index needs to be updated.

If you have a NoSQLQuery that references an unindexed attribute it throws the following warning:

Caused by: java.lang.IllegalArgumentException: You cannot have 'result.id' in your sql query since id has no @Index annotation on the field in the entity
at com.alvazan.orm.parser.antlr.ScannerSql.processColumnName(ScannerSql.java:541)
at com.alvazan.orm.parser.antlr.ScannerSql.processSide(ScannerSql.java:443)

It should throw the warning:
You cannot have 'result.id' in your sql query since id has no @NoSqlIndexed annotation on the field in the entity.

Add support for Delete in SJQL

It would be useful to be able to have deletes in SJQL e.g.

delete from Budget as b where b.userId = :userId and b.status = :status

RFE: Add method "parameter(String,Object)" as enhancement

Please add an additional method

public Query parameter(String name, Object value);

which does exactly the same as setParameter() but returns "this" to be able to use fluent query parametrization.

Just like this.

entityManager.createNamedQuery(User.class, "byManyParameters").
parameter("firstName", "Robert").
parameter("email", "[email protected]").
paramerter(........).
paramerter(........).
paramerter(........).
getSingleResult();

Astyanax connection pool options are missing as configuration options

To make PlayOrm more configurable it would be very useful to add configuration for Astyanax's array of connection pool configuration options:

So for example in application.conf we could have:

nosql.cassandra.initialConnectionsPerHost=2
nosql.cassandra.maxConnectionsPerHost=2
nosql.cassandra.writeConsistencyLevel=Quorum
nosql.cassandra.readConsistencyLevel=Quorum

nosql.cassandra.useLatencyAwareScoring=true
nosql.cassandra.latencyAwareUpdateInterval=10000
nosql.cassandra.latencyAwareResetInterval=10000
nosql.cassandra.latencyAwareBadnessThreshold=0.5
nosql.cassandra.latencyAwareWindowSize=100

See here:
https://github.com/Netflix/astyanax/wiki/Configuration

Base class for Model objects

Just wondering whether there was a plan to have a base class for model objects:

e.g. User extends NoSQLModel.

Or if you definitely prefer having a normal POJO. I personally don't have an opinion either way.

Publish playorm in maven Nexus repository

It would be easier to use the project if it was available at a maven repo

Simple details on how to do it from the maven side: http://apache.org/dev/publishing-maven-artifacts.html#signing-up

Wiki: document what and how "schema" changes are supported

One of the reasons I would like to use Cassandra+playorm is that the "database" itself is not bound to a fixed schema. Means: I can add and remove fields and relations to/from my entities as required.

Can you document what kind of schema updates are possible and recommended and what schema updates are not possible and will cause conflicts (e.g. changing field types from string to int - resp. binary to string representation).

Even the possibility to change *ToOne to *ToMany relations and vice versa is interesting. But changing a *ToMany relation to a *ToOne relation will cause problems (which problems exactly??) when a single object as multiple references in the *ToMany.

Play support needs to be added

Noticed there was some rumor of this being added.

But wanted an issue just to keep track of it.

Change in command line tool output

For composite keys, command line tool is giving Long number =null instead of Entity.instance= null

Possibility to easily import data mapped by playOrm to SOLR

We have an old system which uses DataImportHandler to import data from PostgreSQL. The way we use it is:

We configure a xml with selects what will be indexed into solr from our database. Take a look at this: http://wiki.apache.org/solr/DataImportHandler#Full_Import_Example
We call URLs to perform the import. http://localhost:8983/solr/db/dataimport?command=full-import performs a full import and http://localhost:8983/solr/dataimport?command=delta-import imports only what wasn't imported since last time.

I am not saying we need exactly this solution, I just told you about DataImportHandler because we are used to it. However, your idea of Listeners sounds like something that would help realtime indexing, so that could be even better. Would just need some documentation or explanation on how to do it.
The most important thing is: we need to select what goes indexed by SOLR and how it goes. For instance, I might have two CFs: User and City

I might want to import just a document to SOLR, User document, and each user goes together with his/her associated cities. I might want to import the address column, but not import the birthDate column
I might want to import to documents to SOLR, User document and City document, each one independently.

---

use case:

Imagine a very big and complex data model.
I have the following entities:
User (has id, name, birthDate, List...)
UserRequests
City
UserLikes
UserInterests
etc.

All that goes to solr, but there is only one document: user.
A second scenario would be City as separate document in Solr too.
What if I receive a new user interest? I will add one more interest to the UserInterests CF, that would be the change in cassandra.
In Solr, however, I would need to reindex the entire user, as SOLR, AFAIK, doesn't allow you to reindex only part of the document, you can whether delete the indexed document or replace it, you can't update.

In the example bellow, savingEntity object would receive UserInterest entity, wouldn't it? But I want to reindex user.

Remember that realtime is good, but having the possibility to do it in batches is also desirable. Suppose the following scenario, indexing in real time:
I add a new interest to the user.
I add a new like to the user.
I add a new request to the user.

Indexing the entire user every event could be problematic in some cases, so in some cases it might be better to perform a delta index every hour, for instance... Final users still have near real time data, but the processing amount needed from the server decreases a lot.

artifact playorm-*-sources.jar should contain the sources

Last ticket for today ;-)

Would be nice to have the sources really in the sources jar.

Sources just blow up the "classes" jar and for example IntelliJ IDEA does not find any sources because the -sources.jar is "empty".

cannot add @NoSqlIndexed to DateTime right now

Also check if we can support LocalDateTime(I thought we could). Also in this ticket, add querying using some kind of date format as well in the JQL statements so we can make this much easier for people.

need @NoSqlIndexed(byKeyOnly=true)

I keep needing this so I am finally adding the issue in here.....

I need a property byKeyOnly so when we have a class like this

private UserDbo {

@NoSqlId
private String id;

@NoSqlIndexed(byKeyOnly=true)
 private String email;

}

where this will create the typical UserDbo table BUT instead of creating a normal index, it will create a table called emailToUserDbo such that I can do a JPA call of query="select u from TABLE as u where u.email = :email" BUT this query ONLY does = and does not < nor > or anything else.

Updating this is also interesting as if email changes you must delete the previous row in the xref table.

The emailToUserDbo table is very simple...one key=email and one column with name="value" and value="user row key"

It is VERY important that an index is NOT used as this pattern scales to trillions of UserDbo's where using an index does not

later,
Dean

Warning for custom converter needs to be updated

When dealing with objects that have no built in converter the following is shown:

Caused by: java.lang.IllegalArgumentException: .... You need to either add one of the @*ToOne annotations, @Embedded, @transient or add your own converter calling EntityMgrFactory.setup(Map<Class, Converter>) which will then work for all fields of that type OR add @column(customConverter=YourConverter.class) or finally if we missed a standard converter, we need to add it in file InspectorField.java in the constructor and it is trivial code(and we can copy the existing pattern)

This should be updated to mention the new @NoSqlConverter annotation.

And this part (EntityMgrFactory.setup) should be removed as you can't do this from the Play plugin.

command line up arrow/down arrow should work+history

unfortunately this requirement requires a little rework of the command line as to have the up arrow show the last command in the history, the command line will have to change to read and buffer every character typed in instead of what it does currently.

currently it only reads in an entire line so it will not read until you hit the enter key. This needs to change so we can "hear" the up arrows and such and display the last command and such on the command line (just like bash shell works with it's history).

(I keep hitting the up arrow and it is kind of annoying that it doesn't work).

Not implemented for update

Hi, I am using cassandra with play and when I try to use the "update" action in AutoCreateEnum, i got "Not Implemented" exception. Are there any plan to implement update as well in near future or any approximate timeline when we can expect?

Adding contains method to NoSqlEntityManager

I don't know the best way to do this but it would be nice if there was a method in NoSqlEntityManager:

public boolean contains(Class entityType, String key, String value)

That didn't load in any data.

add documentation ...deletes, etc.

we need to add documentation on JPA deletes in the wiki(and updates as well?) and how to use them, and add documentation into the command line tool as well for anything we have added.

We need documentation on deleting a single column in command line tool (in the wiki and in command line tool maybe for this one).

Possibility to add List<String> fields in playORM

I would like to be able to have a field of type List in my playOrm entity. Also, I would like this field to be magically persisted no my mapped column family.

Original stackoverflow issue:
http://stackoverflow.com/questions/12945656/is-it-possible-to-have-a-converter-for-a-list

Note:
It doesn't need to be a converter at all, Dean were suggesting to use the "@NoSqlEmbedded" annotation instead. If there is a way of storing a list instead of creating another CF just for that, it would be preferable. I would like to know the possibilities and limitations to indexes and partitions when I use this feature.

@NoSqlEmbedded need to work for user defined entities

@NoSqlEmbedded is working for Primitive data types only. Need to work on MetaEmbedded Entity to support other user defined entities

Wiki: document which object instances are "automatically" updated

Comparing with Hibernate:
If you retrieve an object from Hibernate, change any fields and just flush() the Hibernate session without explicitly updating the changed object, the object's changed get persistent.

This is not what playorm does.

Is there any mechanism when objects get automatically updated?
Are changes to transitively reachable objects persisted? For example put() with objectA that has a reference to changed objectB, but objectA has not been changed.

BaseEntityManagerFactory needs to remove @Inject annotation

We could not re-use playorm in another Guice bound application since it tries to rebind BaseEntityManagerFactory. That is the only one that should not be allowed to have @Inject annotation and instead should be injected manually in our BootstrapImpl class or something. We can test if it works with MonitorService in playorm which we tried to bind and instead had to manually inject stuff all over the place.

Support for ALTER table

Need to see, to what extent support can be provided for ALTER table statement.

i.e., Check if we can add/drop/modify a column!

ALTER TABLE table_name
DROP COLUMN column_name

ALTER TABLE table_name
ADD column_name datatype

ALTER TABLE table_name
ALTER COLUMN column_name datatype

ALTER TABLE table_name
MODIFY column_name datatype

convert playorm builds to gradle

Also, in the conversion to gradle, see if we can run all the tests in one JVM as this speeds up the build by a factor of 10 times. Right now, ant runs each junit class in it's own jvm and so it is very slow rescanning all the classes every single time. Also, gradle cuts down on a huge amount of code needed in the template.

ALSO, the gradle build file should be made such that we can easily re-use on other projects but still runs checkstyle and still runs verifydesign, etc. etc.

Lastly, gradle should be checked in and the correct files ignored before checkin(like .gradle should not be checked in ;) ). At the end of this change, I should run ./build.sh and it works or ./build.sh and it prints out the gradle help.

UniqueKeyGenerator should be able to work without network cards

I just changed my WLAN connection, started my app while my mac had no WLAN connected and got "RuntimeException: Call to InetAddress.getLocalHost().getHostname() == localhost. ...."

No idea yet how to solve this, but another thing I would like to see to be configurable in NoSqlEntityManagerFactory.

Just imagine that a customer has some data centers and some racks in each data center and a FQDN structure like this:
machine-n.rack-n.dc-n.lan

(off topic: @deanhiller now that I've seen UniqueKeyGenerator code I know what you meant with currentTimeMillis+1 ;-) )

Do we always need Index to query?

If I have an entity like:
public class Entity{

@NoSqlId
private String id;

private String name;

}
and then, when I am doing "select * from Entity"?

I am getting an exception like this:

layorm >> 2013-01-14 18:26:03,737 - Caller+0 at com.alvazan.ssql.cmdline.PlayOrm.processAnotherLine(PlayOrm.java:124)
WARN: Exception occurred
java.lang.IllegalArgumentException: The table=Entity has no columnes with indexes. ie. no entity attributes had the @NoSqlIndexed annotation
at com.alvazan.orm.api.z8spi.meta.DboTableMeta.getAnyIndex(DboTableMeta.java:402) ~[eclipsegen/:na]
at com.alvazan.orm.layer5.query.SpiIndexQueryImpl.getResultListImpl(SpiIndexQueryImpl.java:82) ~[eclipsegen/:na]
at com.alvazan.orm.layer5.query.SpiIndexQueryImpl.getResultList(SpiIndexQueryImpl.java:73) ~[eclipsegen/:na]
at com.alvazan.orm.layer3.typed.NoSqlTypedSessionImpl.createQueryCursor(NoSqlTypedSessionImpl.java:227) ~[eclipsegen/:na]
at com.alvazan.ssql.cmdline.CmdSelect.processSelect(CmdSelect.java:22) ~[eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.processCommand(PlayOrm.java:174) [eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.process(PlayOrm.java:138) [eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.processAnotherLine(PlayOrm.java:116) [eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.start(PlayOrm.java:105) [eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.main(PlayOrm.java:84) [eclipsegen/:na]

Is it an expected behavior? I think now as we have capability to query on primary key we should query on primary key instead of any index column in absence of WHERE clause?

need to support the in attribute (especially for row keys)

Let's implement row keys on this one first as it is very useful as I need to select x from TABLE as x where x.id in ('key1', 'key2', 'key3');

thanks,
Dean

ToOne needs to change storage model to match ToMany

MetaProxyField needs to be more like MetaToMany when it comes to how data is stored..

ToMany is stored like so. If you have a Shirt which has a List users; then the storage is colname=users:userFk1 val=null and colname=users:Fk2 value=null, etc. etc for each user in the list.

For a ToOne, we store it like colname=users value=Fk1. We need to change it to store like the above ToMany relationship instead.

Possibility to add UUID fields to entities and map these fields in playORM

Original issue at Stackoverflow: http://stackoverflow.com/questions/12945338/is-there-any-converter-for-uuid-type-in-playorm

I would like to have a UUID field in my playORM mapped class. It can be
type 1 UUID as in that 3rd party library (http://johannburkard.de/blog/programming/java/Java-UUID-generators-compared.html) which is guaranteed to be unique, for instance. I would like playORM to have such a feature and a recommended way of using it, so as a user I don't have to worry much about it, I would like just to map my entity in playORM and start using, if it's possible.

PlayOrm does not work with PlayFramework 2.x

We are starting new projects using the latest version of the play framework, it makes sense for us to use the newest version of the framework as we are starting from scratch

We would very much like to use the PlayOrm with Play and would be interested in getting a 2.x plugin for the PlayOrm

We would be using Java, at least initially...

Thanks

NPE when saving null list with toMany attribute

public class Goal {
@NoSqlOneToMany
public List cards;
}

Will throw a NullPointerException when saving.
com.alvazan.orm.impl.meta.data.MetaToManyField.translateToColumnImpl(MetaToManyField.java:164

Which is this line in my version of Playorm:
//now process all the existing columns (we can add same entity as many times as we like and it does not get duplicated)
for(PROXY proxy : toBeAdded)

NoSqlManyToOne relation to entity with UUID as ID fails

persisting an object that has a NoSqlManyToOne relation to another entity with UUID as ID type fails with "InvalidRequestException(why:String didn't validate.)"

command line needs noSQL type join in this case

In many cases, fk's are embedded in the row. It would be nice to do a join and just use those fks and not even use indices to get info on child objects of some other rows.

Also, make the up and down arrows work in the command line tool.

Run-time dependencies

There is a need to de-couple Astyanax runtime dependency from PlayOrm for those who want to evaluate PlayOrm with in-memory database only.

need a list tables query in command line tool

I am starting to forget table names so I need a list function of some sort so I can see them. We have 30,000 virtual tables so I would really like to filter out virtual tables as well sometimes.

thanks,
Dean

To support JOIN in UPDATE query

There is a need to support JOIN in UPDATE query..something like:

update product as p inner join p.productPrice as pp SET pp.price = 0.8 WHERE p.dateCreate < '2004-01-01'

Right now it is simple UPDATE like
update Account set (name="newName", isActive = true) where id = "acc1";

Separators for composite / embedded column names

Would be nice to have separators between the logical column name parts.

For example: The column "namemainfirstName" in the following screenshot is the field "firstName" in the composite "name" with id "main".

It would get more readable with separators.
And (I did not try it) but the following might generate a column name conflict:

@NoSqlEntity
class User {
String avalue;
@NoSqlEmbedd
A a;
}
@NoSqlEmbeddable
class A {
String value;
}

Would say: it is possible that a composite field (NoSqlEmbedded in List) accidently might get the same column name as another field - when the embedded's id is specified by the user.

NPE while iterating over "empty" cursor

Scenario:

I've removed "nosqlkeyspace".
Then I execute
Query q = entityManager.createNamedQuery(User.class, "findByUsername");
q.setParameter("username", "user-a");
Cursor<KeyValue> cursor = q.getResults(false);
while (cursor.next())
entityManager.remove(cursor.getCurrent().getValue());

and the following exception occurs:

Caused by: java.lang.NullPointerException
at com.alvazan.orm.layer9z.spi.db.cassandra.CursorReturnsEmptyRows.nextImpl(CursorReturnsEmptyRows.java:31)
at com.alvazan.orm.layer0.base.CursorRow.nextImpl(CursorRow.java:40)
at com.alvazan.orm.api.z8spi.iter.AbstractCursor.next(AbstractCursor.java:10)

Note that the "User" table does not exist.

Entity with ID of type com.eaio.uuid.UUID plus any NoSqlIndexed column cannot be deleted

Playorm complains about a missing CF "BytesIndice" when trying to delete an entity like the following. It works if the ID type is changed to java.lang.String.

@NoSqlEntity
@NoSqlQueries({@NoSqlQuery(name = "findByUsername", query = "select * from User as a WHERE a.username=:username")})
public class User {

@NoSqlId
private UUID id;

@NoSqlIndexed
private String username;

private String password;

@NoSqlIndexed
private String email;

@NoSqlEmbedded
private Name name;

...
}

@NoSqlEmbeddable
public class Name implements Serializable {

@NoSqlId
private BigInteger id;

private String firstName;

private String midName;

private String lastName;

...
}

Astyanax stable is now at 1.56.18.

There seems to be some breakage so thought would create an issue to track.

e.g. no such method for com.netflix.astyanax.Cluster.addKeyspace when starting.

Constants in NoSqlEmbeddables handled as instance field

It seems that constants like

@NoSqlEmbeddable
public class Name {
private static final String MAIN = "main";
...
}

are handled as instance fields.

Playorm persisted a column called "MAIN" but fails on later operations with the "MAIN" field.