deanhiller / playorm Goto Github PK
View Code? Open in Web Editor NEWORM for nosql with SCALABLE SQL
Home Page: https://github.com/deanhiller/playorm/wiki
License: Other
ORM for nosql with SCALABLE SQL
Home Page: https://github.com/deanhiller/playorm/wiki
License: Other
Playorm complains about a missing CF "BytesIndice" when trying to delete an entity like the following. It works if the ID type is changed to java.lang.String.
@NoSqlEntity
@NoSqlQueries({@NoSqlQuery(name = "findByUsername", query = "select * from User as a WHERE a.username=:username")})
public class User {
@NoSqlId
private UUID id;
@NoSqlIndexed
private String username;
private String password;
@NoSqlIndexed
private String email;
@NoSqlEmbedded
private Name name;
...
}
@NoSqlEmbeddable
public class Name implements Serializable {
@NoSqlId
private BigInteger id;
private String firstName;
private String midName;
private String lastName;
...
}
Noticed there was some rumor of this being added.
But wanted an issue just to keep track of it.
Please add an additional method
public Query parameter(String name, Object value);
which does exactly the same as setParameter() but returns "this" to be able to use fluent query parametrization.
Just like this.
entityManager.createNamedQuery(User.class, "byManyParameters").
parameter("firstName", "Robert").
parameter("email", "[email protected]").
paramerter(........).
paramerter(........).
paramerter(........).
getSingleResult();
To make PlayOrm more robust it would be very useful to add configuration for Astyanax's retry policy:
CassandraSession.java:
ColumnList<String> result = keyspace.prepareQuery(CF_STANDARD1)
.withRetryPolicy(new ExponentialBackoff(250, 5))
.getKey(rowKey)
.execute().getResult();
So for example in application.conf we could have:
nosql.cassandra.retryPolicy=none, ntimes, exponential, constant, boundedexponential
nosql.cassandra.retryPolicy.ntimes.maxAttempts=5
nosql.cassandra.retryPolicy.constant.sleepTime=250
nosql.cassandra.retryPolicy.constant.maxAttempts=5
nosql.cassandra.retryPolicy.exponential.baseSleepTime=250
nosql.cassandra.retryPolicy.exponential.maxAttempts=5
nosql.cassandra.retryPolicy.boundedexponential.baseSleepTime=250
nosql.cassandra.retryPolicy.boundedexponential.maxSleepTime=5000
nosql.cassandra.retryPolicy.boundedexponential.maxAttempts=5
Alternatively we can make it simpler in the config and then decide which mode to use in the code:
nosql.cassandra.retry=true
nosql.cassandra.retry.maxAttempts=10
nosql.cassandra.retry.useExponentialSleepTime=true
nosql.cassandra.retry.sleepTimeBetweenAttempts=250
nosql.cassandra.retry.maxSleepTimeBetweenAttempts=5000
See here:
https://github.com/Netflix/astyanax/wiki/Customizing-calls
In many cases, fk's are embedded in the row. It would be nice to do a join and just use those fks and not even use indices to get info on child objects of some other rows.
Also, make the up and down arrows work in the command line tool.
If I have an entity like:
public class Entity{
@NoSqlId
private String id;
private String name;
}
and then, when I am doing "select * from Entity"?
I am getting an exception like this:
layorm >> 2013-01-14 18:26:03,737 - Caller+0 at com.alvazan.ssql.cmdline.PlayOrm.processAnotherLine(PlayOrm.java:124)
WARN: Exception occurred
java.lang.IllegalArgumentException: The table=Entity has no columnes with indexes. ie. no entity attributes had the @NoSqlIndexed annotation
at com.alvazan.orm.api.z8spi.meta.DboTableMeta.getAnyIndex(DboTableMeta.java:402) ~[eclipsegen/:na]
at com.alvazan.orm.layer5.query.SpiIndexQueryImpl.getResultListImpl(SpiIndexQueryImpl.java:82) ~[eclipsegen/:na]
at com.alvazan.orm.layer5.query.SpiIndexQueryImpl.getResultList(SpiIndexQueryImpl.java:73) ~[eclipsegen/:na]
at com.alvazan.orm.layer3.typed.NoSqlTypedSessionImpl.createQueryCursor(NoSqlTypedSessionImpl.java:227) ~[eclipsegen/:na]
at com.alvazan.ssql.cmdline.CmdSelect.processSelect(CmdSelect.java:22) ~[eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.processCommand(PlayOrm.java:174) [eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.process(PlayOrm.java:138) [eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.processAnotherLine(PlayOrm.java:116) [eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.start(PlayOrm.java:105) [eclipsegen/:na]
at com.alvazan.ssql.cmdline.PlayOrm.main(PlayOrm.java:84) [eclipsegen/:na]
Is it an expected behavior? I think now as we have capability to query on primary key we should query on primary key instead of any index column in absence of WHERE clause?
I have two classes:
BaseModel {
@NoSqlIndexed
public String id;
}
@NoSqlQuery(name="findById" ....)
Car extends BaseModel {
}
Playorm will complain about @NoSqlIndexed being missing because it is set in the super class. The @inherited annotation should be set on all relevant annotations e.g. NoSqlOneToMany etc.
During re-scan, the hex-code of the column name is still coming if we delete that field in PlayFramework without re-starting the server.
Would be nice to have separators between the logical column name parts.
For example: The column "namemainfirstName" in the following screenshot is the field "firstName" in the composite "name" with id "main".
It would get more readable with separators.
And (I did not try it) but the following might generate a column name conflict:
@NoSqlEntity
class User {
String avalue;
@NoSqlEmbedd
A a;
}
@NoSqlEmbeddable
class A {
String value;
}
Would say: it is possible that a composite field (NoSqlEmbedded in List) accidently might get the same column name as another field - when the embedded's id is specified by the user.
Users LOVE the eclipse feature of javadoc popping up except we really lack javadoc info in the api packages so let's add all the javadoc info we need to so we can make the api even more usable.
thanks,
Dean
Also, in the conversion to gradle, see if we can run all the tests in one JVM as this speeds up the build by a factor of 10 times. Right now, ant runs each junit class in it's own jvm and so it is very slow rescanning all the classes every single time. Also, gradle cuts down on a huge amount of code needed in the template.
ALSO, the gradle build file should be made such that we can easily re-use on other projects but still runs checkstyle and still runs verifydesign, etc. etc.
Lastly, gradle should be checked in and the correct files ignored before checkin(like .gradle should not be checked in ;) ). At the end of this change, I should run ./build.sh and it works or ./build.sh and it prints out the gradle help.
We could not re-use playorm in another Guice bound application since it tries to rebind BaseEntityManagerFactory. That is the only one that should not be allowed to have @Inject annotation and instead should be injected manually in our BootstrapImpl class or something. We can test if it works with MonitorService in playorm which we tried to bind and instead had to manually inject stuff all over the place.
MetaProxyField needs to be more like MetaToMany when it comes to how data is stored..
ToMany is stored like so. If you have a Shirt which has a List users; then the storage is colname=users:userFk1 val=null and colname=users:Fk2 value=null, etc. etc for each user in the list.
For a ToOne, we store it like colname=users value=Fk1. We need to change it to store like the above ToMany relationship instead.
It would be useful to be able to have deletes in SJQL e.g.
delete from Budget as b where b.userId = :userId and b.status = :status
When dealing with objects that have no built in converter the following is shown:
Caused by: java.lang.IllegalArgumentException: .... You need to either add one of the @*ToOne annotations, @Embedded, @transient or add your own converter calling EntityMgrFactory.setup(Map<Class, Converter>) which will then work for all fields of that type OR add @column(customConverter=YourConverter.class) or finally if we missed a standard converter, we need to add it in file InspectorField.java in the constructor and it is trivial code(and we can copy the existing pattern)
This should be updated to mention the new @NoSqlConverter annotation.
And this part (EntityMgrFactory.setup) should be removed as you can't do this from the Play plugin.
Hi,
is there some kind of log, that a developer can use to trace which operations are performed against cassandra?
Just like Hibernate's "show_sql" switch.
Would be cool to see find/findAll operations,
row creates, updates, deletes,
how SJQL queries are executed,
more??
It would be nice to allow NoSqlEmbeddable classes that are just a simple container.
For example:
@NoSqlEntity
public class User {
@NoSqlId
private String id;
@NoSqlEmbedded
private Name name;
}
@NoSqlEmbeddable
public class Name {
private String firstName;
private String lastName;
}
A user (usually) just has one name. But "Name" could be easily reused in other entities.
So Name would not require an ID and all fields in Name could be persisted using column names like "name.firstname", "name.lastname" etc.
This implies that Name cannot be used in lists.
We have an old system which uses DataImportHandler to import data from PostgreSQL. The way we use it is:
We configure a xml with selects what will be indexed into solr from our database. Take a look at this: http://wiki.apache.org/solr/DataImportHandler#Full_Import_Example
We call URLs to perform the import. http://localhost:8983/solr/db/dataimport?command=full-import performs a full import and http://localhost:8983/solr/dataimport?command=delta-import imports only what wasn't imported since last time.
I am not saying we need exactly this solution, I just told you about DataImportHandler because we are used to it. However, your idea of Listeners sounds like something that would help realtime indexing, so that could be even better. Would just need some documentation or explanation on how to do it.
The most important thing is: we need to select what goes indexed by SOLR and how it goes. For instance, I might have two CFs: User and City
I might want to import just a document to SOLR, User document, and each user goes together with his/her associated cities. I might want to import the address column, but not import the birthDate column
I might want to import to documents to SOLR, User document and City document, each one independently.
Imagine a very big and complex data model.
I have the following entities:
User (has id, name, birthDate, List...)
UserRequests
City
UserLikes
UserInterests
etc.
All that goes to solr, but there is only one document: user.
A second scenario would be City as separate document in Solr too.
What if I receive a new user interest? I will add one more interest to the UserInterests CF, that would be the change in cassandra.
In Solr, however, I would need to reindex the entire user, as SOLR, AFAIK, doesn't allow you to reindex only part of the document, you can whether delete the indexed document or replace it, you can't update.
In the example bellow, savingEntity object would receive UserInterest entity, wouldn't it? But I want to reindex user.
Remember that realtime is good, but having the possibility to do it in batches is also desirable. Suppose the following scenario, indexing in real time:
I add a new interest to the user.
I add a new like to the user.
I add a new request to the user.
Indexing the entire user every event could be problematic in some cases, so in some cases it might be better to perform a delta index every hour, for instance... Final users still have near real time data, but the processing amount needed from the server decreases a lot.
I just changed my WLAN connection, started my app while my mac had no WLAN connected and got "RuntimeException: Call to InetAddress.getLocalHost().getHostname() == localhost. ...."
No idea yet how to solve this, but another thing I would like to see to be configurable in NoSqlEntityManagerFactory.
Just imagine that a customer has some data centers and some racks in each data center and a FQDN structure like this:
machine-n.rack-n.dc-n.lan
(off topic: @deanhiller now that I've seen UniqueKeyGenerator code I know what you meant with currentTimeMillis+1 ;-) )
There is a need to de-couple Astyanax runtime dependency from PlayOrm for those who want to evaluate PlayOrm with in-memory database only.
Command line tool is not supporting NoSqlEmbedded yet.
Have the following situation (note that ids are manually generated in the constructor and I have the proper annotation):
However if I do a NoSql.em().flush() after the create then the named query finds the cars.
I have checked the hashcode for the session and both the creates/saves are done in the same session.
For composite keys, command line tool is giving Long number =null instead of Entity.instance= null
One of the reasons I would like to use Cassandra+playorm is that the "database" itself is not bound to a fixed schema. Means: I can add and remove fields and relations to/from my entities as required.
Can you document what kind of schema updates are possible and recommended and what schema updates are not possible and will cause conflicts (e.g. changing field types from string to int - resp. binary to string representation).
Even the possibility to change *ToOne to *ToMany relations and vice versa is interesting. But changing a *ToMany relation to a *ToOne relation will cause problems (which problems exactly??) when a single object as multiple references in the *ToMany.
Last ticket for today ;-)
Would be nice to have the sources really in the sources jar.
Sources just blow up the "classes" jar and for example IntelliJ IDEA does not find any sources because the -sources.jar is "empty".
Comparing with Hibernate:
If you retrieve an object from Hibernate, change any fields and just flush() the Hibernate session without explicitly updating the changed object, the object's changed get persistent.
This is not what playorm does.
Is there any mechanism when objects get automatically updated?
Are changes to transitively reachable objects persisted? For example put() with objectA that has a reference to changed objectB, but objectA has not been changed.
To make PlayOrm more configurable it would be very useful to add configuration for Astyanax's array of connection pool configuration options:
So for example in application.conf we could have:
nosql.cassandra.initialConnectionsPerHost=2
nosql.cassandra.maxConnectionsPerHost=2
nosql.cassandra.writeConsistencyLevel=Quorum
nosql.cassandra.readConsistencyLevel=Quorum
nosql.cassandra.useLatencyAwareScoring=true
nosql.cassandra.latencyAwareUpdateInterval=10000
nosql.cassandra.latencyAwareResetInterval=10000
nosql.cassandra.latencyAwareBadnessThreshold=0.5
nosql.cassandra.latencyAwareWindowSize=100
See here:
https://github.com/Netflix/astyanax/wiki/Configuration
Hi, I am using cassandra with play and when I try to use the "update" action in AutoCreateEnum, i got "Not Implemented" exception. Are there any plan to implement update as well in near future or any approximate timeline when we can expect?
There is a need to support JOIN in UPDATE query..something like:
update product as p inner join p.productPrice as pp SET pp.price = 0.8 WHERE p.dateCreate < '2004-01-01'
Right now it is simple UPDATE like
update Account set (name="newName", isActive = true) where id = "acc1";
If you have a NoSQLQuery that references an unindexed attribute it throws the following warning:
Caused by: java.lang.IllegalArgumentException: You cannot have 'result.id' in your sql query since id has no @Index annotation on the field in the entity
at com.alvazan.orm.parser.antlr.ScannerSql.processColumnName(ScannerSql.java:541)
at com.alvazan.orm.parser.antlr.ScannerSql.processSide(ScannerSql.java:443)
It should throw the warning:
You cannot have 'result.id' in your sql query since id has no @NoSqlIndexed annotation on the field in the entity.
It would be easier to use the project if it was available at a maven repo
Simple details on how to do it from the maven side: http://apache.org/dev/publishing-maven-artifacts.html#signing-up
It seems that constants like
@NoSqlEmbeddable
public class Name {
private static final String MAIN = "main";
...
}
are handled as instance fields.
Playorm persisted a column called "MAIN" but fails on later operations with the "MAIN" field.
Also check if we can support LocalDateTime(I thought we could). Also in this ticket, add querying using some kind of date format as well in the JQL statements so we can make this much easier for people.
I keep needing this so I am finally adding the issue in here.....
I need a property byKeyOnly so when we have a class like this
private UserDbo {
@NoSqlId
private String id;
@NoSqlIndexed(byKeyOnly=true)
private String email;
}
where this will create the typical UserDbo table BUT instead of creating a normal index, it will create a table called emailToUserDbo such that I can do a JPA call of query="select u from TABLE as u where u.email = :email" BUT this query ONLY does = and does not < nor > or anything else.
Updating this is also interesting as if email changes you must delete the previous row in the xref table.
The emailToUserDbo table is very simple...one key=email and one column with name="value" and value="user row key"
It is VERY important that an index is NOT used as this pattern scales to trillions of UserDbo's where using an index does not
later,
Dean
Let's implement row keys on this one first as it is very useful as I need to select x from TABLE as x where x.id in ('key1', 'key2', 'key3');
thanks,
Dean
I am starting to forget table names so I need a list function of some sort so I can see them. We have 30,000 virtual tables so I would really like to filter out virtual tables as well sometimes.
thanks,
Dean
We are starting new projects using the latest version of the play framework, it makes sense for us to use the newest version of the framework as we are starting from scratch
We would very much like to use the PlayOrm with Play and would be interested in getting a 2.x plugin for the PlayOrm
We would be using Java, at least initially...
Thanks
There seems to be some breakage so thought would create an issue to track.
e.g. no such method for com.netflix.astyanax.Cluster.addKeyspace when starting.
Just wondering whether there was a plan to have a base class for model objects:
e.g. User extends NoSQLModel.
Or if you definitely prefer having a normal POJO. I personally don't have an opinion either way.
we need to add documentation on JPA deletes in the wiki(and updates as well?) and how to use them, and add documentation into the command line tool as well for anything we have added.
We need documentation on deleting a single column in command line tool (in the wiki and in command line tool maybe for this one).
unfortunately this requirement requires a little rework of the command line as to have the up arrow show the last command in the history, the command line will have to change to read and buffer every character typed in instead of what it does currently.
currently it only reads in an entire line so it will not read until you hit the enter key. This needs to change so we can "hear" the up arrows and such and display the last command and such on the command line (just like bash shell works with it's history).
(I keep hitting the up arrow and it is kind of annoying that it doesn't work).
@NoSqlEmbedded is working for Primitive data types only. Need to work on MetaEmbedded Entity to support other user defined entities
public class Goal {
@NoSqlOneToMany
public List cards;
}
Will throw a NullPointerException when saving.
com.alvazan.orm.impl.meta.data.MetaToManyField.translateToColumnImpl(MetaToManyField.java:164
Which is this line in my version of Playorm:
//now process all the existing columns (we can add same entity as many times as we like and it does not get duplicated)
for(PROXY proxy : toBeAdded)
Scenario:
I've removed "nosqlkeyspace".
Then I execute
Query q = entityManager.createNamedQuery(User.class, "findByUsername");
q.setParameter("username", "user-a");
Cursor<KeyValue> cursor = q.getResults(false);
while (cursor.next())
entityManager.remove(cursor.getCurrent().getValue());
and the following exception occurs:
Caused by: java.lang.NullPointerException
at com.alvazan.orm.layer9z.spi.db.cassandra.CursorReturnsEmptyRows.nextImpl(CursorReturnsEmptyRows.java:31)
at com.alvazan.orm.layer0.base.CursorRow.nextImpl(CursorRow.java:40)
at com.alvazan.orm.api.z8spi.iter.AbstractCursor.next(AbstractCursor.java:10)
Note that the "User" table does not exist.
I don't know the best way to do this but it would be nice if there was a method in NoSqlEntityManager:
public boolean contains(Class entityType, String key, String value)
That didn't load in any data.
I would like to be able to have a field of type List in my playOrm entity. Also, I would like this field to be magically persisted no my mapped column family.
Original stackoverflow issue:
http://stackoverflow.com/questions/12945656/is-it-possible-to-have-a-converter-for-a-list
Note:
It doesn't need to be a converter at all, Dean were suggesting to use the "@NoSqlEmbedded" annotation instead. If there is a way of storing a list instead of creating another CF just for that, it would be preferable. I would like to know the possibilities and limitations to indexes and partitions when I use this feature.
When calling EntityManager.put with an entity like this:
@NoSqlEntity
public class Breed {
@NoSqlId
private UUID id;
@NoSqlIndexed
private String officialName;
@NoSqlEmbedded
private List<LocalizedString> i18nNames;
}
where just "officialName" is not null, causes the following exception. I'd expect playorm to assume that the list is empty. I do not like to initialize lists in with "new ArrayList()", because these instances are usually nothing else than garbage.
java.lang.NullPointerException
at com.alvazan.orm.impl.meta.data.MetaEmbeddedEntity.translateToColumnImpl(MetaEmbeddedEntity.java:203)
at com.alvazan.orm.impl.meta.data.MetaEmbeddedEntity.translateToColumnList(MetaEmbeddedEntity.java:180)
at com.alvazan.orm.impl.meta.data.MetaEmbeddedEntity.translateToColumn(MetaEmbeddedEntity.java:154)
at com.alvazan.orm.impl.meta.data.MetaClassSingle.translateToRow(MetaClassSingle.java:87)
at com.alvazan.orm.layer0.base.BaseEntityManagerImpl.putImpl(BaseEntityManagerImpl.java:108)
at com.alvazan.orm.layer0.base.BaseEntityManagerImpl.put(BaseEntityManagerImpl.java:93)
persisting an object that has a NoSqlManyToOne relation to another entity with UUID as ID type fails with "InvalidRequestException(why:String didn't validate.)"
Original issue at Stackoverflow: http://stackoverflow.com/questions/12945338/is-there-any-converter-for-uuid-type-in-playorm
I would like to have a UUID field in my playORM mapped class. It can be
type 1 UUID as in that 3rd party library (http://johannburkard.de/blog/programming/java/Java-UUID-generators-compared.html) which is guaranteed to be unique, for instance. I would like playORM to have such a feature and a recommended way of using it, so as a user I don't have to worry much about it, I would like just to map my entity in playORM and start using, if it's possible.
Need to see, to what extent support can be provided for ALTER table statement.
i.e., Check if we can add/drop/modify a column!
ALTER TABLE table_name
DROP COLUMN column_name
ALTER TABLE table_name
ADD column_name datatype
ALTER TABLE table_name
ALTER COLUMN column_name datatype
ALTER TABLE table_name
MODIFY column_name datatype
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.