yakworks / gorm-tools Goto Github PK

With latest grails, Grails makes it possible to auto create a transactional proxy, or use Transactional annotations, and if Transactional annotation is present it will not create transactional proxy

We should do the same with dao and borrow the code from services plugin.

This way, when dao is used within services etc, we wont be mixing two approaches (Transactional annotated services and Transactional proxy daos) And we get more flexibility, if for some dao we need very specific transaction behavior we can do that with annotation.

@basejump Any commentes ?

is the null on the assocatiation expected behavior? grails 3.3 treats things differently. See various FIXMEs for this

add tests for DbDialectService and make it work with h2.

• doesn't seem to have move over completely from rally

see our gorm-rest-api or https://github.com/grails/gorm-graphql to get an idea.

move tests and examples into an examples dir.

Hot reloading doesnt work for DAO artefacts, it compiles, but thn fails to reload

in fact we should just retunr the domain and not the map for that method too

[3:50]
add an issue in dao

get rid of DaoMessage.created(entity) - it slows things down

@basejump

Beef them up a bit here. Added TODO to explain it a bit more

As IdGenerators are moved to gorm-tools, it will need NEWOBJECTID table to function.
Document the requirement for the table and how to create it.

finish #81

• on update it needs to figure out what belongs to it and update fields if the map contains them. if its doesn't belong then it should be ignored and only look at the id to see if it needs to be changed.

• on a create it needs to create new associations if they belong and only load if they don't. the bindMethod can be used to know if we should be doing something special in each case.

• a good example to create in tests is to simulate how we might create and update customers or arTrans in ar. In fact I think all of our tests should eb centered around using a customer/invoice type of simulation.

Lets call it MapBinder and rename FastBinder to GormMapBinder
The default bean will be setup as `gormMapBinder(GormMapBinder)'
in the GormDao trait we can have

@Autowired MapBinder mapBinder

Then if we have an explicitley definded dao we could abstract out special binding into its own bean and inject/override the mapBinder.

FooDao implements GormDao {
  @Autowired FooMapBinder mapBinder
....

add something like this explanation (but better 😀) above to the docs to explain how it can be done.

inital work is started here https://github.com/yakworks/gorm-tools/tree/81-mapBinder

Some of this is done already. See CriteriaUtils & GormHibernateCriteriaBuilder in Rally
assuming we have something like

{
  "ponum":"abc", /* if its a single value eq is default, if it contains % then it uses ilike */
  "reconciled":true, /* boolean */
  "tranDate":"2012-04-23T00:00:00.000Z", /* date */
  "customer.id":101, 
  "customerId":101, /* or like this should work */
  "customer":{"id":101}, /* or object way */
  "or":{
    "customer.name":["ilike()","wal%"],
    "customer.num":["ilike()","wal%"]
  },
  "customer":{"id":101}, /* shortcut above for  */
  "docType":["PA","CM"], /* an array means it will use in/inList */  
  "docType":"PA,CM", /* this should work too */
  "docType":["in()",["PA","CM"]], /* the above ins would be a short cut for this*/
  "tranType.id":["not in()",[1,2,3]],/* will translate to "not{ in("tranType.id",[1,2,3])]" */
  "refnum":["ilike()","123%"], /* pass object if its a different criteria */
  "amount":["between()",0,100], /* between value */
  "oldAmount":["gt()","origAmount"], /* greater than value */
  "amount":["ne()",50], /*not equal*/
  "status.id":[1,2,3], /* an array means it will use in/inList */
  "status":[{"id":1},{"id":2},{"id":3}], /* an array means it will use in/inList */
  "or":{
    "closed[0]":["isNull()"], /* translates to isNull*/
    "closed[1]":0
  },
  "order":[{"tranDate":"ASC"},{"customer.name","desc"}]
}

• test the "event" methods
• test that a new binder can be injected into a dao to replace the default fastbinder. I think we should be programming to an interface or trait in order to do this
• [ ]

restructure project as gradle multimodule project similar to view-tools

• CriteriaUtils
• ScrollableQuery
• GrailsParameterMapRowMapper
• MultiFormatDateConverter
• date converters (e.g. LocalDateJsonConverter)
• more tests for BeanPathTools
• more tests for DateUtil
• GormHibernateCriteriaBuilder
• GormUtils

joshua
[10:25 AM]
I don’t see any tests for MultiFormatDateConverter either
[10:26]
no tests for any of the LocalDateJsonConverter?
nickpavlov [10:27 AM]
yes, there are no tests for converters
joshua
[10:27 AM]
CriteriaUtilsSpec can get copied over from rally
[10:28]
GrailsParameterMapRowMapper
[10:29]
ScrollableQuery
[10:29]
don’t have tests either
joshua [10:34 AM]
basically figure out if we have good test coverage
[10:35]
and don’t test what we don’t need, if it looks like its not used then see if it is and remove or deprecate. will need to search across our grails3 apps/plugins

remove generic DomainException. All Exceptions should inherit from DataAccessException to stay within the Spring model. In Gorm see ValidationException extends DataIntegrityViolationException.

• we should possibly raise an issue and maybe issue a pull request so that Gorm keeps it instead of using grails.validation.ValidationException

• we still want to intercept and refire for ValidationExceptions

extend AbstractPersistenceEvent.

• Post and PreDaoPersistEvent
• Post and PreDaoRemoveEvent
• Post and PreDaoCreateEvent - will have the params accessible from bind.
• Post and PreDaoUpdateEvent - will have the params accessible from bind.

These can be fired like they are in Gorm's EntityPersister class with publisher.publishEvent(event)

• add javadocs
• document loadTotalFromDb where is it used, how to use it and whats it needed for if the paging list in grails has a count?
  no test anywhere either that I can see for that either.

lots off common copy/paste code that should be refactored to common methods. DaoPluginHelper has many of the same needs too.
"${GrailsNameUtils.getPropertyName(domain.name)}Dao" is one big example.
I don't see why DaoHibernateSpec should be so different than DaoDataTest that it shares no common code. They should opperate more or less that same.

see https://stackoverflow.com/questions/11121600/writing-proper-javadoc-with-see
and https://stackoverflow.com/questions/5011291/usage-of-see-in-javadoc

┆Issue is synchronized with this Asana task

incorporate and start using this helpful plugin to make sure it works with dao. extend where needed and help get it working with grails3.3
see https://github.com/longwa/build-test-data

• is the @value working on the trait for criteriaKeyName? show a test where this is set in config and works
• refactor to a MangoQuery bean with a MangoQueryApi trait so query, countTotals, buildCriteria logic are in that bean. there is to much logic in the GormDao trait.
• a custom dao should be able to @autowire its custom MangoQuery bean to use so we can implement and inject a variation if desired.
• criteriaKeyName can then be moved into the MangoQuery

This can used in beefing up the angle-grinder demo to use a more standard rest interface architecture

With grails 3. We should use traits to enhace domain classes with dao methods.
See http://docs.grails.org/latest/guide/traits.html

This would be compile time enhancement, giving us ability to use persist() with CompileStatic

@basejump

Grails 3.3 has removed support for spring proxies based transactions, and supports just AST. Dao should remove the support for proxies too.

So
When registering beans, we dont check if we should create proxies or simple spring beans. should always be regular spring beans, as transaction stuff would have been weaved using AST

Look at services plugin for example.

@basejump

We don't have the second level cache enabled and need to benchmark its impact.
should be able to see what it does enable and also show how to disable it for a batch insert. For examples, it may be beneficial to keep it on for Regions and Counties as we load the cities.

┆Issue is synchronized with this Asana task

We want to be able to have external groovy scripts to run custom business logic during the dao events. For example, at one customer we do special customer.num generation before a new customer object is saved.

this plugin should detect an external scripts executor plugin.
we should have a contract on how the scripts are called. Either we have the script return an object implementing a trait or interface and then it can get registerd, or we have the script run and register a listener using the grails 3.3 events.

see the end of this as an example. https://github.com/Netflix/asgard/blob/master/grails-app/conf/spring/resources.groovy
we could setup reloadable groovy beans.

see https://docs.spring.io/spring/docs/current/spring-framework-reference/languages.html#groovy and http://mrhaki.blogspot.com/2013/03/grails-goodness-use-constructor.html

https://github.com/yakworks/gorm-tools/blob/master/plugin/src/main/groovy/gorm/tools/beans/MultiFormatDateConverter.groovy

• fix broken tests in BeanPathToolsSpec
• MapFlattener
• GormDaoSupport
• (BeanPathTools) add tests for flattenMap, buildMapFromPaths
• (BeanPathTools) check if getFieldValue is used and remove/mark as deprecated
• (BeanPathTools) refactor error handling for case when a property is not exist (2x)
• refactor BeanPathTools.propsToMap (don't require a domain, etc)
• BeanPathTools

joshua [10:23 AM]
DateUtil need better javadocs and tests. need to also check if some of those are even used or needed.
[10:23]
search the code for XXX and FIXME’s
joshua [10:34 AM]
basically figure out if we have good test coverage
[10:35]
and don’t test what we don’t need, if it looks like its not used then see if it is and remove or deprecate. will need to search across our grails3 apps/plugins

• initial docs to spec it out
• Normalizer to convert the allowed short cuts to a map of maps and lists. ['a.b.c':'foo'] -> [a:[b:[c:['$eq':'foo']]]] for example
• MangoCriteria to extend and use DetachedCriteria. Should have a 'build' method that accepts a map instead of a closure. iterate of the keySet and build.

Currently GormDaoSupport uses def for every thing, eg save, update, remove
If Any dao which extend GormDaoSupport, and want to override any of the methods, or want to define event hooks such as beforeSave or beforeInsertSave, the method arguments has to define using def, and can not use the Real Domain class type eg beforeSave(Book b) wont work.
Because of this, we can not use @CompileStatic, if we want to.

If GormDaoSupport used generic - GormDaoSupport<T> - we can specify static types everywhere

Also, log field in GormDaoSupport should have a static type, and not def

https://github.com/yakworks/gorm-tools/blob/master/test-projects/app/src/integration-test/groovy/gorm/tools/CriteriaUtilsSpec.groovy

DaoMessage.notFound fails for DaoMessage.notFound("xxx.MockDomain",[id: null])
And throws error - Can not get Id on null object.

This is because, at line no 19 (notFound(domainClassName, params.id)) when params.id is null, groovy fails to identify which overloaded method to call, and calls the same method with map params argument, so it fails on line 19.

creating a gorm.tools.FastDataBinder in gorm-tools. Should be a bean that we can inject into the daos and use. We can move the bindFast out of a static in GormUtils, that static can still be there but it should redirect to FastDataBinder bean. See BenchmarkDatabindingService.setPropsFastIterate for more on what should be done to keep it fast but fall down to the converters if need be. you guys can figure out who takes it.
I leaned on the grails source for SimpleDataBinder to figure out how they are inject and getting the registered ValueConverters. They are slow as fuck which is why I do the 2 checks for the most common occurences, which would be numbers and dates. there is room to improve the DateUtil.parseJsonDate if we make those SimpleDate formats at the class level instead of creating them on every call.

The performance should match or be better than 4.5s setPropsFastIterate and within 1 second of the useStaticSettersInDomain- static setters benchmark. to keep it working I munged together but I think the static lookups on bean is slowing it way down to 13.1s.

• setters or property copy on associations with 20 fields
  3.517s useStaticSettersInDomain CityFat | 111690 rows
  4.549s setPropsFastIterate CityFat | 111690 rows
  13.118s daoCreateNewFast CityFat | 111690 rows
  8.387s useDynamicSettersFat CityFat | 111690 rows

DaoDataTest can extend or implement from DataTest trait. so when we do this

    void setupSpec() {
        mockDomains TestClazzA, TestClazzB, TestClazzC
    }

it also wires up the daos. should refactor how we do our dao setups so we can more easily reuse the same code to setup the unit test.

https://testing.grails.org/latest/guide/index.html#unitTesting
can also look into getIncludePlugins

should also create a DaoHibernateSpec that extends HibernateSpec and does the same stuff needed to wire up the daos when we use the full HibernateSpec which works differently than using mockDomains with DataTest.

leaning on the build-test-data and either adding the ability in with PR, extending or as a last resort forking it.
The example: constraint, which is used when generating docs for the domain both here and in the https://yakworks.github.io/gorm-rest-api , should also be leaned on when automatically building test data. basically if it exists, use it, otherwise fall back to what the build-test-data generates.

whatever we come up with should also be able to be used when generating the rest openApi/swagger docs so that if an example constaint is there then use it, otherwise produce docs with the sample that build-test-data uses.

Index.md needs to be beefed up for install and quick start. There are a million plugins out there to use as example on what we should have.

dao.md should be beefed up with how to setup a new dao, what happens if we don't set one up, links to groovy docs, etc.. We want this to be really solid. If example is needed then look at the gorm docs for what we are after.

use the @Supress where needed.
See https://github.com/grails/gorm-graphql for how to do it

• Make grails3 master and move master to grails 2.5.x
• update build to more mordern way to deploy
• currently build will fail on my machine unless I comment out the following. fails artifactoryUsername. build is failing too

 // publishing {
//     repositories {
//         maven {
//             url "http://repo.9ci.com/artifactory/grails-plugins"
//             credentials {
//                 username "$artifactoryUsername"
//                 password  "$artifactoryPassword"
//             }
//         }
//     }
// }

• want to see the impact of certain exception handling, such as validation.

https://stackoverflow.com/questions/36343209/which-part-of-throwing-an-exception-is-expensive

https://stackoverflow.com/questions/22170984/how-do-i-make-grails-get-to-throw-exception-on-record-not-found

https://github.com/grails-plugins/grails-spring-security-core/blob/master/src/main/groovy/grails/plugin/springsecurity/userdetails/NoStackUsernameNotFoundException.groovy

http://normanmaurer.me/blog/2013/11/09/The-hidden-performance-costs-of-instantiating-Throwables/

http://java-performance.info/throwing-an-exception-in-java-is-very-slow/

• first update all Transactional annotations to be from gorm vs grails
• add bindFast option as benchmarks show grails binding to be super slow
• refactor to use interfaces and traits. grails-datastore-gorm as an example. GormEntityApi, and GormEntity provide examples.

why did we name it this? what was wrong with insert?
can we call it create? don't like the name.

• mkdocs for BeanPathTools
• mkdocs for GormUtils
• mkdocs for GormMetaUtils
• mkdocs for ScrollableQuery
• mkdocs for DateUtil

• javadocs for DateUtil
• improve javadocs for BeanPathTools
• javadocs for GormUtils
• javadocs for GormDaoSupport
• improve javadocs for GormMetaUtils
• improve javadocs for GrailsParameterMapRowMapper
• improve javadocs for ScrollableQuery

[10:36]
start beefing up the docs. mkdocs serve to see them

[10:36]
make sure you can run them

[10:36]
javadoc the shit out of the classes

joshua [10:39 AM]
uploaded and commented on this image: Pasted image at 2017-10-12, 2:39 AM

I want to be able to hit ctrl-j in idea and get meaningfull help like this

joshua [10:40 AM]
and we want to gen the api docs that built in and get something that is useful like this https://grails.github.io/gorm-graphql/latest/api/index.html