GithubHelp home page GithubHelp logo

jackrusher / spicerack Goto Github PK

View Code? Open in Web Editor NEW
80.0 6.0 5.0 76 KB

A Clojure wrapper for MapDB, which is a fast, disk-persistent data-structures library.

License: Eclipse Public License 1.0

Clojure 86.91% Shell 13.09%
clojure mapdb

spicerack's Introduction

The spicerack logo

Sometimes, while making tasty computer programs, one needs a place to store labeled containers of data. Spicerack is that kind of place.

This library is a Clojure wrapper for MapDB โ€” a fast, disk-persistent data-structures library. Like many Clojure wrappers around Java libraries, this one is incomplete. MapDB supports several data structures (Map, Set and List, implemented with trees and hashes), and can do fun things with them, like creating a hash table that acts as a cache with automatic eviction and change listeners. This wrapper doesn't support any of that. It just provides an idiomatic way to store something like a Clojure hash-map on disk.

I plan to add the other data types over time, as I need them for my own projects. In the meantime, these features are well-tested, deployed in production, and quite useful.

Usage

[spicerack "0.1.6"]

There are only a handful of functions in this wrapper. It provides open-database and close (though it's best to use clojure's with-open macro to handle closing), assoc!, dissoc!, and update! for mutation, on top of which clojure.core's get function can be used to access the value of a given key.

(require '[spicerack.core :refer [open-database open-hashmap assoc! update!]])

(with-open [db (open-database "./baking-db")]
  (let [ingredients (open-hashmap db "ingredient-hashmap")]
    (assoc! ingredients :apple-pie [:flour :butter :sugar :apples])
    ;;=> [:flour :butter :sugar :apples]
    (update! ingredients :apple-pie conj :cinnamon)))
    ;;=> [:flour :butter :sugar :apples :cinnamon]

In addition, Spicerack's hash-map implementation can be used like a normal Clojure hash-map with sequence functions such as map and reduce:

(with-open [db (open-database test-filename)]
  (let [hm (open-hashmap db "test-hashmap")]
    (doseq [[a b] (map vector (range 10) (range 0 1.0 0.1))]
      (assoc! hm a b))

    (get hm 1)
    ;;=> 0.1

    (get hm 47 :hi)
    ;;=> :hi

    (reduce (fn [acc [k v]] (+ acc v)) 0 hm)
    ;;=> 4.5

    (reduce (fn [acc [k v]] (+ acc k)) 0 hm)
    ;;=> 45

    (mapv (fn [[k v]]
            {:key k
             :val v})
          hm)
    ;;=>
    [{:key 0, :val 0}
     {:key 8, :val 0.7999999999999999}
     {:key 5, :val 0.5}
     {:key 3, :val 0.30000000000000004}
     {:key 2, :val 0.2}
     {:key 6, :val 0.6}
     {:key 7, :val 0.7}
     {:key 1, :val 0.1}
     {:key 9, :val 0.8999999999999999}
     {:key 4, :val 0.4}]))

Note that returning a lazy sequence from inside of a with-open block, then trying to realize that sequence outside of the block, will cause an exception to be thrown.

There are more examples in the test suite. In additon, there is automatically generated codox API documentation here.

License

Copyright ยฉ 2016-2019 Jack Rusher

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.

spicerack's People

Contributors

daveliepmann avatar jackrusher avatar plexus avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

spicerack's Issues

Error when deserializing

It seems like MapDB allows storing things that it is unable to retrieve, thus rendering the database inaccessible.

I'm not 100% sure what I did here, but it would appear I stored an anonymous function.

Would it be possible to detect cases like these so the error happens when storing the value, rather than later on when retrieving it?

org.mapdb.DBException$SerializationError
	at org.mapdb.StoreDirectAbstract.deserialize(StoreDirectAbstract.kt:234)
	at org.mapdb.StoreDirect.get(StoreDirect.kt:546)
	at org.mapdb.HTreeMap.valueUnwrap(HTreeMap.kt:1198)
	at org.mapdb.HTreeMap.getprotected(HTreeMap.kt:644)
	at org.mapdb.HTreeMap.containsKey(HTreeMap.kt:584)
	at clojure.lang.RT.getFrom(RT.java:786)
	at clojure.lang.RT.get(RT.java:778)
	at redolist.core$new_routes$fn__58298$fn__58299.invoke(core.clj:51)
	at compojure.core$make_route$fn__30790.invoke(core.clj:135)
	at compojure.core$wrap_route_middleware$fn__30783.invoke(core.clj:122)
	at compojure.core$wrap_route_info$fn__30787.invoke(core.clj:126)
	at compojure.core$if_route$fn__30735.invoke(core.clj:45)
	at compojure.core$if_method$fn__30725.invoke(core.clj:27)
	at compojure.core$routing$fn__30797.invoke(core.clj:151)
	at clojure.core$some.invokeStatic(core.clj:2674)
	at clojure.core$some.invoke(core.clj:2665)
	at compojure.core$routing.invokeStatic(core.clj:151)
	at compojure.core$routing.doInvoke(core.clj:148)
	at clojure.lang.RestFn.invoke(RestFn.java:460)
	at redolist.core$new_routes$fn__58298.invoke(core.clj:49)
	at compojure.core$if_context$fn__30861.invoke(core.clj:220)
	at compojure.core$routing$fn__30797.invoke(core.clj:151)
	at clojure.core$some.invokeStatic(core.clj:2674)
	at clojure.core$some.invoke(core.clj:2665)
	at compojure.core$routing.invokeStatic(core.clj:151)
	at compojure.core$routing.doInvoke(core.clj:148)
	at clojure.lang.RestFn.applyTo(RestFn.java:139)
	at clojure.core$apply.invokeStatic(core.clj:659)
	at clojure.core$apply.invoke(core.clj:652)
	at compojure.core$routes$fn__30801.invoke(core.clj:156)
	at compojure.core$routing$fn__30797.invoke(core.clj:151)
	at clojure.core$some.invokeStatic(core.clj:2674)
	at clojure.core$some.invoke(core.clj:2665)
	at compojure.core$routing.invokeStatic(core.clj:151)
	at compojure.core$routing.doInvoke(core.clj:148)
	at clojure.lang.RestFn.applyTo(RestFn.java:139)
	at clojure.core$apply.invokeStatic(core.clj:659)
	at clojure.core$apply.invoke(core.clj:652)
	at compojure.core$routes$fn__30801.invoke(core.clj:156)
	at ring.middleware.keyword_params$wrap_keyword_params$fn__32128.invoke(keyword_params.clj:35)
	at ring.middleware.params$wrap_params$fn__32404.invoke(params.clj:64)
	at ring.middleware.absolute_redirects$wrap_absolute_redirects$fn__32578.invoke(absolute_redirects.clj:38)
	at ring.middleware.content_type$wrap_content_type$fn__32531.invoke(content_type.clj:30)
	at ring.middleware.default_charset$wrap_default_charset$fn__32553.invoke(default_charset.clj:26)
	at ring.middleware.not_modified$wrap_not_modified$fn__32506.invoke(not_modified.clj:52)
	at ring.middleware.format_params$wrap_format_params$fn__34361.invoke(format_params.clj:119)
	at ring.middleware.format_params$wrap_format_params$fn__34361.invoke(format_params.clj:119)
	at ring.middleware.format_params$wrap_format_params$fn__34361.invoke(format_params.clj:119)
	at ring.middleware.format_params$wrap_format_params$fn__34361.invoke(format_params.clj:119)
	at ring.middleware.format_params$wrap_format_params$fn__34361.invoke(format_params.clj:119)
	at ring.middleware.format_params$wrap_format_params$fn__34361.invoke(format_params.clj:119)
	at ring.middleware.format_response$wrap_format_response$fn__34538.invoke(format_response.clj:194)
	at org.httpkit.server.HttpHandler.run(RingHandler.java:91)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.ClassNotFoundException: redolist.core$new_routes$fn__58298$fn__58306$fn__58308$fn__58309
	at org.mapdb.serializer.SerializerJava.deserialize(SerializerJava.java:30)
	at org.mapdb.StoreDirectAbstract.deserialize(StoreDirectAbstract.kt:229)
	... 57 more
Caused by: java.lang.ClassNotFoundException: redolist.core$new_routes$fn__58298$fn__58306$fn__58308$fn__58309
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:628)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
	at org.mapdb.serializer.SerializerJava.deserialize(SerializerJava.java:28)
	... 58 more

Storing custom type

Do you know any way to store objects created by deftype or defrecord? (deftype with java.io.Serializable interface).

installing locally library with spicerack deps

lein install creates following warning:

$ lein install
WARNING!!! version ranges found for:
[spicerack "0.1.6"] -> [org.mapdb/mapdb "3.0.7" :exclusions [org.eclipse.collections/eclipse-collections-api org.eclipse.collections/eclipse-collections org.eclipse.collections/eclipse-collections-forkjoin com.google.guava/guava]] -> [org.jetbrains.kotlin/kotlin-stdlib "[1.2.41,1.2.90)"]
Consider using [spicerack "0.1.6" :exclusions [org.jetbrains.kotlin/kotlin-stdlib]].

Created /home/xxx/clojure/clojure2d/utils/palettes/target/palettes-0.1.0-SNAPSHOT.jar
Wrote /home/xxx/clojure/clojure2d/utils/palettes/pom.xml
Installed jar and pom into local repo.

database as a resource in jar

The last concert before I go further. spicerack seems to work with the file. What if I want to include read-only database in the library bundled in the jar. Such files are later accessible via resource url. How to make it here?

read only access

Hi, first, thanks for making MapDB wrappings. I wasn't aware it exists.

Quick question: for read only access can I keep opened database around unless JVM is dead? Or should I everytime wrap into with-open? What is the proper strategy?

Opening and closing DB to access single value can be inefficient, however currently it's 6ms on my laptop which is perfectly ok.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.