Can't serialize vars in state about arcadia

[saikyun]

I am able to solve it by doing the following, but it's really hacky, so I don't really want to make a PR for it. Not sure how to deal with vars otherwise.

In state_help.clj:

(defn varify [x] (eval `(var ~x)))

(defn deserialize [^ArcadiaState as]
  (.BuildDatabaseAtom as true)
  (let [objdb (.objectDatabase as)]
    (binding [arcadia.data/*object-db* objdb]
      (try
        (let [^JumpMap jm (.state as)]
          (.Clear jm)
          (.AddAll jm
                   ;; in the future, switch on (.serializedFormat as)
                   ;; assuming edn for now
                   (edn/read-string
                    {:readers (assoc *data-readers*
                                     'arcadia.internal.state-help/varify
                                     varify)}
                    (str/replace
                     (.serializedData as)
                     "#'"
                     "#arcadia.internal.state-help/varify "))))
        (catch Exception e
          (Debug/Log "Exception encountered in arcadia.internal.state-help/deserialize:")
          (Debug/Log e)
          (Debug/Log "arcadia.data/*object-db*:")
          (Debug/Log arcadia.data/*object-db*)
          (throw (Exception. "wrapper" e)))))))

[timsgardner]

We don't currently have a good story for this. The problem is that it isn't completely clear how one should deserialize vars, in general, and no answer is baked into the language. For example, if a var is deserialized, should its namespace also be required? The best answer might well be something like what you have here, where readers are locally extended on deserialize, though it is difficult to do the same for serialization. That is, Clojure doesn't expose good robust thread-safe ways of locally modifying such serialization facilities as it has out of the box (eg, pr-str). The last time I looked at this problem vars seemed to be serializing not as #'some-var but as (var some-var), though that may have changed in the interim, and unfortunately it is much more difficult to gracefully intervene on print-method than on the readers. In the past we tried to walk the deserialized-but-unevaluated data structure in State with clojure.core/walk and manually required and eval'd all instances of (var some-var), but we pulled that because it's very hard to make fast, and it's very hard to do a walk like that for user-defined data structures loaded via tagged literals. We could expose some walk or recursive convert-vars protocol or multimethod or something for user data to extend, but didn't have the bandwidth to design that API and were unsettled by the prospect of introducing more complexity into our serialization system, especially to address this one very specific problem.

The current, rather unsatisfactory approach is to either:

• wrap vars in a user-defined data structure that knows how to serialize and deserialize itself via tagged literals
• use some other data abstraction that's easier to serialize and deserialize, such as a keyword that a multimethod dispatches on to replicate whatever logic would have consumed the var

That said, if vars have started serializing as "#'var" now, locally intervening on the reader might be a good solution. We have a prototype of a pseudo-hook allowing the user to wrap serialization and deserialization with dynamic var bindings, but we could also just introduce custom var deserialization logic ourselves.

[saikyun]

Thanks for the lengthy answer. I see that it's not a straight forward problem to solve.

How are the vars in hooks serialized? E.g. (hook+ go :start #'start) works fine, so that's why I thought it'd work with state as well. :)

Thanks for the alternatives. I'll close this issue, since there doesn't really seem to be a good way to solve this cleanly. I do think however it might be worth suggesting alternatives in documentation/tutorials. :)