juliadynamics / drwatson.jl

The perfect sidekick to your scientific inquiries

JuliaDynamics

This repository serves the following purposes:

• Contains the source code for the JuliaDynamics website in the src and build folders.
• Hosts the website via GitHub-pages and Jekyll.
• Contains tutorials for all packages of JuliaDynamics in the tutorials folder.
• Contains video resources for all packages of JuliaDynamics in the videos folder.

---

The website was modeled after the website of QuantumOptics.jl and most code that builds the site was re-used from the repository of QuantumOptics.jl (with permission).

---

To build locally do follow the instructions from here: https://jekyllrb.com/docs/

(install Jekyll and then do bundle exec jekyll serve which serves by default to http://localhost:4000)

drwatson.jl's Issues

usage of quickactivate is tricky

Hey,
I noticed that the usage of quickactivate is tricky.

It can only work if the DrWatson is installed in both the local and global environment.
(The same goes for all other packages.
In the scripts you gave me you load all packages before calling quickactivate.)
This is misleading. The packages will be loaded from the global environment
if installed. This may give you a wrong version of the same package.

Real World Examples

I believe we have a very concrete base to invite people to beta test and suggest more features. But before that we should be sure that we have useful examples in the documentation.

Thankfully, our scientific projects already use DrWatson and thus can be immediatelly made into real world examples.

I'll list here what I Think is useful:

• Basic example of using DrWatson, activating directory and saving something with savename and tag!. I will take this example from the billiards project.
• Basic example of using Parameters.jl and customizing savename. I will do that while creating tests for Parameters.jl
• Some convenient usage of produce_or_load. I can use it for lyapunov exponents of billiards. Connect this example with making a figure for a paper.
• Complex real world usage of preparing dictionaries for runs. We can use the embedding project for that.
• Real world result of reading data that were produced from the above example, using collect_results.

I think this is enough for now!

Test savename with composite types

the function savename should also be tested to work with composite types. We could use Parameters.jl as well to make the testing smoother and more general.

Include a diff with the saved model run?

When experimenting, I often edit the source-code, to check what changing a parameter does.
I don't want to check-in commit this code as I'm just trying things out.
(Sure, I should refactor and make that a settable parameters, but life is too short.)

Also, sometimes I just forget to checkin comit before starting a run. So, how about storing a diff/patch with the model results? Just the output of git diff should be fine.

Reproduction would then be:

git checkout saved-sha
git apply stored-diff
julia -e ...

Edit: not sure where the "check-in" came from. Probably a flash back to svn, probably because I recently cloned a svn repository...

Default project should include test directory

Encourage scientists to write unit test for src!

Running `build` in Pkg mode fails

When running build in Pkg mode the following error appears:

ERROR: The following package names could not be resolved:
 * PackageName (not found in project or manifest)
Please specify by known `name=uuid`.

This is presumably because the Project.toml contains name = "PackageName" which was given when the project was initialized, but nothing but that name.
A workaround is to run build pkg1 pkg2 ... instead, but it appears the error also prevents automatic building after installing a new package.

Running on Julia 1.1.0 on Windows 7 with DrWatson 0.5.3.

Make `tag!` also add the name of the producing script?

Should we enhance tag!, so that besides a commit ID it also adds a field like script_name which is the name of the script that produces the result or that calls the function?

The script name must be a full local path with respect to the project directory, i.e. scripts/lalala/file.jl.

I don't know how to do this.

---

If we do this it would be good to add script_name and commit as default black lists in collect_results!.

Add OS-specific files to .gitignore

I think that the default .gitignore file DrWatson generates should include the following OS-specific patterns:

.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db

Also, ignore .ipynb_checkpoints for Jupyter Notebooks.
See
https://stackoverflow.com/questions/107701/how-can-i-remove-ds-store-files-from-a-git-repository
https://stackoverflow.com/questions/36306017/should-ipynb-checkpoints-be-stored-in-git

Tests failing "randomly" on mac CI

Test on mac CI seem to fail randomly with:

Project Setup: Test Failed at /Users/travis/build/JuliaDynamics/DrWatson.jl/test/project_tests.jl:45
  Expression: projectname() == name
   Evaluated: "test project" == "lala"

Julia version doesn't matter. Also, I'm on macOS and can't reproduce.

Broken collect_results! when white_list is empty but special_list is not

When running

using DrWatson, BSON, DataFrames

data = Dict("foo" => :bar)
save(joinpath("tmp","a.bson"), data)
save(joinpath("tmp","b.bson"), data)

collect_results!("tmp", white_list=[], special_list=[:foo2 => data -> :bar2])

I get an empty data frame:

julia> include("mwe.jl")
[ Info: Starting a new result collection...
[ Info: Scanning folder tmp for result files.
[ Info: Added 2 entries.
┌ Warning: `T` is deprecated, use `nonmissingtype` instead.
│   caller = compacttype(::Type, ::Int64) at show.jl:39
└ @ DataFrames ~/.julia/packages/DataFrames/Iyo5L/src/abstractdataframe/show.jl:39
0×2 DataFrame

If, instead, I provide some non-empty white_list everything is fine:

julia> collect_results!("tmp", white_list=["foo"], special_list=[:foo2 => data -> :bar2])
[ Info: Starting a new result collection...
[ Info: Scanning folder tmp for result files.
[ Info: Added 2 entries.
2×3 DataFrame
│ Row │ foo     │ foo2    │ path       │
│     │ Symbol⍰ │ Symbol⍰ │ String⍰    │
├─────┼─────────┼─────────┼────────────┤
│ 1   │ bar     │ bar2    │ tmp/a.bson │
│ 2   │ bar     │ bar2    │ tmp/b.bson │

Assertion error when instantiating project

Code that gives me this error is just navigating to my project folder then:
pkg> activate .
pkg> instantiate

ERROR: AssertionError: haskey(hashes, uuid)
Stacktrace:
[1] version_data!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Pkg\src\Operations.jl:443
[2] #instantiate#67(::Nothing, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Pkg\src\API.jl:466
[3] instantiate at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Pkg\src\API.jl:442 [inlined]
[4] do_instantiate!(::Dict{Symbol,Any}, ::Array{String,1}, ::Dict{Symbol,Any}) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Pkg\src\REPLMode.jl:634
[5] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Any, ::Any, ::Vararg{Any,N} where N) at .\essentials.jl:742
[6] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at .\essentials.jl:741
[7] do_cmd!(::Pkg.REPLMode.PkgCommand, ::REPL.LineEditREPL) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Pkg\src\REPLMode.jl:563
[8] #do_cmd#31(::Bool, ::Function, ::REPL.LineEditREPL, ::String) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Pkg\src\REPLMode.jl:538
[9] do_cmd at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Pkg\src\REPLMode.jl:534 [inlined]
[10] (::getfield(Pkg.REPLMode, Symbol("##53#56")){REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\Pkg\src\REPLMode.jl:988
[11] #invokelatest#1 at .\essentials.jl:742 [inlined]
[12] invokelatest at .\essentials.jl:741 [inlined]
[13] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\REPL\src\LineEdit.jl:2273
[14] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\REPL\src\REPL.jl:1035
[15] run_repl(::REPL.AbstractREPL, ::Any) at C:\Users\julia\AppData\Local\Julia-1.1.1\share\julia\stdlib\v1.1\REPL\src\REPL.jl:192
[16] (::getfield(Base, Symbol("##734#736")){Bool,Bool,Bool,Bool})(::Module) at .\client.jl:362
[17] #invokelatest#1 at .\essentials.jl:742 [inlined]
[18] invokelatest at .\essentials.jl:741 [inlined]
[19] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at .\client.jl:346
[20] exec_options(::Base.JLOptions) at .\client.jl:284
[21] _start() at .\client.jl:436

Data versioning

Plain git repos are great for code, but really not so great for large amounts of data. For versioning data, the standard approach is to version only the hashes and store the actual data elsewhere. There's various tools to do this, most notably git-lfs and git-annex.

I wondered whether you have given some thought to this, and wanted to point you in the direction of two very interesting projects which might serve as additional inspiration:

• gin - a radically simple versioning tool for research data with web interface and local tooling https://web.gin.g-node.org/
• Datalad - a power user's tool for versioning datasets https://www.datalad.org/

Both of these are built on top of git and git-annex, but hide some of the complexity of using those low level tools directly. Personally I think gin might be interesting to you as they are really trying to simplify the problem of curating scientific datasets, and your mission for DrWatson seems to be to keep it as simple as possible.

savename(datadir(), c) does not work as expected

After the latest changes with #64 and #66 , the following thing happens:

using DrWatson
c = (T = 100, dt = 0.1, N = 25, mode = "bi")
savename(datadir(), c, "dat")

"C:\\Users\\datseris\\.julia\\environments\\v1.1\\data_N=25_T=100_dt=0.1_mode=bi.dat"

What you will see is that the data directory is not properly chained with the savename(c). This is easily fixed by doing instead datadir(savename(c)).

The problem however is that produce_or_load does not do this. Instead as seen in its docs, we have:

produce_or_load([prefix="",] c, f; kwargs...) -> file, path

Let s = savename(prefix, c, suffix)...

---

To resolve this I propose to change produce_or_load so that it defines s as joinpath(prefix, savename(c, suffix). This also ties nicely with the default prefixes that savename supports.

(this is breaking)

Give warning on `projectdir` if project is standard Julia project

if there is a way to consistently tell if the current project is the "julia default" (whatever that means) then we should warn when the function projectdir() is called and showing the default dir.

update to new API of DataFrames.jl

DataFrames.jl has released a new version.
Functions like collect_results throw lots of deprecation warnings now.

We should update this before errors start showing up.

[FR] "force" argument for tag! and derivatives

At the moment tag! (and thus all derivatives up to @tagsave) do not do anything if the given dictionary already has fields like gitcommit etc. This is done for safety purposes.

However, I now see that an argument force would be helpful, so that a dictionary that is already tagged can be "retagged". This is useful in the following scenario: when there is a processing pipeline that adds data to an existing dictioanry, which is saved on disk. The dictionary is updated as new experimental data are accessible. I am now in exactly this scenario and I can see how this is useful.

As this can only be a non-breaking change, since we are now in 1.0, I'd suggest to do this after #84 , and simply make force a keyword argument.

Error tagging new release

The REQUIRE file could not be found.
cc: @Datseris

compat in Project.toml breaks on rc versions

PR #94 includes a line with the Julia version for compat info. However, if the Julia version is anything other that x.y.z (for example, the rc version 1.3.0-rc2.0 I'm currently using) this breaks because Pkg can't/won't parse that string.

I'm not sure if this should be fixed in Pkg (I'm assuming it won't because I don't think rc versions are included into SemVer?) or in DrWatson. I can try to provide a PR if the decision is made for the latter.

tagsave and document it

Add function tagsave which is save and tag!. Document those, and also document produce_or_load.

Type of the output of `dict_list`

Thank you for your work on this tool!

I'd like to report a small inconsistency with the output of dict_list. Whenever possible, I'd like the result to be a list of Dict{Symbol,Any} or Dict{String,Any} instead of a list of Dict{Any,Any}.
Here is an example in which it is not the case:

julia> test_matrix = Dict(:a => [1, 2], :b => ["x", "y"])
Dict{Symbol,Array{T,1} where T} with 2 entries:
  :a => [1, 2]
  :b => ["x", "y"]

julia> dict_list(test_matrix)
4-element Array{Dict{Any,Any},1}:
 Dict(:a=>1,:b=>"x")
 Dict(:a=>2,:b=>"x")
 Dict(:a=>1,:b=>"y")
 Dict(:a=>2,:b=>"y")

It works fine in many other cases such as the example in the doc or this one:

julia> test_matrix = Dict(:a => [1, 2], :b => [8, 9])
Dict{Symbol,Array{Int64,1}} with 2 entries:
  :a => [1, 2]
  :b => [8, 9]

julia> dict_list(test_matrix)
4-element Array{Dict{Symbol,Any},1}:
 Dict(:a=>1,:b=>8)
 Dict(:a=>2,:b=>8)
 Dict(:a=>1,:b=>9)
 Dict(:a=>2,:b=>9)

I tried some variants but I couldn't figure out exactly in which cases it returns Dict{Any,Any} or Dict{Symbol,Any}.

savename digits respect "1e-6" syntax

If I specify an argument like :lr=>1e-5 I would like savename to not attempt to round this. At the moment this is beyond the default digits=4 and gets saved as lr=0.

Is it easy to distinguish floats written as 1e-2 from 0.01? If so, can the former be preserved in the savename?

`initialize_project` breaks on 1.2.0

After updating to Julia 1.2.0 I have the following problem:

On Julia 1.1.0:

 julia> initialize_project(raw"S:\path_to_project", "FederOpt")                                                                         
   Updating registry at `C:\Users\jk\.julia\registries\General`                   
   Updating git-repo `https://github.com/JuliaRegistries/General.git`                    
  Resolving package versions...                                                          
   Updating `S:\path_to_project\Project.toml`   
   [634d3b9d] + DrWatson v0.7.0                                                          
   Updating `S:\path_to_project\Manifest.toml`  
   [634d3b9d] + DrWatson v0.7.0                                                          
   [...]
   [4ec0a83e] + Unicode                                                                  
 "S:\\path_to_project" 

On Julia 1.2.0:

julia> initialize_project(raw"S:\path_to_project", "FederOpt")                                                                         
Activating new environment at `S:\path_to_project\Project.toml`                                                                        
 Resolving package versions...                                                          
  Updating `UNC\$(intranet_ip)\$(folder that's mounted as S:\)\path_to_project\Project.toml`                                                               
  [634d3b9d] + DrWatson v0.7.0                                                          
  Updating `S:\path_to_project\UNC\$(intranet_ip)\$(folder that's mounted as S:\)\path_to_project\Manifest.toml`  
  [634d3b9d] + DrWatson v0.7.0                                                          
  [...]
  [4ec0a83e] + Unicode                                                                  
ERROR: SystemError: opening file "S:\\path_to_project\\Project.toml": No such file or directory                                   
Stacktrace:                                                                             
 [1] #systemerror#44(::Nothing, ::typeof(systemerror), ::String, ::Bool) at .\error.jl:134                                                                                      
 [2] systemerror at .\error.jl:134 [inlined]                                            
 [3] #open#311(::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::typeof(open), ::String) at .\iostream.jl:289                                                            
 [4] open at .\iostream.jl:281 [inlined]                                                
 [5] #open#312(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(open), ::getfield(Base, Symbol("##276#277")){String}, ::String) at .\iostream.jl
:373                                                                                    
 [6] open at .\iostream.jl:373 [inlined]                                                
 [7] read at .\io.jl:310 [inlined]                                                      
 [8] #initialize_project#1(::Bool, ::Bool, ::Nothing, ::Bool, ::typeof(initialize_project), ::String, ::String) at C:\Users\jk\.julia\packages\DrWatson\T1n17\src\project
_setup.jl:198                                                                           
 [9] initialize_project(::String, ::String) at C:\Users\jk\.julia\packages\DrWatson\T1n17\src\project_setup.jl:158                                                       
 [10] top-level scope at REPL[9]:1                                                      

Somehow Julia/Pkg/DrWatson/? now chokes on the network drive and messes up the path after which it can't find the file it's just created.

Running project tests

I'd like to discuss how to nicely run tests defined for a DrWatson project.

• So by default we have the test directory containing a runtests.jl. This matches the default for julia packages / projects.
• Pkg.test can be invoked without additional arguments, which results in running the tests for the currently active project.

So in a freshly initialized DrWatson project

julia> using DrWatson

julia> quickactivate(@__DIR__)

(watson test) pkg> test
ERROR: The following package names could not be resolved:
 * watson test (not found in project or manifest)
Please specify by known `name=uuid`.

After adding a UUID to project.toml

(watson test) pkg> test
   Testing watson test
ERROR: expected the file `src/watson test.jl` to exist for package watson test at /Users/sebastianpech/test/julia/watson test

After adding the file src/watson test.jl:

(watson test) pkg> test
   Testing watson test
 Resolving package versions...
   Testing watson test tests passed

The tests run successfully.

What do you think about that approach? Alternatively we could add a function like quickruntests(@__DIR__) or something similar which allows us the keep the current project structure.

Keyword arguments for tag! and tagsave

The arguments of tag! and tagsave have become quite extensive. Maybe it is time to define keyword-based versions of these functions. This is very easy and also non-breaking.

Here is where is tag! defined: https://github.com/JuliaDynamics/DrWatson.jl/blob/master/src/saving_tools.jl#L125

I don't really know how keywords work with macros, so I don't know how @tagsave can be affected.

Use FileIO instead of BSON

We can't use it in EmbeddingResearch since all result files are .jld2 and I really don't want to change this in the middle of a project.

Yeah don't do that use either the local version on the embedding repo or do a branch in this branch 2 commits in earlier.

But if we get the BSON FileIO thingy, then you won't have to worry about it and in the end be able to use everything!

---

Alright, then I'll put it behind a require for now and we see in the future if BSON has fixed the problem.

Originally posted by @Datseris in #19 (comment)

See JuliaIO/FileIO.jl#224 and JuliaIO/BSON.jl#31

These two PRs implement the interface for FileIO to work with BSON.

Recollection option for collect_results

After a bit of usage of collect_results it seems like an option to "recollect" is useful.

By that I mean that by giving a keyword argument e.g. recollect = true all existing data should be re-collected instead of being skipped.

I believe that this functionality should not delete the saved dataframe and just start over. The reason for that is some old results still saved in the dataframe may not exist in the data folders any more. Then it would be irreversible to remove them. The best approach would be to just "replace" existing files if they still exist.

@JonasIsensee do you have any idea how this can be done? If some please give some help here. (Or if you can even implement this then you're the boss!)

Make the keyword arguments of `savename` dispatch on the type of `c`

This will allow customization of savename for user-defined types. In addition it will not break anything.

I suggest to define e.g.

default_allowedtypes(::Any) = (Real, String, Symbol)

and to change savename to:

..., allowedtypes = default_allowedtypes(c)

Default arguments / config file for DrWatson

It would be great to be able to create local (and global) configuration files for DrWatson. Ideally, these config files would be able to set default prefixes, suffixes and separators, as well as file formats. I'm not sure what the best way to implement this is, though.

Increase verbosity of `collect_results`

Hi Jonas,

I am writing the final "real world example" which uses your collect_results. However both arguments black_list and special_list seem to fail. There is also no warning/error thrown so I don't get why.

I think I am using them as the documentation suggests. Can you have a look in https://juliadynamics.github.io/DrWatson.jl/dev/real_world/#Listing-completed-runs-1 and tell me what is wrong? The code is runnable you can run it locally by running the make.jl file (and you will have all the results almost instantly).

1.0 Release

This package has come a long way! It started by my wish that scientists shouldn't have to code the same thing over and over again. Then @JonasIsensee joined and made significant contributions and we released the first beta. Since then, I was extremely happy to see three more people doing (significant) contributions, @tamasgal @sebastianpech and @asinghvi17 .

Its probably time to reflect and see what steps are required for a 1.0 release. I believe DrWatson is now used by enough people to have undergone the basic necessary testing. I'll write below the list that comes to my mind, but everyone (tagged here or not) should contribute their thoughts or wishes.

• Update the Poster
• Rethink #43 . I am not so sure this was a good idea, as it tries to suggest structuring your project into a "Julia package form", which was never the intention. I am also not sure how many scientists write tests for their research projects? (I don't...)
• Implement #36 ? It is not really breaking, and only additive, but in the end of the day it does change what tagsave will output.
• Close #63 . I don't think we can do anything about it.
• Resolve #78 .
• Finish #81 .
• Fix #82

---

After 1.0 release I'll make a 10-minutes video introduction to DrWatson and upload it to youtube.

Integration with ploting

It would be nice if figures produced from simulations had the info to reproduce them associated with them. Maybe as metadata tags (which most formats support)? I'm just putting this out here as today I inherited a bunch of figures which I wouldn't know where they came from.

Recursive access

It may be worth it to extend access to be able to accept many keys, so that access(c, k1, k2) is translated to access(access(c, k1), k2).

Suggestion: Use PkgTemplates.jl to generate Project.toml?

The DrWatson.jl generated Project.toml is not well-formatted and no CI/CD supports:

julia> using DrWatson

julia> initialize_project("MMM", authors="me")
Activating new environment at `~/Downloads/MMM/Project.toml`
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `~/Downloads/MMM/Project.toml`
  [634d3b9d] + DrWatson v1.0.1
  Updating `~/Downloads/MMM/Manifest.toml`
  [634d3b9d] + DrWatson v1.0.1
  [5789e2e9] + FileIO v1.0.7
  [ae029012] + Requires v0.5.2
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 
"MMM"

Project.toml:

name = "MMM"
authors = ["me"]
[deps]
DrWatson = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1"

While PkgTemplates.jl generated Project.toml looks much better and has CI/CD/documentation/license supports:

julia> t = Template(;
                  user="xxxx",
                  license="MIT",
                  authors=["me"],
                  julia_version=v"1.0",
                  ssh=true,
                  plugins=[
                      TravisCI(),
                      Codecov(),
                      Coveralls(),
                      AppVeyor(),
                      GitHubPages(),
                      CirrusCI(),
                  ],
              )
Template:
  → User: xxxx
  → Host: github.com
  → License: MIT (me 2019)
  → Package directory: ~/.julia/dev
  → Minimum Julia version: v1.0
  → SSH remote: Yes
  → Add packages to main environment: Yes
  → Commit Manifest.toml: No
  → Plugins:
    • AppVeyor:
      → Config file: Default
      → 0 gitignore entries
    • CirrusCI:
      → Config file: Default
      → 0 gitignore entries
    • Codecov:
      → Config file: None
      → 3 gitignore entries: "*.jl.cov", "*.jl.*.cov", "*.jl.mem"
    • Coveralls:
      → Config file: None
      → 3 gitignore entries: "*.jl.cov", "*.jl.*.cov", "*.jl.mem"
    • GitHubPages:
      → 0 asset files
      → 2 gitignore entries: "/docs/build/", "/docs/site/"
    • TravisCI:
      → Config file: Default
      → 0 gitignore entries

julia> generate(t, "MMM");

will generate a lot of files

.
├── LICENSE
├── .appveyor.yml
├── .cirrus.yml
├── .gitignore
├── .travis.yml
├── Manifest.toml
├── Project.toml
├── README.md
├── docs
│   ├── Manifest.toml
│   ├── Project.toml
│   ├── make.jl
│   └── src
│       └── index.md
├── src
│   └── MMM.jl
└── test
    └── runtests.jl

4 directories, 10 files

The Project.toml looks especially better:

name = "MMM"
uuid = "0ca299d9-6e6b-49b5-857c-f01b2625cc38"
authors = ["me"]
version = "0.1.0"

[compat]
julia = "1"

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test"]

Could we switch to use PkgTemplates.jl to generate these files?

Convenience signature datadir(args...)

I propose a single one-liner
DrWatson.datadir(arg1, args...) = joinpath(datadir(), arg1, args...)

This will allow the following common pattern to be shorter.

path = joinpath(datadir(), "results", "mymodel")
path = datadir("results", "mymodel")

The following
datadir(args...) = joinpath(datadir(), args...)
causes a stackoverflow because args... may also be no arguments at all.

Preferred way to develop source

@Datseris could you give an example for how you set up your src for a project.

I had some trouble following the REPL-Based Workflow. In which I created a module file in src which includes other source files. Then in a script I would include(srcdir()*"module.jl") and using .ModuleName which worked fine. However, developing wasn't great because changes to the source files or module would not be updated by Revise, and reloading them got conflict warnings.

Instead I'm currently pushing the src directory to my LOAD_PATH and using the module in a usual way. This works well with Revise, but isn't really nice.

You mentioned that a better solution would be to ]dev? Is this in the src directory or the entire project directory? I probably misunderstand what ]dev does, but I thought it needed to be a package (though I'm not sure what that entails).

Move DrWatson to JuliaPhysics?

It seems to me that DrWatson would fit more into JuliaPhysics, revolving around general physics packages, than into JuliaDynamics. After all, it has nothing to do with dynamical systems.

What do you guys think about moving the package over?

Unknown option -C : data not saved

Hi @mauro3 , I am running some code that uses DrWatson on the cluster and I got this error when trying to save:

Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
┌ Warning: The Git repository is dirty! Appending _dirty to the commit ID
└ @ DrWatson ~/.julia/packages/DrWatson/nuHUe/src/saving_tools.jl:54
ERROR: LoadError: failed process: Process(`git -C /home/datseris/embeddingresearch-george diff HEAD`, ProcessExited(129)) [129]
Stacktrace:
 [1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
 [2] pipeline_error at ./process.jl:785 [inlined]
 [3] read(::Cmd) at ./process.jl:699
 [4] read(::Cmd, ::Type{String}) at ./process.jl:708
 [5] gitpatch(::String) at /home/datseris/.julia/packages/DrWatson/nuHUe/src/saving_tools.jl:89
 [6] #tag!#12(::String, ::Bool, ::Bool, ::LineNumberNode, ::Function, ::Dict{String,Any}) at /home/datseris/.julia/packages/DrWatson/nuHUe/src/saving_tools.jl:128
 [7] #tag! at ./none:0 [inlined]
 [8] #tagsave#25(::Bool, ::String, ::Bool, ::Bool, ::LineNumberNode, ::Function, ::String, ::Dict{String,Any}) at /home/datseris/.julia/packages/DrWatson/nuHUe/src/saving_files.jl:87
 [9] (::getfield(DrWatson, Symbol("#kw##tagsave")))(::NamedTuple{(:source,),Tuple{LineNumberNode}}, ::typeof(tagsave), ::String, ::Dict{String,Any}) at ./none:0
 [10] top-level scope at /home/datseris/embeddingresearch-george/scripts/lightcone/lightjob.jl:27
 [11] include at ./boot.jl:326 [inlined]
 [12] include_relative(::Module, ::String) at ./loading.jl:1038
 [13] include(::Module, ::String) at ./sysimg.jl:29
 [14] exec_options(::Base.JLOptions) at ./client.jl:267
 [15] _start() at ./client.jl:436
in expression starting at /home/datseris/embeddingresearch-george/scripts/lightcone/lightjob.jl:25

My entire job crashed afterwards. This came from the change to save the git DIFF. I think we must add a failsafe try-catch there as well, so that if it fails then there is no collapse just before the data are saved...

Overload `save` and `load` instead of aliasing?

Seems to me that wsave and wload is used everywhere internally for saving and loading files. At the moment these are just aliases for FileIO.save and FileIO.load respectively:

DrWatson.jl/src/DrWatson.jl

Lines 20 to 21 in 28ac0a1

	wsave = save
	wload = load

Would it be a good idea to instead do something like

wsave(args...) = save(args...)
wload(args...) = load(args...)

so that the user can overload the saving behavior used internally without overloading FileIO.save and FileIO.load? E.g. might want to save plots using Plots.save, in which case could just implement wsave(p::Plots.Plot, f) = Plots.savefig(p, f) and thus, for example, get safesave for "free" rather than having to overload safesave itself.

I'll can make a PR if this sounds like a good idea.

Error tagging new release

The REQUIRE file could not be found.
cc: @Datseris

function addrun!

A function addrun!(csvfile, d) that adds a simulation/experiment/whatever run to a csv file would be great.

This will allow to make csv tables out of existing simulations. Here is some things that quickly come to mind:

1. The function should operate similarly to savename, i.e. allow choosing which keys to save, which types, etc.
2. if the csvfile doesn't exist, it creates a new one and adds headers.
3. This function should be able to cooperate with (or even internally create) DataFrame.
4. It should somehow allow adding a run that has some additional parameters. How to do this is to be discussed. It could add missing entries to all other existing runs that do not have this parameter.
5. As above, when adding a run with fewer parameters, it should add missing/NA/Idon'tknowwhat to the parameters that do not exist.

Logo for DrWatson

Extensive investigations revealed that the famous Dr Watson didn't wear a monocle... :)

Saving in produce_or_load does not work for producing anonymous functions

I want to use produce_or_load in my code, and I am using some anonymous functions for preprocessing data before feeding them to model which I want to persist to file too, as an output from my function along with the trained model.
But then the saving fails, because it is not able to save it.

function train_model(settings)
        # ... some training
	# model is Flux model
	# edge_descriptor is anonymous function
	@dict(model, edge_descriptor)
end

things = produce_or_load(datadir("sims"), settings, train_model, prefix="model", tag=false)
# this does not save thing, thows warning that the saving failed

I noticed serialization is not used here, it would be nice if I could specified e.g. jls suffix and then serialization would be used, so it would work for this usecase.

make it so `produce_or_load` doesn't load the file with a keyword

produce_or_load is cool, but sometimes you want to use a similar approach but not even load the file.

e,g, in preparing simulations, you often need to be sure that some files exist, and if not to produce them, but you may not need to load them immediately.

`@savename` macro

Would be nice to have a macro @savename a b c that simply first calls d = @dict a b c and then returns savename(d).

reverse engineer `savename`

It would be nice to have a function that attempts to reverse engineer a string that is the output of savename. This function would always return prefix, params, suffix with prefix, suffix being strings (if non found then the empty strings) and params always a Dict with String keys (arbitrary choice).

To make this function is easy: first split by the connector argument (by default _), then identify prefix, suffix if any, and then find all the pairs connected by = and make them into the Dict.

Perfect project structure

What is the perfect project structure? The project structure is described in the src/project_structure file. I think the original version is already quite okay. Keep in mind we want to keep things basic: adding more folders, not using other folders, all should be flexible.

Add julia-compat line to initialize_project

The function initialize_project generates a project folder and structure. Then the Project.toml file is further edited here :

DrWatson.jl/src/project_setup.jl

Lines 211 to 216 in c25df1d

	pro = read(joinpath(path, "Project.toml"), String)
	w = "name = \"$name\"\n"
	if !(authors === nothing)
	w *= "authors = "*sprint(show, vecstring(authors))*"\n"
	end
	write(joinpath(path, "Project.toml"), w, pro)

In this part of the code we must add this:

[compat]
julia = "vx.y.z"

to be written in the Project.toml file. x.y.z should be the same as major.minor version of Julia that called initialize_project, which is of course simply string(VERSION)

.gitignore generated with initialize_project is write protected by default

Hi, I'm on Arch linux,

After initialising a project using the initialize_project function I get a .gitignore file that is write protected for me (I have to use sudo to change it).

Is this the intended outcome? It breaks with what cookiecutter does for example.

Path separators

I think it would be better to encourage the users the use of joinpath instead of * and hard coded OS dependent path separators.

At this moment the path accessor functions like datadir(), srcdir() etc. are designed to always return a path with a leading / as path separator just for convenience so that the users can type datadir()*"whatever" to generate a path to whatever.

Given that DrWatson's primary design goal is reproducibility, I think it would be better to switch to a more generic approach.

I already filed this pull request which was merged (thanks!): #64

So we now have things like datadir("whatever", "path", "you", "want") yielding a path to whatever:path:you:want with correct path separators. Also the projectdir() was extended with a method to concatenate the path-parts correctly.
However, there is still this leading / everywhere, which of course causes errors, especially when users get used to it.

What do you think?

Automated generation of project based on a text file

In my dreams, the function initialize_project somehow reads the project specification text file and then knows how to create the project.

This is great for documentation and consistency: the project specification text file src/project_structure is loaded and expanded into the documentation. The same file is used for project creation and thus everything is robust and consistent.

The question is, how to structure the project_structure file and how to read it in Julia to make this possible?