datalad-handbook / repro-paper-sketch

A template to create a reproducible paper with LaTeX, Makefiles, Python, and DataLad

The DataLad handbook 📙

This is a living resource on why and - more importantly - how to use DataLad. The rendered version is here: https://handbook.datalad.org, and is currently under initial development.

The handbook is a practical, hands-on crashcourse to learn and experience DataLad. You do not need to be a programmer, computer scientist, or Linux-crank. If you have never touched your computer's shell before, you will be fine. Regardless of your background and personal use cases for DataLad, the handbook will show you the principles of DataLad, and from chapter 1 onwards you will be using them.

Find more general information about the idea behind the handbook in the poster presented at the 2020 OHBM or dive straight into your DataLad adventure.

Contributing

Contributions in any form - pull requests, issues, content requests/ideas, ... are always welcome. If you are using the handbook and find that something does not work, please let us know. Likewise, if you are using DataLad for your individual project, consider contributing by telling us about your use-case. You can find out more on how to contribute here, and a list of all contributors so far below, in CONTRIBUTORS.md, and in .zenodo.json.

Notes for Instructors

The book is the basis for workshops and lectures on DataLad and data management. The handbook's course repository among other things contains live casts from the code examples in this book and slides. It is constantly growing, and everyone is free to use the material under the license terms below. Contributions and feedback are very welcome.

License

CC-BY-SA: You are free to

• share - copy and redistribute the material in any medium or format
• adapt - remix, transform, and build upon the material for any purpose, even commercially

under the following terms:

1. Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

2. ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Adina S. Wagner 💻 🖋 📖 🎨 🤔 🚇 🚧 📆 👀 📓 📢 ⚠️ 🐛 💡 💬 ️️️️♿️	Laura Waite 🤔 🚧 👀 📢 💬 🖋	Michael Hanke 💬 🐛 💻 🖋 📖 🎨 💡 🤔 🚇 🚧 🔌 📆 👀 🔧 ⚠️ 📢 📓 ️️️️♿️	Kyle Meyer 🐛 👀 💬 🖋 🤔	Marisa Heckner 🤔 📓 🐛 🖋	Benjamin Poldrack 💬 🤔 💡 ✅	Yaroslav Halchenko 👀 🖋 🤔 🐛
Chris Markiewicz 🐛	Pattarawat Chormai 🐛 💻	Lisa N. Mochalski 🐛 🖋 💡 🤔	Lisa Wiersch 🐛	Jean-Baptiste Poline 🖋	Nevena Kraljevic 📓	Alex Waite 👀 🐛 🤔
Lya K. Paas 🐛 💻	Niels Reuter 🖋	Peter Vavra 🤔 📓	Tobias Kadelka 📓	Peer Herholz 🤔	Alexandre Hutton 🖋 🐛	Sarah Oliveira 👀 🤔
Dorian Pustina 🤔	Hamzah Hamid Baagil 📓 🐛	Tristan Glatard 🐛 🖋	Giulia Ippoliti 🖋 💡	Christian Mönch 🖋	Togaru Surya Teja 🖋	Dorien Huijser 🐛 📓
Ariel Rokem 🐛	Remi Gau 🐛 🤔 🚧 👀 🚇 💻 🎨	Judith Bomba 🐛	Konrad Hinsen 🐛	Wu Jianxiao 🐛	Małgorzata Wierzba 📓 👀 ✅	Stefan Appelhoff 🚇 🔧 🐛
Michael Joseph 🤔 🖋 🐛	Tamara Cook 👀 🚇	Stephan Heunis 🐛 🚧 🖋 💡 👀	Joerg Stadler 🐛	Sin Kim 🐛 🖋 👀	Oscar Esteban 🐛	Michał Szczepanik 👀 🐛 🖋
eort 🐛	Myrskyta 🐛	Thomas Guiot 🐛	jhpb7 🐛	Ikko Ashimine 🐛	Arshitha Basavaraj 🖋 🐛 🚧	Anthony J Veltri 📓
Isil Bilgin 🐛 🚧	Julian Kosciessa 🖋	Isaac To 🚧 🖋 🐛	Austin Macdonald 🐛	Christopher S. Hall 🐛	jcf2 🐛	Julien Colomb 🖋
Danny Garside 🐛 🚧	Justus Kuhlmann 🖋	melanieganz 🐛	Damien François 🐛 🖋	Tosca Heunis 🐛 📓	Jeremy Magland 🐛

This project follows the all-contributors specification. Contributions of any kind welcome!

repro-paper-sketch's Issues

Testing the template on MacOS

I have tested the template on MacOS and it runs pretty much flawlessly. :)

There is one line that needs minor adjustment, I couldn't avoid creating a backup file with sed and needed to run this call instead:
sed -i'.bak' 's/muted/Blues/g' code/mk_figuresnstats.py

But let me use this opportunity to introduce several other minor changes to the README:

• I added a broader overview of how the manuscript is actually built at the beginning of the README (this is a copy-paste of what you already have in the .tex file, but I think it nicely clarifies the general idea of this reproducible paper template).
• I reorganised the software requirements a bit: I would skip introducing the requirements.txt until it's needed. Otherwise, people may try to install seaborn, sklearn and DataLad API themselves only to realise that they only needed to create a virtual environment as explained later on. On the other hand it might be worthwhile to mention the virtualenv as a requirement (especially assuming that not all readers are fluent Python users).
• I tried to stress the difference between datalad get and datalad get -n more, becase this can easily be missed by a beginner.
• I would encourage people to actually READ the manuscript once it's created. :)

Also, I had the general feeling that a lot of details on the basics of DataLad is provided, but not so much on other elements (e.g. LaTeX, Makefiles). But this is understandable, it's a DataLad tutorial after all.

make not finishing during first call

When trying to run make for the first time, it does never finish.
Tried debugging by deactivating the chronic in Makefile: CHRONIC=$( echo '' )
Latex then reports and waits for user input:

(/usr/share/texlive/texmf-dist/tex/generic/gettitlestring/gettitlestring.sty))
ABD: EveryShipout initializing macros (./results_def.tex
! Missing $ inserted.
<inserted text> 
                $
l.1 install(ok): /home/twutz/vtwutz_
                                    ext/repro2/input (dataset) [Installed su...

? 

Hence, the chronic, in this case, is counterproductive, because it suppresses the prompt.

This seems to be caused by the first lines in results_def.tex:

install(ok): /home/twutz/vtwutz_ext/repro2/input (dataset) [Installed subdataset in 
order to get /home/twutz/vtwutz_ext/repro2/input/iris.csv]
get(ok): input/iris.csv (file) [from web...]
action summary:
  get (ok: 1)
  install (ok: 1)

Should code/mk_figuresnstats.py be adapted to suppress these messages, probably from dl.get(data) or adapted to write write results_def.tex directly instead of relying on redirecting stdout?
I am not enough familiar with python, to trace this.

Encountered with environment:
OS: mint uma (ubuntu 20.04)
datalad command line (from conda): 0.16.5

python venv versions:
(repro) (base) twutz@mate:~/vtwutz_ext/repro2$ pip list --local

Package            Version  
------------------ ---------
annexremote        1.6.0    
boto               2.49.0   
certifi            2022.6.15
cffi               1.15.1   
chardet            4.0.0    
charset-normalizer 2.1.0    
cryptography       37.0.4   
cycler             0.11.0   
datalad            0.16.7   
distro             1.7.0    
fasteners          0.17.3   
fonttools          4.34.3   
humanize           4.2.3    
idna               3.3      
importlib-metadata 4.12.0   
iso8601            1.0.2    
jeepney            0.8.0    
joblib             1.1.0    
keyring            23.6.0   
keyrings.alt       4.1.0    
kiwisolver         1.4.3    
matplotlib         3.5.2    
msgpack            1.0.4    
numpy              1.23.0   
packaging          21.3     
pandas             1.4.3    
patool             1.12     
Pillow             9.2.0    
pip                20.0.2   
pkg-resources      0.0.0    
platformdirs       2.5.2    
pycparser          2.21     
pyparsing          3.0.9    
python-dateutil    2.8.2    
python-gitlab      3.6.0    
pytz               2022.1   
requests           2.28.1   
requests-toolbelt  0.9.1    
scikit-learn       1.1.1    
scipy              1.8.1    
seaborn            0.11.2   
SecretStorage      3.3.2    
setuptools         44.0.0   
simplejson         3.17.6   
six                1.16.0   
sklearn            0.0      
threadpoolctl      3.1.0    
tqdm               4.64.0   
urllib3            1.26.10  
wheel              0.34.2   
Whoosh             2.7.4    
zipp               3.8.0