lhcb / scheduling-event-model Goto Github PK

@GerhardRaven I am giving a try to running your code.
First issue : Ninja is not installed by default on lxplus it seems

[yamhis@lxplus0067 TransformAlgorithm_playground]$ source setup-lhcb-build.sh
/usr/bin/which: no ninja in (/afs/cern.ch/group/z5/bin:/afs/cern.ch/user/y/yamhis/scripts:/usr/kerberos/sbin:/afs/cern.ch/lhcb/software/releases/LBSCRIPTS/LBSCRIPTS_v8r4p2/InstallArea/scripts:/afs/cern.ch/sw/lcg/releases/LCG_79/Python/2.7.9.p1/x86_64-slc6-gcc48-opt/bin:/afs/cern.ch/lhcb/software/releases/COMPAT/COMPAT_v1r19/CompatSys/x86_64-slc6-gcc48-opt/bin:/afs/cern.ch/sw/contrib/CMT/v1r20p20090520/Linux-x86_64:/afs/cern.ch/lhcb/bin:/afs/cern.ch/sw/lcg/contrib/CMake/2.8.9/Linux-i386/bin:/usr/sue/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin)
Ninja is not installed. Will Install Ninja ..

Find attached the work that Florian(*) has done during the summer to “accelerate” tracking code.
We discussed with Ben and Lionel the possibility of having him involved in the workshop. My idea was to ask him to give us feed back (a sort of review) on what were the
bottle necks and what seem to slow down any code acceleration in the current LHCb software. Since he is didn’t work on the current Event Model he might have an unbiased view.

What do you think ?
Florian.pdf
(*) Florian was a technical student and he will start his PhD tomorrow supervised by Ben at CERN and Lionel Lacassagne in Paris.

From Roel :

• Backwards compatibility of the data files. Right now if a new object is
  stored in a ROOT file, the files becomes completely unreadable by old
  software versions although they would understand everything else fine. We
  should allow parts of the event to be "blindly" copied if they are not
  understood and everything we understand to be used. For the swimming this
  made things extremely painful.
• Data preservation. I think we should make proper specifications and
  versions of all event model classes such that they can always be
  understood later. This is currently sort of in the code, but not really on
  paper. It would be best if they were digital too, somehow.

After discussing with @betatim though of this issue.
This is a list of small projects people could work on during the parallel sessions :

• Make different level of abstraction.
• Make it easier to store new objects/classes in output files.
• More to be added.

• Dataset content: is the TurboDST idea the right direction, how do we deal with global event data (e.g. isolation variables), how do we ensure independence of stripping line outputs etc.
• How do people interact with the event model objects: role and evolution of LoKi, Turbo/MDST vs. NTuples, role of ROOT (what are the use cases for directly accessing our event data interactively from with Root?)
• Is it still necessary to have an XML description of our event classes (à la GaudiObjDesc), what are the use cases?

I discussed with Pierre (Billoir some of you know him already), who has been since he started working on LHCb (and for a while before) thinking about potential speed ups and improvements to the upgrade-tracking. For the time being we can summarize the discussion in two points :

• Weight matrix formalism for the Kalman Filter.
• Trials to reduce computations in some steps of the reconstruction.

It seems to me this can nicely fit in the Event Model. The way I see it he could share with us his thoughts in a // session. @betatim @GerhardRaven @manuelschiller Thoughts ?

• [Monday + Common Scheduling and Framework ] Setting up the scene (Roel + Yas).
• [Monday + Common Scheduling and Framework ] What are the issues with the current EM, what does the future one need to address (Francesca + Matt K.).
• [Monday + Common Scheduling and Framework ] FCC event model (Benedikt).
• [Monday + Common Scheduling and Framework ] Structure of arrays (Manuel).
• [Tuesday] Hacking.
• [Wednesday]Vectorization (Florian).
• [Wednesday]Track fit - matrices (Pierre).
• [Wednesday] Packing/Unpacking - Precision (Chris Jones).
• [Wednesday] Data preservation (Ana?).
• [Wednesday] Files summaries (FSR, headers ) (Roel).
• [Thursday] Hacking.

How to maintain some state in an algorithm for histograms, etc that is machine checkable for that state not influencing the out come.

CONTRIBUTING tries to point to gh-fork-pull, gh-fork and gh-pull, but those do not get resolved...

https://www.youtube.com/watch?v=90-S1M7Ny_o&t=21m2s

It is super trendy, etc, etc and there are lots of other libraries that handle the specification and then scheduling of the graph in a similar way, but I just found this video first.

The things I like:

• split between specifying the graph and running it (frontend vs backend)
• user never sees task/thread/cuda/etc low levelness

This issue is to organise the schedule of the Scheduling/Framework sessions.

Please post your contributions/ideas below and we will update the table

A suggestion (feel free to reject it) for the workshop (and after it) is to draw up
a ‘design guidelines’ for the various topics. This is what has already started here.

Let me add my three favourite ‘commandments’:

1. EDM classes shall be efficiently move-constructible
2. EDM classes shall be composable
3. EDM classes shall be immutable once made available

rationale:

2. is driven by 3)
3. is driven by sanity, and making the system 'understandable' (and multi threading friendly)
4. will allow producers to ‘pass by value’ and we avoid explicit heap storage.

If you look at GaudiAlg/GaudiAlg/TransformAlgorithm.h (assuming you’ve run
the script I sent out yesterday -- I will add a pull request for it) you will see a clear
example of why I want 1), and the impact it has (i.e. simplifies things) if you look at eg.
Tf/PatAlgorithms/src/PatMatch.cpp and Calo/CaloReco/src/CaloSinglePhotonAlg.cpp
which are two examples of code adapted to use TransformAlgorithm...

The current performances of the upgrade tracking will be probably used as input for discussion in various sessions.
I wonder if we want to have a dedicated contribution ? Or simply gather the information (slides?) and s have them available. What do you think ?

Hi,

your script cannot work in chroot environments, since it changes MYSITEROOT to cvmfs which may or may not be available, and it changes CMTCONFIG as well (again, binaries for the new setting may or may not be available). I've sort of hacked things by hand to point to the right locations on my laptop (and use gcc 4.8 which is the only version for which I have binaries installed), but even after that, I cannot start compiling:

[mala@fenchurch]~/paris2015/moo/Gaudi% make
mkdir -p /home/mala/paris2015/moo/Gaudi/build.x86_64-slc6-gcc48-opt
cd /home/mala/paris2015/moo/Gaudi/build.x86_64-slc6-gcc48-opt && cmake -DCMAKE_USE_CCACHE=ON -DCMAKE_TOOLCHAIN_FILE=/home/mala/paris2015/moo/Gaudi/toolchain.cmake -GNinja /home/mala/paris2015/moo/Gaudi
CMake Error: CMake was unable to find a build program corresponding to "Ninja". CMAKE_MAKE_PROGRAM is not set. You probably need to select a different build tool.
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_C_COMPILER_ENV_VAR
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_C_COMPILER
CMake Error: Could not find cmake module file: /home/mala/paris2015/moo/Gaudi/build.x86_64-slc6-gcc48-opt/CMakeFiles/2.8.12.2/CMakeCCompiler.cmake
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_CXX_COMPILER_ENV_VAR
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_CXX_COMPILER
CMake Error: Could not find cmake module file: /home/mala/paris2015/moo/Gaudi/build.x86_64-slc6-gcc48-opt/CMakeFiles/2.8.12.2/CMakeCXXCompiler.cmake
-- Configuring incomplete, errors occurred!
make: *** [/home/mala/paris2015/moo/Gaudi/build.x86_64-slc6-gcc48-opt/build.ninja] Error 1

If that's expected behaviour, and if I should therefore compile this on lxplus, please let me know. In any case, things need at least some kind of clarification...

Cheers,

Manuel