go-hep / hep

hep is the mono repository holding all of go-hep.org/x/hep packages and tools

hep

hep is a set of libraries and tools to perform High Energy Physics analyses with ease and Go

See go-hep.org for more informations.

Forum

Drop an email at ~sbinet/[email protected] or visit the web interface lists.sr.ht/~sbinet/go-hep to discuss about Go-HEP or ask for help.

License

hep is released under the BSD-3 license.

Documentation

Documentation for hep is served by GoDoc.

Contributing

Guidelines for contributing to go-hep are available here: go-hep.org/contributing

Contributors

This project exists thanks to all the people who contribute.

Motivations

Writing analyses in HEP involves many steps and one needs a few tools to successfully carry out such an endeavour. But - at minima - one needs to be able to read (and possibly write) ROOT files to be able to interoperate with the rest of the HEP community or to insert one's work into an already existing analysis pipeline.

Go-HEP provides this necessary interoperability layer, in the Go programming language. This allows physicists to leverage the great concurrency primitives of Go, together with the surrounding tooling and software engineering ecosystem of Go, to implement physics analyses.

Content

Go-HEP currently sports the following packages:

• go-hep.org/x/hep/brio: a toolkit to generate serialization code
• go-hep.org/x/hep/fads: a fast detector simulation toolkit
• go-hep.org/x/hep/fastjet: a jet clustering algorithms package (WIP)
• go-hep.org/x/hep/fit: a fitting function toolkit (WIP)
• go-hep.org/x/hep/fmom: a 4-vectors library
• go-hep.org/x/hep/fwk: a concurrency-enabled framework
• go-hep.org/x/hep/groot: a pure Go package for ROOT I/O (WIP)
• go-hep.org/x/hep/hbook: histograms and n-tuples (WIP)
• go-hep.org/x/hep/hplot: interactive plotting (WIP)
• go-hep.org/x/hep/hepmc: HepMC in pure Go (EDM + I/O)
• go-hep.org/x/hep/hepevt: HEPEVT bindings
• go-hep.org/x/hep/heppdt: HEP particle data table
• go-hep.org/x/hep/lcio: read/write support for LCIO event data model
• go-hep.org/x/hep/lhef: Les Houches Event File format
• go-hep.org/x/hep/rio: go-hep record oriented I/O
• go-hep.org/x/hep/sio: basic, low-level, serial I/O used by LCIO
• go-hep.org/x/hep/slha: SUSY Les Houches Accord I/O
• go-hep.org/x/hep/xrootd: XRootD client in pure Go

Installation

Go-HEP packages are installable via the go get command:

$ go get go-hep.org/x/hep/fads

Just select the package you are interested in and go get will take care of fetching, building and installing it, as well as its dependencies, recursively.

hep's Issues

fwk: investigate HCL as a high-level configuration language

From @sbinet on August 20, 2014 14:55

people might cringe a bit at configuring a fwk job in go.
even if the github.com/go-hep/fwk/job.Job interface can be considered quite nice.

also, providing a "Turing-complete" language as a configuration language introduces problems of its own (one has to debug the program which creates the program one wants to run.)
c.f. all the issues with e.g. the Athena/Gaudi python joboptions.

Using a declarative language like hcl might fit the bill.

Copied from original issue: go-hep/fwk#24

csvutil: sql support

From @sbinet on July 7, 2016 15:41

we may need to support (a subset of?) SQL queries.

parsers & SQL generators

• https://github.com/influxdata/influxdb/tree/master/influxql
• https://github.com/benbjohnson/sql-parser/blob/master/parser.go
• https://github.com/knq/xo
• http://0value.com/A-PEG-parser-generator-for-Go

ROW-wise / COL-wise

• http://db.csail.mit.edu/pubs/abadi-column-stores.pdf

Copied from original issue: go-hep/csvutil#1

hbook: I/O compat with AIDA

From @sbinet on December 7, 2016 16:11

add support for reading/writing from/to AIDA files

Copied from original issue: go-hep/hbook#12

pawgo: add support for hbook.S2D

From @sbinet on December 15, 2016 9:49

Copied from original issue: go-hep/pawgo#2

fwk: implement a job-server REST API

From @sbinet on August 21, 2014 11:56

using job configurations from issue go-hep/fwk#25 , it should be possible to submit jobs to a web server with simple GET/POST queries.

Copied from original issue: go-hep/fwk#26

fwk: investigate tsk.Process introspection

From @sbinet on August 19, 2014 7:2

investigate whether usage of reflect on a Task.Process code would be possible to automagically extract the data flow.

Copied from original issue: go-hep/fwk#15

rootio: Scanner leaks Tree implementation details

Scanner does not properly seal user from implementation details of Tree.
if the underlying file pointer is not at the correct position (ie pos=0), Scanner will fail.

see: c58ed3e.

rio: data type codecs

From @sbinet on September 16, 2014 13:12

• investigate whether encoding/decoding of types could be stored into rio files
• assuming the proposal for Go plugins API goes in, it would be possible to dynamically load code. => just write the package name ?
• fwd/bwd compat of types. write the AST or some serialized version of the data type for versionning (instead of just the package name + data type name) (AKA, schema evolution)

Copied from original issue: go-hep/rio#2

fwk: investigate go's context.Context

From @sbinet on August 15, 2014 9:18

blog.golang.org has a nice write-up on contexts:

• http://blog.golang.org/context
• http://godoc.org/code.google.com/p/go.net/context

especially the part about hierarchical contexts.

would this be a better fit for fwk.Ctxt:

• to handle updates to a collection ?
• to handle aborts ? (e.g. abort a task because of a timeout, abort run because no more events in input file, etc...)

Copied from original issue: go-hep/fwk#11

fwk: implement a job-configuration dumper+loader for user-defined types

From @sbinet on September 10, 2014 15:32

it should be possible to dump a fwk.App configuration into:

a human-readable format
a machine-readable format (serialization/gob/json/...)
for later re-use, comparison or regression-tests purposes.

it should also work for user-defined types. (currently, the JSON based format only works for builtins)

Copied from original issue: go-hep/fwk#34

fwk: implement a n-tuple svc

From @sbinet on August 19, 2014 13:34

Copied from original issue: go-hep/fwk#19

fwk: multi-node distributed tasks

From @sbinet on August 14, 2014 19:17

investigate feasibility of a multi-process, multi-node fwk.Task process.

• rpc
• golang circuit

Copied from original issue: go-hep/fwk#8

fwk: implement a property documentation system

From @sbinet on September 11, 2014 13:27

concrete components may have properties.
should we provide a way to expose/enforce/attach documentation for them ?
godoc is useless for them.

Copied from original issue: go-hep/fwk#37

rootio/cmd/root-srv: support arbitrarily nested objects

right now, root-srv will fail to plot foo.root/dir1/dir2/h1.

rootio: add support for TChain

it should be possible to combine multiple rootio.Trees (with the same layout) from multiple ROOT files into one rootio.Tree.

right, now, to get at a rootio.Tree, one does:

func main() {
	f, err := rootio.Open("testdata/chain.1.root")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()

	obj, err := f.Get("tree")
	if err != nil {
		log.Fatal(err)
	}
	tree := obj.(rootio.Tree)
	fmt.Printf("entries= %v\n", tree.Entries())
}

the rootio.Tree interface is:

// Tree is a collection of branches of data.
type Tree interface {
	Named
	Entries() int64
	TotBytes() int64
	ZipBytes() int64
	Branch(name string) Branch
	Branches() []Branch
	Leaves() []Leaf

	getFile() *File
	loadEntry(i int64) error
}

it should be possible to have a type that implements the rootio.Tree interface but logically concatenates multiple rootio.Trees together.
Something like the io.MultiReader function: https://godoc.org/io#MultiReader

Initially, one could just require the user to pass already created/retrieved rootio.Tree to a, e.g., Chain function:

// Chain returns a new Tree that is the logical concatenation of all the input Trees.
func Chain(trees ...Tree) Tree { ... }

one should think about resource ownership:

• who should close the ROOT file containing a Tree when that Tree has been completely read ?
• how to integrate the chain type with the Scanners ?
• is it better to have a Chain function that takes a tree name and a list of file names ? (so resource ownership is clearer). it's less composable...
• perhaps introduce a ChainScanner that opens and closes files as needed when each Tree has been consumed ?

hbook: add support for Scatter3D

From @sbinet on December 7, 2016 16:33

Copied from original issue: go-hep/hbook#16

rootio: support for TGraph

@aphecetche wants them

hsvc: add support for hbook.H2D

From @sbinet on December 15, 2016 9:47

Copied from original issue: go-hep/fwk#42

hbook: add rebinning operations to H1D

From @sbinet on December 7, 2016 16:9

Copied from original issue: go-hep/hbook#11

rootio: provide a database/sql driver backend for rootio.Tree

this would allow to use rootio.Tree from hbook/ntup.Ntuple (and use hbook/ntup.Ntuple.Scan() and friends)

fwk: memory locality & garbage collector

From @sbinet on August 14, 2014 19:21

investigate usage of sync.Pool

• usage in event-based banks/arenas
• interplay with I/O
• interplay with garbage collection

Copied from original issue: go-hep/fwk#9

hbook/yoda

From @sbinet on December 12, 2016 16:23

add a helper package to read/write "archives" of YODA files, so one can read/write a complete collection of hbook values (H1D, H2D, Scatter2D, ...) from/to a single YODA file.

Copied from original issue: go-hep/hbook#26

csvutil/csvdriver: add ability to name columns

From @sbinet on November 16, 2016 9:15

right now, opening a CSV connection is done like so:

func foo() {
	// Open the CSV file as a database table.
	db, err := csvdriver.Conn{
		File:    "../data/iris.csv",
		Comment: '#',
		Comma:   ',',
	}.Open()
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	// Start a database transaction.
	tx, err := db.Begin()
	if err != nil {
		log.Fatal(err)
	}
	defer tx.Commit()

	// Define a SQL query that we will execute against the CSV file.
	query := "SELECT var3, var4, var5 FROM csv WHERE var5 = \"Iris-versicolor\""

	// etc...
}

this makes a query rather cumbersome to read.

it should be possible to open a CSV connection with an optional Columns []string field:

// Open the CSV file as a database table.
db, err := csvdriver.Conn{
	File:    "../data/iris.csv",
	Comment: '#',
	Comma:   ',',
	Columns: []string{"sepal_length", "sepal_width", "petal_length", "petal_width", "species"},
}.Open()

such that the query reads:

// Define a SQL query that we will execute against the CSV file.
query := `SELECT petal_length, petal_width, species FROM csv WHERE species = "Iris-versicolor"`

What do you think @dwhitena ?

Copied from original issue: go-hep/csvutil#2

data type codecs

From @sbinet on September 16, 2014 13:12

• investigate whether encoding/decoding of types could be stored into rio files
• assuming the proposal for Go plugins API goes in, it would be possible to dynamically load code. => just write the package name ?
• fwd/bwd compat of types. write the AST or some serialized version of the data type for versionning (instead of just the package name + data type name) (AKA, schema evolution)

Copied from original issue: go-hep/rio#2

fwk: detect double-put in dflowsvc

From @sbinet on September 13, 2014 8:49

make sure consistency is upheld in face of double-put:

store := ctx.Store()
data := 42
// store data
err := store.Put("my-data", &data)

// oops, another time, under the same key
err = store.Put("my-data", &data)

otherwise, this could wreak havoc in the sync'ing of achan-s.

Copied from original issue: go-hep/fwk#39

pawgo: add support for hbook/ntup.Ntuple

From @sbinet on December 15, 2016 9:49

Copied from original issue: go-hep/pawgo#3

fwk: investigate concurrent start-up of components

From @sbinet on August 22, 2014 16:19

gaudi had issues with initialization of components (especially cycles in services/tools: ToolA -> ToolB -> ToolC -> ... -> ToolA)

the lack of resource dependencies declaration (component A needs component B) prevented to efficiently discover such issues.
it also prevented to initialize components in parallel.

investigate whether fwk couldn't do better here.

Copied from original issue: go-hep/fwk#28

fwk: consider using gonum/graph/formats/dot instead of awalterschulze/gographviz

to reduce the amount of dependencies, consider using "gonum/graph/formats/dot" instead of "awalterschulze/gographviz".

rootio/cmd/root-srv: do not keep files in memory

right now, when a user uploads a file, root-srv gobbles it up all in memory and keeps it there (until the user session expires).

• introduce a local mode where the file is directly open from the disk
• when remote or on appengine, store the file on some temporary storage (a tmp directory keyed on the user session name)

hbook: add rebinning operations to H2D

From @sbinet on December 7, 2016 16:9

Copied from original issue: go-hep/hbook#10

fwk: detect r/w cycles in dflowsvc

From @sbinet on September 12, 2014 18:27

detect and prevent components from applying r/w updates of items in the data-store

Copied from original issue: go-hep/fwk#38

fastjet: implement delaunay triangulation

From @sbinet on October 20, 2015 11:48

to provide an NlnN combinatorics (instead of O(N^3)) fastjet should implement Delaunay triangulations and voronoi tesselation (as is done in the C++ version of fastjet via the CGAL library)

https://en.wikipedia.org/wiki/Delaunay_triangulation
https://en.wikipedia.org/wiki/Voronoi_diagram
https://github.com/sbinet-staging/poly2tri.golang/blob/master/testbed/main.go

Copied from original issue: go-hep/fastjet#1

hsvc: add support for hbook.P1D

From @sbinet on December 15, 2016 9:48

Copied from original issue: go-hep/fwk#44

fwk: investigate a virtual filesystem

From @sbinet on August 18, 2014 14:48

do we need a virtual (r/w) filesystem ?

• check out possible candidates: https://github.com/sourcegraph/rwvfs

Copied from original issue: go-hep/fwk#13

hbook: add support for variable size bins for H2D

From @sbinet on December 9, 2016 18:5

Copied from original issue: go-hep/hbook#25

rootio: add support for reading TTrees with user classes

rootio: convert TH1x to hbook.H1D

rootio: convert TH2x to hbook.H2D

hsvc: add support for hbook.S2D

From @sbinet on December 15, 2016 9:47

Copied from original issue: go-hep/fwk#43

hbook: add support for Profile2D

From @sbinet on December 7, 2016 16:35

Copied from original issue: go-hep/hbook#17

hbook: add support for variable size bins for H1D

From @sbinet on December 9, 2016 18:5

Copied from original issue: go-hep/hbook#24

hbook: add support for Scatter1D

From @sbinet on December 7, 2016 16:33

Copied from original issue: go-hep/hbook#15

fwk: evt-level timeout

From @sbinet on August 13, 2014 6:45

fwk should support the ability to abort the processing of an event when it reaches a given timeout

Copied from original issue: go-hep/fwk#5

pawgo: add support for hbook.H2D

From @sbinet on December 15, 2016 9:49

Copied from original issue: go-hep/pawgo#1

rio: define a better on-disk format

From @sbinet on September 16, 2014 9:44

• define a format friendlier to concurrent I/O
• inspiration from Linux/BSD VFS ?
  • ZFS
  • Ext4
  • BTRFS
• inspiration from HDF5 ?

Requirements:

• allow streaming from http ?
• optimize for main usage: write once, read many
• optimize for read-speed
• use cases: user analysis, bulk reconstruction
• MPI I/O compliant ?

BTRFS

http://git.kernel.org/cgit/linux/kernel/git/mason/btrfs-progs.git/tree/
https://btrfs.wiki.kernel.org/index.php/Btrfs_design
http://www.cs.ucdavis.edu/~pandey/Teaching/ECS150/Lects/07fs.pdf

Ext4

https://ext4.wiki.kernel.org/index.php/Main_Page
http://foss.in/2006/cfp/slides/ext4-foss.pdf
https://ols.fedoraproject.org/OLS/Reprints-2008/kumar-reprint.pdf
https://github.com/torvalds/linux/tree/master/fs/ext4
http://atrey.karlin.mff.cuni.cz/~jack/papers/lk2009-ext4-btrfs.pdf

ZFS

http://maczfs.googlecode.com/files/ZFSOnDiskFormat.pdf
https://blogs.oracle.com/bonwick/en_US/entry/zfs_block_allocation
https://blogs.oracle.com/bonwick/entry/space_maps
https://github.com/zfsonlinux/zfs/

XFS

http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf

HDF5

http://www.hdfgroup.org/HDF5/doc/H5.format.html

ROOT

https://github.com/root-mirror/root/tree/master/io/doc/TFile
http://root.cern.ch/root/htmldoc/TFile.html
ftp://root.cern.ch/root/doc/11InputOutput.pdf

A4

https://github.com/a4/a4
http://arxiv.org/abs/1208.1600

blosc

http://www.blosc.org/
http://www.blosc.org/blosc-in-depth.html

FUSE

http://bazil.org/fuse/

LTFS

http://en.m.wikipedia.org/wiki/Linear_Tape_File_System

XZ

http://tukaani.org/xz/xz-file-format.txt

ACQt

https://indico.in2p3.fr/event/10875/session/7/material/0/3.pdf

Compression

Brotli

http://www.ietf.org/id/draft-alakuijala-brotli-07.txt
https://github.com/google/brotli/

Copied from original issue: go-hep/rio#1

rootio/cmd/root-srv: support plotting branches of builtins

fwk: task-level timeout

From @sbinet on August 13, 2014 6:46

fwk should (perhaps) support task-level timeout (and then abort event) when the task takes too much time to process.

Copied from original issue: go-hep/fwk#6

hbook: retrieve x position of maximum within given range for H1D

From @ebusato on October 12, 2016 15:10

Hi,

I'd like to retrieve the position on the x axis of the y maximum of the histogram within a certain range provided by the user. Right now you have the Max() method that returns the y maximum. What do you think would be the best option to implement what I need :

1. Change the Max() method by adding two parameters to specify the range and one return value which is the maximum position on the x axis
2. Create a new method and let Max() untouched in order to not impact clients.

?

Let me know and I'll send a PR.

Emmanuel

Copied from original issue: go-hep/hbook#8

fwk: investigate a boltdb-like interface for detector store

From @sbinet on August 19, 2014 6:58

boltdb has a nice interface for transactions (updates and read-only):
https://github.com/boltdb/bolt
https://github.com/boltdb/bolt#transactions

• evaluate API impact on fwk
• evaluate performances impact

Copied from original issue: go-hep/fwk#14

pawgo: add support for hbook.P1D

From @sbinet on December 15, 2016 9:50

Copied from original issue: go-hep/pawgo#4