cms-gem-daq-project / gem-plotting-tools Goto Github PK
View Code? Open in Web Editor NEWRepository for GEM commissioning plotting tools
License: GNU General Public License v3.0
gem-plotting-tools's People
Contributors
Stargazers
Watchers
Forkers
jsturdy cbravo135 bdorney akotsoke lmoureaux eklavyafcb mexanick benjaminrs huangh927 elizabethrosestarling brentstone andrewlevin caruta fraivone hyunyong xdlyu fsimone91 simonepv adeiorio ibubanja lpetre-ulb giovanni-mocellin sbutalla bregnery dteague vetensgem-plotting-tools's Issues
Feature Request: Add functionality for anaSBitThresh.py to ana_scans.py
Brief summary of issue
Now have v3 detectors in P5. Should have v3 functionality for ana_scans.py. Right now this is true for all but anaSBitThresh.py.
Steps outlined below. Testing may show that additional modifications are needed but these should be the main ones.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Modify ana_config dictionary
Need to modify ana_config:
Lines 3 to 10 in 0ce8667
To have an additional entry:
"rateSBITs":"anaSBitThresh.py", #channelOR case
"rateSBITsch":"anaSBitThresh.py", #perchannel case
Modify tree_names dictionary
Need to modify tree_names:
Lines 16 to 28 in 0ce8667
To have additional entries:
"rateSBITs":("SBitRateData.root", "rateTree"), #channelOR case
"rateSBITsch":("SBitRateData.root", "rateTree"), #perchannel case
"rateSBITsAna":("SBitRateData/SBitRatePlots.root", "thrAnaTree"),
Modify getDirByAnaType(...) to support new filepaths
Need to modify getDirByAnaType:
gem-plotting-tools/anautilities.py
Lines 73 to 85 in 0ce8667
To support additional filepaths:
elif anaType == "rateSBITs":
dirPath = "%s/%s/%s/channelOR/"%(dataPath,cName,"sbitRate")
elif anaType == "thresholdvftrig":
dirPath = "%s/%s/%s/perchannel/"%(dataPath,cName,"sbitRate")
Modify launchAnaArgs(...) to support new tool
Then we would need to modify launchAnaArgs(...) to support the new tool. An elif statement similar to this one for scurve analysis would need to be added:
gem-plotting-tools/ana_scans.py
Lines 54 to 76 in 0ce8667
This should support all the input arguments that are defined (and imported) by anaSBitThresh.py. To ensure this the main block may need to be modified to support options:
gem-plotting-tools/ana_scans.py
Lines 165 to 179 in 0ce8667
And then the debug block:
gem-plotting-tools/ana_scans.py
Lines 190 to 204 in 0ce8667
parallel submission block:
gem-plotting-tools/ana_scans.py
Lines 233 to 248 in 0ce8667
and series submission block:
gem-plotting-tools/ana_scans.py
Lines 206 to 223 in 0ce8667
would all need to be modified to support the new input arguments.
Expected Behavior
For v3 electronics in P5 ana_scans.py should support anaSBitThresh.py. Adding support for this analysis routine will not disrupt v2b operation/analysis.
Current Behavior
ana_scans.py does not support anaSBitThresh.py.
Context (for feature requests)
Presently not possible to analyze sbit rate vs. threshold scans in parallel. Would be desirable.
Your Environment
- Version used: 0ce8667
- Shell used: any
Bug Report: anaUltraScurve.py still has event count hard coded in one spot
Brief summary of issue
In anaUltraScurve.py the value for events/2 is still harded at line 151. Here it is set as 500 but should be an input parameter from the input TTree.
This should be changed since if a user takes an scurve with non-default settings (nevt=1000) this part of the code may not work as intended.
Low priority since this will work on scans taken with default parameters.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Value for events/2 at line 151 should not be hard coded but instead be an input parameter.
Current Behavior
Value is hard coded at events/2 = 500 at line 151.
Possible Solution (for bugs)
Change Line 151 to take an input instead of a hard coded value.
Your Environment
- Version used: 164fcef
- Shell used: /bin/bash
Discussion: Charge in fC For HighNoise or HighEffPed?
Brief summary of issue
When analyzing s-curves the results will now be calculated in fC. Before the maskReason for HighNoise and HighEffPed was determined in DAC units of VCAL or CFG_CAL_DAC. This presents a problem as there is chip-to-chip variations. Additionally there are differences between the vfat3 and vfat2 calibration modules so the step size of the DAC unit (in mV) is also different.
I think we should make a push for determining what HighNoise and HighEffPed classify as in terms of fC instead of DAC units. It would be good if we could also keep the MIP signal charge in mind.
Maybe this is a job for the DPG team but since it's in the code thought perhaps should open an issue page.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
- Discussion
Expected Behavior
HighNoise and HighEffPed classifiers should be defined on a value of fC.
Current Behavior
In the master branch they compared against hard coded values that are in terms of DAC units. This means that a HighEffPed in one chip will mean a different amount of charge in another chip:
gem-plotting-tools/anaUltraScurve.py
Lines 286 to 287 in 1360709
In the develop branch these quantities are in fC by default but compared against the old DAC unit cuts:
https://github.com/cms-gem-daq-project/gem-plotting-tools/blob/develop/anaUltraScurve.py#L373-L374
Probably need some sort of study that defines this.
Context (for feature requests)
At some point CMS will ask us for our dead channel fraction again and may ask more detailed question than just "channels masked/unmasked."
Feature Request: fitScanData.py Takes Number of Injections as Input
Brief summary of issue
Right now the form of the modified erf function that is used to fit s-curves has parameters that are hardcoded. These parameters should be changed from 500 to 0.5 * nInjections. See L#34 of fitScanData.py
However now the --nevt option can be passed from command line to the scan tool of interest. Right now the --nevt defaults to 1000 so there is no conflict. However if the user supplies a different value the fit may not work as planned.
This was also mentioned in:
cms-gem-daq-project/vfatqc-python-scripts#113
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
In L#1 fitScanData() should take an additional parameter as input: nInjections=1000. The default will ensure the same behavior as present.
Then the equation of Fit at L#34 should be modified from:
fitTF1 = r.TF1('myERF','500*TMath::Erf((TMath::Max([2],x)-[0])/(TMath::Sqrt(2)*[1]))+500',1,253)
to be instead:
fitTF1 = r.TF1('myERF','%i*TMath::Erf((TMath::Max([2],x)-[0])/(TMath::Sqrt(2)*[1]))+%i'%(0.5*nInjections),1,253)
Current Behavior
Currently if the default value of --nevt=1000 but if the user specifies a value outside of this the modified erf applied to the scurve data may not work as intended
Context (for feature requests)
Ensuring this ensures correct fitting when number of events in scurve/trimming differs from default
Feature Request: Change Default ztrim Value
Brief summary of issue
Default value of ztrim in this repository does not match default value in vfatqc-python-scripts/qcoptions.py. Suggest to resolve this mismatch.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
In vfatqc-python-scripts/qcoptions.py at L#13 and also in vfatqc-python-scripts/run_scans.py L#7 the default ztrim is 4.0.
However in anaoptions.py at L#22 the default is 0.0.
While the instructions to the shifter:
https://twiki.cern.ch/twiki/bin/view/CMS/GEMDOCDoc#How_to_Produce_Scan_Plots_Taken
State that a ztrim value should be given at cmd line when analyzing data to prevent potential mismatches suggest to change the default to 4 in anaoptions.py
Current Behavior
Default ztrim in anaoptions.py is 0. This might lead to a mismatch between data taken and data analyzed.
Context (for feature requests)
Forgetting to provide this value at the time of analysis may lead to fitting scurves improperly.
packageFiles4Docker.py is missing from the package
Brief summary of issue
The package gempython_gemplotting-1.0.0.tar.gz doesn't contain packageFiles4Docker.py. This file is referenced in the README instructions to analyze P5 data.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
After installing the package and setting up $PATH, packageFiles4Docker.py should be available.
Current Behavior
packageFiles4Docker.py isn't even present in the package.
Steps to Reproduce (for bugs)
- Install the
gemplottingpackage - Run
find path/to/your/venv | grep packageFiles4Docker.py - Nothing is found
Context (for feature requests)
Trying to get some data to analyze to tackle #95
Your Environment
- Version used: 1.0.0
- Shell used:
bash
vthr TTree Attribute error
Brief summary of issue
The ana_scans.py script terminates normally but reports a caught exception. The anaLog.log files shows that an AttributeError involving a TTree occurred.
Types of issue
Bug report
Expected Behavior
I have never successfully run the script before.
Current Behavior
[gemuser@gem904qc8daq gemdaq]$ ana_scans.py --scandate=current --ztrim=4 --anaType=scurve 2>&1 | tee -a ana_scurve_log.txt
Running jobs in parallel mode (using Pool(12))
Analysis Requested: scurve
Analysis Requested: scurve
Analysis Requested: scurve
Analysis Requested: scurve
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
[1, 1, 1, 1]
Normal termination
Results: [1, 1, 1, 1]
Caught non-Python Exception <type 'exceptions.SystemExit'>
[gemuser@gem904qc8daq gemdaq]$ cat /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-VII-S-CERN-0001/scurve/current/anaLog.log
mkdir: cannot create directory ‘/data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-VII-S-CERN-0001/scurve//current/SCurveData’: File exists
Analyzing: '/data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-VII-S-CERN-0001/scurve//current/SCurveData.root'
Traceback (most recent call last):
File "/opt/cmsgemos/bin/anaUltraScurve.py", line 298, in <module>
vthr_list[event.vfatN][event.vfatCH] = event.vthr
AttributeError: 'TTree' object has no attribute 'vthr'
Steps to Reproduce (for bugs)
- ssh to gemuser@gem904qc8daq from lxplus
- execute "ana_scans.py --scandate=current --ztrim=4 --anaType=scurve 2>&1 | tee -a ana_scurve_log.txt"
Possible Solution (for bugs)
I have no idea.
Your Environment
- Version used: Not sure
- Shell used: bash
Feature Request: Summary Plot from anaUltraLatency.py
Right now anaUltraLatency.py plots the data as 24 different plots instead of a single summary plot.
This makes it hard for the DOC or analyst to upload the data to the E-Log for an "at-a-glance" view.
Request this script makes a summary plot.
Add xdaq Latency Scan python routine to repo
The xdaq latency scan routine is currently sitting on the 904 NAS and is not tracked via subversion.
Suggest to incorporate the script into this repository.
Propose also to rename it following some sort of convention. The analysis scripts that deal with the python tools all start with:
anaUltra*
Analysis scripts that deal with scans taken with xdaq might start with:
anaXDAQ*
Bug Report: Outdated VFAT DAC Values Applied At Configuration
Brief summary of issue
VFAT DAC values stored in chamber_vfatDACSettings are outdated and do not match what is stored in L#21-26 of vfat_user_functions.py of cmsgemos
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
DAC Settings shown in L#21-26 of vfat_user_functions.py of cmsgemos should be written by default.
Current Behavior
Outdated VFAT DAC values are written.
Steps to Reproduce (for bugs)
- Configure chamber using
vfatqc-python-scripts/chamberInfo.py - Read registers using
vfat_info_uhal.py - Comment out all entries in
chamber_vfatDACSettingsofchamberInfo.py - Repeat steps 1 & 2
Possible Solution (for bugs)
Update values stored in chamber_vfatDACSettings to match with L#21-26 of vfat_user_functions.py of cmsgemos or remove entirely leaving only a commented out example for future (maybe preferred?).
Your Environment
- Version used: 36165f5 (gem-plotting-tools)
- Version used: 90ceec363e0ee1116f9271b163a875c60685cb1a (vfatqc-python-scripts)
- Shell used: zsh
Feature Request: 3x8 Grid Plots Ordered by Physical VFAT Position
Brief summary of issue
Right now the 3x8 plots that all our plotting software makes orders the VFATs as (from left-to-right):
- Row 1: VFAT0 -> VFAT7
- Row 2: VFAT8 -> VFAT15
- Row 3: VFAT16 -> VFAT23
However if you physically look at the detector you would have to "invert" this plot to match the physical location of the VFATs when facing the readout PCB. Suggest to change to:
- Row 1: VFAT16 -> VFAT23
- Row 2: VFAT8 -> VFAT15
- Row 3: VFAT0 -> VFAT7
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
The 3x8 grid plot more follows physical layout of chips on the detector when facing the readout PCB.
Current Behavior
The 3x8 grid plot is inverted w.r.t. the physical layout of chips on the detector when facing the readout PCB
Context (for feature requests)
It's going to make our lives much easier when mass production and quality control starts given that there will be inexperienced users. Having an "at a glance" representation that matches the physical layout will reduce complexity and potential errors in the long-term.
Feature Request: ana_scans.py uses **kwargs
Brief summary of issue
The ana_scans.py tool has exceptionally bloated arguments and could benefit from use of **kwargs.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Enhanced readability could be achieved by implementing **kwargs.
Current Behavior
Lot of arguments in the command leave room for mistakes, especially if trying to supply them in order rather than by keyword.
Context (for feature requests)
Maintenance.
Your Environment
- Version used:
develop - Shell used: any
Bug Report: gemSCurveAnaToolkit.py generates crash with ROOT v6.X or on cc7?
Brief summary of issue
gemSCurveAnaToolkit.py generates a crash when running on cc7 with ROOT v6.X. Suspect it's a difference between ROOT 5.34 and 6.X, unclear why.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Current Behavior
On gem904daq03
Works
[dorney@gem904daq03]~/scratch0/CMS_GEM/CMS_GEM_DAQ/gem-plotting-tools% gemSCurveAnaToolkit.py -i listOfScanDates_Scurve.txt -c -s30 -v1 --anaType=scurveAna --drawLeg
TClass::TClass:0: RuntimeWarning: no dictionary for class TF1Parameters is available
TStreamerInfo::BuildOld:0: RuntimeWarning: Cannot convert TF1::fParErrors from type:vector<double> to type:Double_t*, skip element
TStreamerInfo::BuildOld:0: RuntimeWarning: Cannot convert TF1::fParMin from type:vector<double> to type:Double_t*, skip element
TStreamerInfo::BuildOld:0: RuntimeWarning: Cannot convert TF1::fParMax from type:vector<double> to type:Double_t*, skip element
TStreamerInfo::BuildOld:0: RuntimeWarning: Cannot convert TF1::fSave from type:vector<double> to type:Double_t*, skip element
TStreamerInfo::BuildOld:0: RuntimeWarning: Cannot convert TF1::fParams from type:TF1Parameters* to type:Double_t*, skip element
Info in <TCanvas::Print>: png file canv_SCurveFitOverlay_VFAT1_Chan30.png has been created
eog canv_SCurveFitOverlay_GE11-X-S-CERN-0003_2018.01.17.14.29_VFAT1_Chan30.png
Your plot is stored in a TFile, to open it execute:
root /afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/data/gemelog/gemSCurveAnaToolkit.root
ROOT is 5.34/36
python is Python 2.6.6
On gem904qc8daq
[dorney@gem904qc8daq]~/scratch0/CMS_GEM/CMS_GEM_DAQ/gem-plotting-tools% gemSCurveAnaToolkit.py -i listOfScanDates_Scurve.txt -c -s30 -v1 --anaType=scurveAna --drawLeg
*** Break *** segmentation violation
===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0 0x00007fe88412bdbc in waitpid () from /lib64/libc.so.6
#1 0x00007fe8840aecc2 in do_system () from /lib64/libc.so.6
#2 0x00007fe87c3041a2 in TUnixSystem::StackTrace() () from /usr/lib64/root/libCore.so.6.10
#3 0x00007fe87c306b3c in TUnixSystem::DispatchSignals(ESignals) () from /usr/lib64/root/libCore.so.6.10
#4 <signal handler called>
#5 0x00007fe86f47289f in TF1::GetParameters() const () from /usr/lib64/root/libHist.so.6.10
#6 0x00007fe86f46904b in TF1::DoCreateHistogram(double, double, bool) () from /usr/lib64/root/libHist.so.6.10
#7 0x00007fe86f46889d in TF1::Paint(char const*) () from /usr/lib64/root/libHist.so.6.10
#8 0x00007fe86fbed5ac in TPad::Paint(char const*) () from /usr/lib64/root/libGpad.so.6.10.08
#9 0x00007fe86fbeb84f in TPad::Print(char const*, char const*) () from /usr/lib64/root/libGpad.so.6.10.08
#10 0x00007fe86fbec86f in TPad::SaveAs(char const*, char const*) const () from /usr/lib64/root/libGpad.so.6.10.08
#11 0x00007fe87ce79bd7 in FastCall(long, void*, void*, void*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#12 0x00007fe87ce7ea0f in PyROOT::TVoidExecutor::Execute(long, void*, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#13 0x00007fe87cea239d in PyROOT::TMethodHolder::CallSafe(void*, long, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#14 0x00007fe87cea0d29 in PyROOT::TMethodHolder::Execute(void*, long, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#15 0x00007fe87ce9ff1e in PyROOT::TMethodHolder::Call(PyROOT::ObjectProxy*&, _object*, _object*, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#16 0x00007fe87ce845ee in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#17 0x00007fe884da09a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#18 0x00007fe884e350f6 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#19 0x00007fe884e3befd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#20 0x00007fe884e393fc in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#21 0x00007fe884e3befd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#22 0x00007fe884e3c002 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#23 0x00007fe884e5543f in run_mod () from /lib64/libpython2.7.so.1.0
#24 0x00007fe884e565fe in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#25 0x00007fe884e57889 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
#26 0x00007fe884e68a3f in Py_Main () from /lib64/libpython2.7.so.1.0
#27 0x00007fe88408ec05 in __libc_start_main () from /lib64/libc.so.6
#28 0x000000000040071e in _start ()
===========================================================
The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum.
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5 0x00007fe86f47289f in TF1::GetParameters() const () from /usr/lib64/root/libHist.so.6.10
#6 0x00007fe86f46904b in TF1::DoCreateHistogram(double, double, bool) () from /usr/lib64/root/libHist.so.6.10
#7 0x00007fe86f46889d in TF1::Paint(char const*) () from /usr/lib64/root/libHist.so.6.10
#8 0x00007fe86fbed5ac in TPad::Paint(char const*) () from /usr/lib64/root/libGpad.so.6.10.08
#9 0x00007fe86fbeb84f in TPad::Print(char const*, char const*) () from /usr/lib64/root/libGpad.so.6.10.08
#10 0x00007fe86fbec86f in TPad::SaveAs(char const*, char const*) const () from /usr/lib64/root/libGpad.so.6.10.08
#11 0x00007fe87ce79bd7 in FastCall(long, void*, void*, void*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#12 0x00007fe87ce7ea0f in PyROOT::TVoidExecutor::Execute(long, void*, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#13 0x00007fe87cea239d in PyROOT::TMethodHolder::CallSafe(void*, long, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#14 0x00007fe87cea0d29 in PyROOT::TMethodHolder::Execute(void*, long, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#15 0x00007fe87ce9ff1e in PyROOT::TMethodHolder::Call(PyROOT::ObjectProxy*&, _object*, _object*, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#16 0x00007fe87ce845ee in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#17 0x00007fe884da09a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#18 0x00007fe884e350f6 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#19 0x00007fe884e3befd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#20 0x00007fe884e393fc in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#21 0x00007fe884e3befd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#22 0x00007fe884e3c002 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#23 0x00007fe884e5543f in run_mod () from /lib64/libpython2.7.so.1.0
#24 0x00007fe884e565fe in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#25 0x00007fe884e57889 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
#26 0x00007fe884e68a3f in Py_Main () from /lib64/libpython2.7.so.1.0
#27 0x00007fe88408ec05 in __libc_start_main () from /lib64/libc.so.6
#28 0x000000000040071e in _start ()
===========================================================
Traceback (most recent call last):
File "/afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/gem-plotting-tools/macros/gemSCurveAnaToolkit.py", line 90, in <module>
vfatChNotROBstr=options.channels
File "/afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/gem-plotting-tools/macros/scurvePlottingUtitilities.py", line 48, in overlay_scurve
canvas.SaveAs("%s.png"%(canvas.GetName()))
SystemError: void TPad::SaveAs(const char* filename = "", const char* option = "") =>
problem in C++; program state has been reset
*** Break *** segmentation violation
===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0 0x00007fe88412bdbc in waitpid () from /lib64/libc.so.6
#1 0x00007fe8840aecc2 in do_system () from /lib64/libc.so.6
#2 0x00007fe87c3041a2 in TUnixSystem::StackTrace() () from /usr/lib64/root/libCore.so.6.10
#3 0x00007fe87c306b3c in TUnixSystem::DispatchSignals(ESignals) () from /usr/lib64/root/libCore.so.6.10
#4 <signal handler called>
#5 0x0000000000000000 in ?? ()
#6 0x00007fe87c288aba in TList::Delete(char const*) () from /usr/lib64/root/libCore.so.6.10
#7 0x00007fe87c17a1c8 in TROOT::EndOfProcessCleanups() () from /usr/lib64/root/libCore.so.6.10
#8 0x00007fe87ce79bd7 in FastCall(long, void*, void*, void*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#9 0x00007fe87ce7ea0f in PyROOT::TVoidExecutor::Execute(long, void*, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#10 0x00007fe87cea239d in PyROOT::TMethodHolder::CallSafe(void*, long, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#11 0x00007fe87cea0d29 in PyROOT::TMethodHolder::Execute(void*, long, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#12 0x00007fe87ce9ff1e in PyROOT::TMethodHolder::Call(PyROOT::ObjectProxy*&, _object*, _object*, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#13 0x00007fe87ce845ee in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#14 0x00007fe884da09a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#15 0x00007fe884e350f6 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#16 0x00007fe884e3befd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#17 0x00007fe884dc594d in function_call () from /lib64/libpython2.7.so.1.0
#18 0x00007fe884da09a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#19 0x00007fe884e345bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#20 0x00007fe884e3befd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#21 0x00007fe884dc5858 in function_call () from /lib64/libpython2.7.so.1.0
#22 0x00007fe884da09a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#23 0x00007fe884e327b7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#24 0x00007fe884e57047 in Py_Finalize () from /lib64/libpython2.7.so.1.0
#25 0x00007fe884e68445 in Py_Main () from /lib64/libpython2.7.so.1.0
#26 0x00007fe88408ec05 in __libc_start_main () from /lib64/libc.so.6
#27 0x000000000040071e in _start ()
===========================================================
The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum.
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5 0x0000000000000000 in ?? ()
#6 0x00007fe87c288aba in TList::Delete(char const*) () from /usr/lib64/root/libCore.so.6.10
#7 0x00007fe87c17a1c8 in TROOT::EndOfProcessCleanups() () from /usr/lib64/root/libCore.so.6.10
#8 0x00007fe87ce79bd7 in FastCall(long, void*, void*, void*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#9 0x00007fe87ce7ea0f in PyROOT::TVoidExecutor::Execute(long, void*, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#10 0x00007fe87cea239d in PyROOT::TMethodHolder::CallSafe(void*, long, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#11 0x00007fe87cea0d29 in PyROOT::TMethodHolder::Execute(void*, long, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#12 0x00007fe87ce9ff1e in PyROOT::TMethodHolder::Call(PyROOT::ObjectProxy*&, _object*, _object*, PyROOT::TCallContext*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#13 0x00007fe87ce845ee in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) () from /usr/lib64/python2.7/site-packages/libPyROOT.so
#14 0x00007fe884da09a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#15 0x00007fe884e350f6 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#16 0x00007fe884e3befd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#17 0x00007fe884dc594d in function_call () from /lib64/libpython2.7.so.1.0
#18 0x00007fe884da09a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#19 0x00007fe884e345bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#20 0x00007fe884e3befd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#21 0x00007fe884dc5858 in function_call () from /lib64/libpython2.7.so.1.0
#22 0x00007fe884da09a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#23 0x00007fe884e327b7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#24 0x00007fe884e57047 in Py_Finalize () from /lib64/libpython2.7.so.1.0
#25 0x00007fe884e68445 in Py_Main () from /lib64/libpython2.7.so.1.0
#26 0x00007fe88408ec05 in __libc_start_main () from /lib64/libc.so.6
#27 0x000000000040071e in _start ()
===========================================================
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib64/python2.7/site-packages/ROOT.py", line 669, in cleanup
gROOT.EndOfProcessCleanups()
SystemError: void TROOT::EndOfProcessCleanups() =>
problem in C++; program state has been reset
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib64/python2.7/site-packages/ROOT.py", line 669, in cleanup
gROOT.EndOfProcessCleanups()
SystemError: void TROOT::EndOfProcessCleanups() =>
problem in C++; program state has been reset
ROOT is 6.10/08
Python is Python 2.7.5
Steps to Reproduce (for bugs)
- From new
cc7machine - Source gem-plotting-tools env
- Execute
gemSCurveAnaToolkit.py -i listOfScanDates_Scurve.txt -c -s30 -v1 --anaType=scurveAna --drawLeg
Contents of listOfScanDates_Scurve.txt is reported below:
ChamberName scandate
#GE11-VI-L-CERN-0002 current
#GE11-X-S-CERN-0001 current
GE11-X-S-CERN-0003 2018.01.17.14.29
Possible Solution (for bugs)
Probably something to do with how the TF1 object is stored/access in the TFile...in ROOT 5.34 it recognizes the TFile has been made with ROOT 6 and so just skips the TF1 (e.g. in ROOT 5.34 the class TF1Parameters does not exist).
However in ROOT 6 there's a crash.
Context (for feature requests)
Cannot investigate individual channel scurves and their fits.
Your Environment
- Version used: Probably all of them
- Shell used:
zsh
Feature Request: Update VFAT Channel to Strip Mapping for LS2 Detectors
Brief summary of issue
We require an update of the strip to vfat channel mapping for the LS2 detectors (and v3 electronics).
This would require updating:
- channelMaps.py, and
- buildMapFiles.py, and
- PanChannelMaps.py.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
We should provide an additional mapping files that can be used with the v3 electronics for long and short detectors.
For a starting point please see this post on the CMS GEM DAQ mattermost which provides a mapping spreadsheet obtained from Aamir and some follow-up questions that are outstanding.
Current Behavior
Presently the mapping only reflects v2b slice test era detector+electronics combinations.
Context (for feature requests)
Should accurately map strips to vfat channel number.
Feature Request: Add a ChipID TBranch To All Produced TTrees
Brief summary of issue
Toward the eventual goal of DB interface all produced TTree objects should have a ChipID branch which stores the hex VFAT as a type int as this will be properly interpreted when casting back to hex.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
All TTree objects have:
vfatID = array( 'i', [ 0 ] )
myT.Branch( 'vfatID', vfatID, 'vfatID/I' )
Which is later filled in the appropriate locations. For example the syntax would be:
vfatID[0] = getChipID(ohboard, options.gtx, vfat, options.debug)
Current Behavior
There is no way to track which VFAT hybrids were used with a given scan. This is especially important for test stands in which this could change more readily; but this would also be important for P5 (e.g. linking ADC calibration data for a given VFAT).
Context (for feature requests)
Towards the long-term goal of DB integration
Bug Report: make rpm fails to build the package on lxplus
Brief summary of issue
Even though lxplus isn't a supported development environment, some of the tools cannot be run on the P5 machines (eg because they require cluster access). Developing and testing on different machines would be painful.
make rpm crashes on lxplus.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
make rpm works on lxplus.
Current Behavior
As recommended, the following wasn't done in a virtualenv:
$ cd GEM/gem-plotting-tools/
$ LANG=C make && LANG=C make rpm
PythonModules=["gempython.gemplotting", "gempython.gemplotting.utils", "gempython.gemplotting.fitting", "gempython.gemplotting.macros", "gempython.gemplotting.mapping" ]
Using BUILD_HOME=/afs/cern.ch/user/l/lmoureau/GEM
OS Detected: slc6
mkdir -p pkg/gempython/gemplotting/utils
PythonModules=["gempython.gemplotting", "gempython.gemplotting.utils", "gempython.gemplotting.fitting", "gempython.gemplotting.macros", "gempython.gemplotting.mapping" ]
Using BUILD_HOME=/afs/cern.ch/user/l/lmoureau/GEM
OS Detected: slc6
mkdir -p pkg/gempython/gemplotting/utils
mkdir -p pkg/gempython/scripts
cp -rf README.md LICENSE CHANGELOG.md MANIFEST.in requirements.txt pkg/gempython/gemplotting
cp: cannot stat `CHANGELOG.md': No such file or directory
make: [preprpm] Error 1 (ignored)
cp -rf README.md LICENSE CHANGELOG.md MANIFEST.in requirements.txt pkg
cp: cannot stat `CHANGELOG.md': No such file or directory
make: [preprpm] Error 1 (ignored)
Running _setup_update target
PackagePath ackagePath
mkdir -p /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/RPMBUILD
if [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup.py ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup.py; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup.py /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/pkg/setup.py ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/pkg/setup.py; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/pkg/setup.py /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/setup.py ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/setup.py; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/setup.py /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/build/setup.py ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/build/setup.py; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/build/setup.py /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/config/setupTemplate.py ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/config/setupTemplate.py; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/config/setupTemplate.py /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/config/build/setupTemplate.py ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/config/build/setupTemplate.pyz; \
cp /afs/cern.ch/user/l/lmoureau/GEM/config/build/setupTemplate.py /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py; \
else \
echo Unable to find any setupTemplate.py; \
exit 1; \
fi
Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/pkg/setup.py
sed -i 's#__author__##' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__project__#gem-plotting-tools#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__summary__#None#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__gitrev__#a2464de#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__builddate__#01062018#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__package__#gemplotting#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__packagedir__#/afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__packagename__#gempython_gemplotting#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__longpackage__#gemplotting#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__pythonmodules__#["gempython.gemplotting", "gempython.gemplotting.utils", "gempython.gemplotting.fitting", "gempython.gemplotting.macros", "gempython.gemplotting.mapping" ]#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__version__#1.0.0#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__release__#slc6#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__prefix__##' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__os__#slc6#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__platform__#Linux-2.6.32-696.23.1.el6.x86_64-x86_64-with-redhat-6.9-Carbon#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
sed -i 's#__description__#None#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.py
if [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup.cfg ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup.cfg; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup.cfg /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/pkg/setup.cfg ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/pkg/setup.cfg; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/pkg/setup.cfg /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/setup.cfg ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/setup.cfg; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/setup.cfg /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/build/setup.cfg ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/build/setup.cfg; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/build/setup.cfg /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/config/setupTemplate.cfg ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/config/setupTemplate.cfg; \
cp /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/setup/config/setupTemplate.cfg /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg; \
elif [ -e /afs/cern.ch/user/l/lmoureau/GEM/config/build/setupTemplate.cfg ]; then \
echo Found /afs/cern.ch/user/l/lmoureau/GEM/config/setupTemplate.cfg; \
cp /afs/cern.ch/user/l/lmoureau/GEM/config/build/setupTemplate.cfg /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg; \
else \
echo Unable to find any setupTemplate.cfg; \
exit 1; \
fi
Found /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/pkg/setup.cfg
sed -i 's#__author__##' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__project__#gem-plotting-tools#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__summary__#None#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__gitrev__#a2464de#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__builddate__#01062018#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__package__#gemplotting#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__packagedir__#/afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__packagename__#gempython_gemplotting#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__longpackage__#gemplotting#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__pythonmodules__#["gempython.gemplotting", "gempython.gemplotting.utils", "gempython.gemplotting.fitting", "gempython.gemplotting.macros", "gempython.gemplotting.mapping" ]#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__version__#1.0.0#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__release__#slc6#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__prefix__##' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__os__#slc6#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__platform__#Linux-2.6.32-696.23.1.el6.x86_64-x86_64-with-redhat-6.9-Carbon#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
sed -i 's#__description__#None#' /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/setup.cfg
Running _rpmsetup target
cd pkg && \
find . -iname 'setup.*' -prune -o -name "*" -exec install -D \{\} /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm/\{\} \;
install: omitting directory `.'
install: omitting directory `./gempython'
install: omitting directory `./gempython/gemplotting'
install: omitting directory `./gempython/gemplotting/utils'
install: omitting directory `./gempython/gemplotting/macros'
install: omitting directory `./gempython/gemplotting/fitting'
install: omitting directory `./gempython/gemplotting/mapping'
install: omitting directory `./gempython/scripts'
Running _rpmbuild target
cd /afs/cern.ch/user/l/lmoureau/GEM/gem-plotting-tools/rpm && python setup.py bdist_rpm \
--release slc6.python2.6 \
--binary-only --force-arch=`uname -m`
Traceback (most recent call last):
File "setup.py", line 2, in <module>
from setuptools import setup,find_packages
File "/afs/cern.ch/user/l/lmoureau/.local/lib/python2.6/site-packages/setuptools/__init__.py", line 13, in <module>
from setuptools.extension import Extension
File "/afs/cern.ch/user/l/lmoureau/.local/lib/python2.6/site-packages/setuptools/extension.py", line 9, in <module>
from .monkey import get_unpatched
File "/afs/cern.ch/user/l/lmoureau/.local/lib/python2.6/site-packages/setuptools/monkey.py", line 10, in <module>
from importlib import import_module
ImportError: No module named importlib
make: *** [_rpmbuild] Error 1
Steps to Reproduce (for bugs)
See the above section.
Context (for feature requests)
Developing for #105. See also the Summary above.
Your Environment
- Version used:
develop - Shell used:
bash
Bug Report: ana_scans.py
http://cmsonline.cern.ch/cms-elog/999538
ana_scans.py has several bugs when being called:
runCommand(cmd, log) doesn't seems to generate the log file but doesn't call cmd given. Changing to runCommand(cmd) calls the cmd. Potential issue in "runCommand?"
Under anaType 'scurve' the cmd is improperly appended
When starting to build commands before the case structure in launchAnaArgs need to get the env variable for the path (e.g. GEM_PLOTTING_PROJECT) and then use this to call the command, e.g.:
cmdPath = os.getenv('GEM_PLOTTING_PROJECT')
cmd = ["python", "%s/%s"%(cmdPath,ana_config[anaType])]
When tool=anaUltraThreshold.py a the depreciated option of --vfatmask was called leading to an exception.
Feature Requests: Unit Tests
Brief summary of issue
In the past week we have let PR's be merged without either proper testing or proper oversight (myself included) as demonstrated by:
Which were either not properly tested originally, or not re-tested after committing on top of a request. To prevent lost time, and most importantly p5 machine being updated with bad code we should finalize a procedure and develop unit tests which should cross-check things. I propose:
- Each PR should be supported with an e-log link that shows tests were properly done.
- New commits after changes are requested should include an e-log link where tests have been repeated and properly passed.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
I would propose the following:
- For changes to
ana_scans.pyfor each--anaTypea re-analysis of old data should be launched - For changes to
anaUltra*.pyscripts see the first item, - For changes to
anaXDAQ*.pyscripts see the first item, - For changes to files in
fitting/ormapping/sub-directories taking a run ofvfatqc-python-scripts/ultraScurve.pyand then analyzing the data immediately
For items 1-3 something like scripts/unitTests_Ana_Ultra.sh script like:
#!/bin/bash
# record branch name and commit hash
cd $GEM_PLOTTING_PROJECT
git status 2>&1 | tee -a unit_test_log.txt
git show 2>&1 | tee -a unit_test_log.txt
# perform tests
ana_scans.py --scandate=current --anaType=latency 2>&1 | tee -a unit_test_log.txt
sleep 2
ana_scans.py --scandate=current --anaType=scurve 2>&1 | tee -a unit_test_log.txt
sleep 2
ana_scans.py --scandate=current --anaType=threshold 2>&1 | tee -a unit_test_log.txt
sleep 2
ana_scans.py --scandate=current --anaType=trim 2>&1 | tee -a unit_test_log.txt
# Done
echo "Unit Tests Completed, Upload:"
echo $PWD/unit_test_log.txt
echo "To the E-Log or Github PR along with the images in:"
echo $ELOG_PATH
With a similar script e.g. scripts/unitTests_Ana_XDAQ.sh
These scripts should be launched on a test stand for at least 1 link in chamber_config of mapping/chamberInfo.py
For item 4 a script like scripts/unitTests_Run_SCurve.sh:
#!/bin/bash
# record branch name and commit hash
cd $GEM_PLOTTING_PROJECT
git status 2>&1 | tee -a unit_test_log.txt
git show 2>&1 | tee -a unit_test_log.txt
# perform tests
run_scans.py --tool=ultraScurve.py --shelf=$1 -s$2 2>&1 | tee -a scurve_log.txt
sleep 2
ana_scans.py --scandate=current --anaType=scurve 2>&1 | tee -a unit_test_log.txt
sleep 2
# Done
echo "Unit Tests Completed, Upload:"
echo $PWD/unit_test_log.txt
echo "To the E-Log or Github PR along with the images in:"
echo $ELOG_PATH
With syntax:
./scripts/unitTests_Run_SCurve.sh <Shelf> <Slot>
Current Behavior
We lack an appropriate unit test procedure.
Context (for feature requests)
We've committed things without proper foresight and testing. This should at least make the developer go through a procedure to check for problems.
ana_scans.py doesn't fail if one of the subcommand fails
Brief summary of issue
ana_scans.py invokes other commands such as anaUltraScurve.py. However, ana_scans.py doesn't notice when one of those commands fails, and will happily keep running. ana_scans.py will exit with status code zero even if somewhat went horribly wrong. It should instead follow the usual conventions and return a non-zero status code.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
If one of the subcommands fails, I expect ana_scans.py to at least report a non-zero status code.
Current Behavior
The status code is always zero (except maybe when killed).
Context (for feature requests)
Was trying to use ana_scans.py to setup unit tests on Travis (see #27).
Your Environment
"Bug" Report: Error Bars on anaUltra*.py Scripts
Brief summary of issue
We've migrated all our DAQ (and I guess also DQM?) machines to cc7 which defaults to ROOT v6.X. In this version of ROOT by default TH1::Sumw2() is automatically set, see the TH1 Class Documentation. This causes error bars of some of our histograms to not be set correctly.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
For a counting experiment (almost all of our scan routines) the error bar should go as sqrt(N) where N is the bin content.
Current Behavior
Due to the differences between ROOT 6.X and 5.34 the TH1::Sumw2() method is automatically called.
Our analysis routines usually use the Int_t TH1::Fill(Double_t x,Double_t w ) method for storing values in the histogram where x and w come from the input TTree object. However this is only being called once per bin so the error bar ends up equalling w e.g. the sum of weights squared is a sum over one number.
See for example:
Steps to Reproduce (for bugs)
- Run any anaUltra script on a machine where ROOT is v6.X or higher.
- Look at the error bars on the produced plots
Possible Solution (for bugs)
Proposal 1
Call TH1::SetDefaultSumw2(False) at the start of each script. Although this is a static ROOT function and I'm not sure how that will play in python.
If it works this is the simplest fix.
Proposal 2
Use event.indepVar to determine the corresponding bin in the histogram, e.g.
for idx in range(1, Histo.GetNbinsX()+1):
if (Histo.GetBinLowEdge(1) + idx*Histo.GetBinLowEdge(1) ) <= event.indepVar and indepVar < (Histo.GetBinLowEdge(1) + (idx+1)*Histo.GetBinLowEdge(1) ):
Histo.SetBinContent(idx, event.depVar)
Disadvantage is that this is pretty slow...
Proposal 3
Change all TH1 objects to TGraph objects (or inherited classes, e.g. TGraphErrors or TGraphAsymErrors) and use the TGraph::SetPoint(...) method for plotting from the TTree.
If you draw the TGraph object you could then use the TGraph::GetHistogram() method to return a TH1 if it's absolutely necessary.
Context (for feature requests)
This is particularly a problem when we are fitting distributions since the error bars on the data points will be accounted in the fitting and having each point as N +/- N is probably not going to give good fit results.
Your Environment
- Version used: I think any commit will have this problem
- Shell used:
zsh
missing shortChannelMap.txt error
Brief summary of issue
ana_scans.py script terminates normally but reports a python error.
Types of issue
Bug report
Expected Behavior
I have never successfully run the script before.
Current Behavior
[gemuser@gem904qc8daq gemdaq]$ ana_scans.py --scandate=current --ztrim=4 --anaType=scurve 2>&1 | tee -a ana_scurve_log.txt
Running jobs in parallel mode (using Pool(12))
Analysis Requested: scurve
Analysis Requested: scurve
Analysis Requested: scurve
Analysis Requested: scurve
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
[1, 1, 1, 1]
Normal termination
Results: [1, 1, 1, 1]
Caught non-Python Exception <type 'exceptions.SystemExit'>
$ cat /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-VII-S-CERN-0001/scurve/current/anaLog.log
Analyzing: '/data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-VII-S-CERN-0001/scurve//current/SCurveData.root'
Exception: [Errno 2] No such file or directory: '/usr/lib/python2.7/site-packages/gempython/gemplotting/mapping//shortChannelMap.txt'
Failed to open: '/usr/lib/python2.7/site-packages/gempython/gemplotting/mapping//shortChannelMap.txt'
Traceback (most recent call last):
File "/opt/cmsgemos/bin/anaUltraScurve.py", line 277, in <module>
dict_vfatChanLUT = getMapping(MAPPING_PATH+'/shortChannelMap.txt')
File "/usr/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 142, in getMapping
mapFile.close()
UnboundLocalError: local variable 'mapFile' referenced before assignment
Steps to Reproduce (for bugs)
- ssh to gemuser@gem904qc8daq from lxplus
- execute "ana_scans.py --scandate=current --ztrim=4 --anaType=scurve 2>&1 | tee -a ana_scurve_log.txt"
Possible Solution (for bugs)
I have no idea.
Context (for feature requests)
This issue is preventing me from analyzing s-curve calibration data.
Your Environment
- Version used: Not sure
- Shell used: bash
Feature Request: Identifying Original maskReason & Date Range of Burned Input
Brief summary of issue
We have 30720 channels in CMS presently. And we need some automated way for tracking when changes occur. Ideally this should be via the DB but this is not up and running (yet...). But we have an automated procedure to make time series plots from the entire calibration dataset, see #87.
So my proposal is to use these time series plots to identify:
- The original
maskReasonthat is applied to a channel, - The smallest date range for when a channel made the switch from "healthy" to burned, e.g. for this example that would be
2017.05.10.20.41to2017.05.31.09.21.
Issue with Channel Mask
Right now when a channel is masked it's maskReason will be recorded in the output TTree of the analysis scan. However once a channel is masked the history will be somewhat "lost." This is because the channel will be masked at the time of acquisition so the s-curve analysis will see that a fit to it fails (because there is no data). The maskReason will then be assigned FitFailed. This is because the scan software does not know the history.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
To accomplish this a secondary analysis tool should be made, macros/timeHistoryAnalyzer.py, which should should have a function like:
def analyzeTimeSeriesPlot(timeSeriesHisto, obsType, deadChanCutLow=4.14E-02, deadChanCutHigh=1.09E-01):
"""
timeSeriesHisto - TH2 object created by gemPlotter.py when making time series plots
obsType - The observable to be investigate from the set {noise, ..., maskReason, etc...} see for example the observables produced here https://github.com/cms-gem-daq-project/gem-plotting-tools/blob/0ce86672ddac3affee27b80517eb34e8cd50b029/macros/plotTimeSeries.py#L10-L26
deadChanCutLow - lower bound on s-curve width for identifying a dead channel, value in fC
deadChanCutHigh - higher bound on s-curve width for identifying a dead channel, value in fC
"""
# import numpy
# import nesteddict from gem sw
chanArray = nesteddict()
# Load info from the histo
for binY in timeSeriesHisto:
lastGoodScanDate = 2017.01.01
firstDateAfterBurn = None
for bin X in timSeriesHisto:
# get the point value and store it
scanDate = timSeriesHisto.GetBinLabel(binX)
chanStripOrPanPin = binY-1 #bins go from 1 to nbins but channel/strips/panpin go 0 to 127
scanVal = timSeriesHisto.GetBinContent(binX,binY)
# See if channel went from bad to good
if obsType == "noise":
if scanVal > deadChanCutHigh:
lastGoodScanDate = scanDate
if deadChanCutLow <= scanVal and scanVal <= deadChanCutHigh:
if firstDateAfterBurn is not None:
firstDateAfterBurn = scanDate
chanArray[chanStripOrPanPin] = (lastGoodScanDate, firstDateAfterBurn)
# Try to determine original mask reason
if obsType == "maskReason":
# Similar to above example
# Display to user
print "| chan | lastGoodScanDate | firstDateAfterBurn |" # in markdown format
print "| :---: | :------------------: | :-----------------: |"
for chan in range(0,len(chanArray)):
print "| %i | %s | %s |"%(chan, chanArray[chan][0], chanArray[chan][1])
# Similarly display a table which shows the number of channels in the VFAT that have the following maskReasons based on the "original" maskReason:
#NotMasked = 0x0
#HotChannel = 0x01
#FitFailed = 0x02
#DeadChannel = 0x04
#HighNoise = 0x08
#HighEffPed = 0x10
Caveats is that this needs to be modified to not be sensitive to transient effects; e.g. in this example two scans failed to complete successfully so the data is missing and the channels and this may throw off the algorithm if this were "maskReason" case. Testing would be required
Current Behavior
Presently the last good scandate and first date after burned channel need to be determined by hand. Also the FitFailed maskReason slowly dominates the dataset.
Context (for feature requests)
We need to be able to accurately report our channel status and history to ourselves and CMS
missing mapping.chamberInfo module
Brief summary of issue
The ana_scans.py script terminates normally but reports a caught exception. The anaLog.log file shows that an ImportError involving mapping.chamberInfo module occurred.
Types of issue
Bug report
Expected Behavior
I have never successfully run the script before.
Current Behavior
[gemuser@gem904qc8daq gemdaq]$ ana_scans.py --scandate=current --ztrim=4 --anaType=scurve 2>&1 | tee -a ana_scurve_log.txt
Running jobs in parallel mode (using Pool(12))
Analysis Requested: scurve
Analysis Requested: scurve
Analysis Requested: scurve
Analysis Requested: scurve
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
[1, 1, 1, 1]
Normal termination
Results: [1, 1, 1, 1]
Caught non-Python Exception <type 'exceptions.SystemExit'>
[gemuser@gem904qc8daq current]$ cat /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-VII-S-CERN-0001/scurve/current/anaLog.log | tail -n8
Traceback (most recent call last):
File "/opt/cmsgemos/bin/anaUltraScurve.py", line 759, in <module>
saveSummary(vSummaryPlots, None, '%s/Summary.png'%filename, trimVcal)
File "/usr/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 562, in saveSummary
from mapping.chamberInfo import chamber_vfatPos2PadIdx
File "/usr/lib64/python2.7/site-packages/ROOT.py", line 318, in _importhook
return _orig_ihook( name, *args, **kwds )
ImportError: No module named mapping.chamberInfo
Steps to Reproduce (for bugs)
- ssh to gemuser@gem904qc8daq from lxplus
- execute "ana_scans.py --scandate=current --ztrim=4 --anaType=scurve 2>&1 | tee -a ana_scurve_log.txt"
Possible Solution (for bugs)
I have no idea.
Your Environment
- Version used: Not sure
- Shell used: bash
Feature Request: wrapper by scandate for TTree::Draw()
Brief summary of issue
Right now gemPlotter.py will let you plot an independent variable against a branch name for a list of scandates from our produced TTrees.
Would like to a script that does a similar function as gemPlotter.py, called treeDrawWrapper.py or similar, where it will make a plot using the TTree::Draw() method from command line for a list of scandates, from each scandate. Somethinglike
treeDrawWrapper.py -iListOfScanDates.txt --branchNameX=<Name> --branchNameY=<Name> --cut="mask!=0"
This should support the full functionality of the TTree::Draw() method. It should make both an image of the plot and safe the plot as the appropriate TObject in an output TFile. Maybe it could also print a table of the values stored in the plot (but not necessary).
This should go in macros/
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Automate the making of plots from a given scandate.
Current Behavior
Right now someone has to write a script for the given plot they want, or by hand go through the list of scandates and make the plot using the viewer in ROOT.
Context (for feature requests)
Quality of life improvement.
Feature Request: Generic Plotter for TObjects Stored in TTree
Brief summary of issue
Provide a generic tool for plotting TObjects stored in TTree's similiarly to how gemTreeDrawWrapper.py works.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Allow for automatic plotting of multiple scandates from command line for TObjects stored in TTrees. The syntax/behavior should be almost identical to how gemTreeDrawWrapper.py works.
Current Behavior
Plots need to be made by hand.
Context (for feature requests)
Allow tools for DAQ experts and the DOC for analyzing data.
Feature Request: anaUltraLatency.py Produces a Plot Summing Over All VFATs
Brief summary of issue
Right now anaUltraLatency.py produces a latency distribution for each VFAT. In addition to this would like one plot/histogram that represents the sum of all VFATs.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
New histogram is created that is:
hHitsVsLat_AllVFATs = dict_hVFATHitsVsLat[0].Clone("hHitsVsLat_AllVFATs")
for vfat in range(1,24):
hHitsVsLat_AllVFATs.Add(dict_hVFATHitsVsLat[vfat])
Then hHitsVsLat_AllVFATs should be stored in the output TFile and also plotted on a TCanvas and saved as a *.png file.
Current Behavior
If you want a summary plot you need to make it manually.
Context (for feature requests)
Quality of life improvement.
Feature Request: Algorithmic Identification of Burned/Disconnected VFAT2 Inputs
Brief summary of issue
Based on slide 22 vfat2 channels that have been burned during operation have a characteristic s-curve width. This s-curve width could be used to mark the channel as having a "burned input" and thus could be factored into our "% active channels" list when reporting to CMS.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
The maskReason class of anaInfo.py is updated to have a new field DisconnectedInput (it could also be called BurnedInput but we would need to know a priori if the channel was always like this or was assigned this classifier after some time). This could be added:
Lines 41 to 54 in 6445dcb
The hex assignment could be BurnedInput = 0x20 which would be 0b100000. This keeps with the standard of the maskReason (maybe we don't mask these but we have a human readable identifier) class where each reason corresponds to a bit in the overall maskReason value (recall channels can have more than one maskReason).
Then in anaUltraScurve.py the "create reason array" block could be updated to search for channels with this behavior:
gem-plotting-tools/anaUltraScurve.py
Lines 361 to 376 in 6445dcb
Here the solution would be something like:
reason[4.14E-02 < scanFitResults[1][vfat] and scanFitResults[1][vfat] < 1.09E-01] = MaskReason.DisconnectInput
Here the 4.14e-2 comes from the lower bound of the high purity burned sample and 1.09e-1 comes from the upper bound of the literature value. Numbers taken from slide 22. However these cut values could be influenced based on the result of #82.
Current Behavior
Presently channels which have become burned, or were always disconnected, are erroneously included in our "live channel fraction" when reporting to CMS.
Context (for feature requests)
Would like to correctly report our "live channel fraction" and monitor the progression of "burned inputs" over time in P5.
Your Environment
- Version used:
develop - Shell used: any
Structure of RPM package
Information
- The packages and modules will be packaged and end up in the system
PYTHONPATH, e.g., (defaultpythononcc7):/usr/lib/python2.7/site-packages/`- I've been toying with adding a global namespace
gempython, under which there would be thegemplotting,vftaqc, etc., modules, but this is still to be finalized into a proposal, and then discussed - This would necessarily involve a small redesign of the
importstatements, but I have for the most part decided on a workable solution for people who are doing private development and not using the production software
- I've been toying with adding a global namespace
- Executable scripts, e.g.,
amc_info_uhal.pywill all go into/opt/cmsgemos/bin, to make it simple to add this to thePATHvariable and have the script ready
Current RPM structure (UPDATED 12-03-2018)
gempython_gemplotting /opt/cmsgemos/bin/anaInfo.py
gempython_gemplotting /opt/cmsgemos/bin/anaUltraLatency.py
gempython_gemplotting /opt/cmsgemos/bin/anaUltraScurve.py
gempython_gemplotting /opt/cmsgemos/bin/anaUltraThreshold.py
gempython_gemplotting /opt/cmsgemos/bin/anaXDAQLatency.py
gempython_gemplotting /opt/cmsgemos/bin/ana_scans.py
gempython_gemplotting /opt/cmsgemos/bin/anaoptions.py
gempython_gemplotting /opt/cmsgemos/bin/anautilities.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/__init__.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/__init__.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/__init__.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/__init__.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/__init__.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/__init__.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/__init__.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/__init__.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/__init__.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/fitScanData.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/fitScanData.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/fitScanData.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/__init__.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/__init__.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/__init__.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemPlotter.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemPlotter.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemPlotter.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemSCurveAnaToolkit.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemSCurveAnaToolkit.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemSCurveAnaToolkit.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemTreeDrawWrapper.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemTreeDrawWrapper.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemTreeDrawWrapper.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/overlay_fit.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/overlay_fit.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/overlay_fit.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_eff.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_eff.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_eff.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_noise_vs_trim.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_noise_vs_trim.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_noise_vs_trim.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_scurves_by_thresh.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_scurves_by_thresh.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_scurves_by_thresh.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_and_channel_Scurve.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_and_channel_Scurve.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_and_channel_Scurve.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_summary.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_summary.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_summary.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plotoptions.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plotoptions.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plotoptions.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/scurvePlottingUtitilities.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/scurvePlottingUtitilities.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/scurvePlottingUtitilities.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/summary_plots.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/summary_plots.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/summary_plots.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/PanChannelMaps.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/PanChannelMaps.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/PanChannelMaps.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/__init__.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/__init__.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/__init__.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/buildMapFiles.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/buildMapFiles.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/buildMapFiles.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/chamberInfo.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/chamberInfo.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/chamberInfo.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/channelMaps.py
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/channelMaps.pyc
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/channelMaps.pyo
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython/gemplotting/requirements.txt
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/PKG-INFO
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/SOURCES.txt
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/dependency_links.txt
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/not-zip-safe
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/requires.txt
gempython_gemplotting /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/top_level.txt
Current tarball structure
gempython_gemplotting-1.0.1/
gempython_gemplotting-1.0.1/gempython/
gempython_gemplotting-1.0.1/gempython/gemplotting/
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaInfo.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaUltraLatency.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaUltraScurve.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaUltraThreshold.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaXDAQLatency.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/ana_scans.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaoptions.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anautilities.py
gempython_gemplotting-1.0.1/gempython/gemplotting/fitting/
gempython_gemplotting-1.0.1/gempython/gemplotting/fitting/__init__.py
gempython_gemplotting-1.0.1/gempython/gemplotting/fitting/fitScanData.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/__init__.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/gemPlotter.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/gemSCurveAnaToolkit.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/gemTreeDrawWrapper.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/overlay_fit.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_eff.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_noise_vs_trim.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_scurves_by_thresh.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_vfat_and_channel_Scurve.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_vfat_summary.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plotoptions.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/scurvePlottingUtitilities.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/summary_plots.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/PanChannelMaps.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/__init__.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/buildMapFiles.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/chamberInfo.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/channelMaps.py
gempython_gemplotting-1.0.1/gempython/gemplotting/__init__.py
gempython_gemplotting-1.0.1/gempython/gemplotting/requirements.txt
gempython_gemplotting-1.0.1/gempython/__init__.py
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/PKG-INFO
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/SOURCES.txt
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/dependency_links.txt
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/not-zip-safe
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/requires.txt
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/top_level.txt
gempython_gemplotting-1.0.1/README.md
gempython_gemplotting-1.0.1/setup.cfg
gempython_gemplotting-1.0.1/setup.py
gempython_gemplotting-1.0.1/PKG-INFO
Docs migration
Brief summary of issue
Current README needs to be refactored into a proper docs style
README should have info about getting set up, links to contributing guidelines, links to full documentation
Types of issue
- Cleanup
Possible Solution (for bugs)
- Create
docsfolder - Migrate documentation portion of README into this, preferably using rst for compatibility with
sphinx/readthedocsstyle documentation
Feature Request: Link Functionality to mapping/chamberInfo.py
Brief summary of issue
We have two test stands and P5. Right now mapping/chamberInfo.py has commented lines for the two test stands and uncommented lines for P5. When using a different test stand the user has to change these lines to match.
Suggest instead to implement the same idea as the connections file that we have in cmsgemos, like:
% ls -lh mapping/chamberInfo*
mapping/chamberInfo_904.py
mapping/chamberInfo_p5.py
mapping/chamberInfo.py -> chamberInfo_p5.py
mapping/chamberInfo.pyc
mapping/chamberInfo_qc8.py
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
We should have one symlink for mapping/chamberInfo.py which defaults to be the p5 file chamberInfo_p5.py and then two additional files:
- GEM-CSC Integration stand:
chamberInfo_904.py - Cosmic Stand:
chamberInfo_qc8.py
Then if you are using one of these two stands you just have to change where the symlink points to rather than continuously editing the mapping/chamberInfo.py file.
Current Behavior
Right now you have to continually edit the mapping/chamberInfo.py file when not at P5 or when switching test stands.
Context (for feature requests)
I have to always edit the mapping/chamberInfo.py file and it is annoying and leads to lost time/mistakes.
Bug Report: plotTimeSeries.py does not exit on KeyboardInterrupt
Brief summary of issue
Pressing Ctrl+C (e.g. KeyboardInterrupt) does not cause plotTimeSeries.py to terminate during execution.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Issuing a KeyboardInterrupt during execution of plotTimeSeries.py should cause the program to stop.
Current Behavior
Program persists through additional commands.
Steps to Reproduce (for bugs)
- Launch
plotTimeSeries.pyfollowing instructions shown here - Issue a
KeyboardInterrupte.g. pressCtrl+C
Possible Solution (for bugs)
Most likely the issue comes from successive uses of os.system( ... ) here:
gem-plotting-tools/macros/plotTimeSeries.py
Lines 10 to 26 in 0ce8667
The Ctrl+C seems to exit one of the os.system(...) calls (since this is a separate process) but plotTimeSeries.py continues.
I tried putting Lines 10-26 inside a try block to catch for KeyboardInterrupt and then exit similarly to how ana_scans.py terminates on KeyboardInterrupt:
gem-plotting-tools/ana_scans.py
Lines 259 to 262 in 0ce8667
but this was unsuccessful.
Context (for feature requests)
Would just prefer not having to spam Ctrl+C if I realized:
- I made a mistake after execution, or
- Want to stop.
Your Environment
- Version used: 0ce8667
- Shell used:
/bin/zsh
Issue: plotTimeSeries.py is not described in README.md
Brief summary of issue
I didn't see any documentation on for plotTimeSeries.py in the README.md or on the DOC Twiki page. Ideally this should go here in the README.md so we don't lose track of it.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
README.md includes some explanation about the command line arguments and an example of the input file and usage.
Current Behavior
No documentation in the README.md
Context (for feature requests)
Would helps the DOC to generate weekly calibration plots.
Discussion: VFAT2 VCal to fC Appears Off
Brief summary of issue
Based on the results shown on slide 22 the conversion of VCal to fC for vfat2 appears to be off by 25 to 34%. This is hardcoded in the function parseCalFile(...) of anautilities.py:
gem-plotting-tools/anautilities.py
Lines 368 to 370 in 6445dcb
If no calFile is supplied the default that will be returned is the vfat2 values that are hardcoded.
I would suggest we review the VCal to fC conversion for vfat2. Additionally instead of applying an aggregate value for all VFAT positions for s-curves taken at P5 I think we could switch to a per-vfat level since in develop anaUltraScurve.py features the option to supply an external calibration file which provides this conversion at the per VFAT level.
Types of issue
- Discussion
Expected Behavior
Naively I would expect channels that do not see the input capacitance of the detector would have s-curve widths consistent with published literature.
Current Behavior
There is a 25-34% discrepancy in the ENC determined for channels which are not connected to a detector input when compared to the value shown in literature.
Context (for feature requests)
This conversion is necessary for evaluating:
- algorithmically identifying burned VFAT2 inputs at P5,
- determining signal-to-noise ratios for MIPs
Your Environment
- Version used:
develop - Shell used: any
Fitting is broken
Fitting was broken in commit bf5d7f2.
Brief summary of issue
[...]
File "/afs/cern.ch/user/l/lmoureau/CosmicStandSw/gem-plotting-tools/fitting/fitScanData.py", line 58, in fit
fitTF1 = r.TF1('myERF','%f*TMath::Erf((TMath::Max([2],x)-[0])/(TMath::Sqrt(2)*[1]))+%f'%(self.Nev/2.),1,253)
TypeError: not all arguments converted during string formatting
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
No crash
Current Behavior
Crashes
Steps to Reproduce
- Try to fit S-curves using
anaUltraScurveortrimChamber
Your Environment
- Version used: master
- Shell used: bash
anaUltraScurve RuntimeWarning in ROOT 6 (Python 2.7.5 & Numpy 1.14.0)
Brief summary of issue
When testing PR #70 the output of running the scurve analyser produced a couple warnings in ROOT 6 with Python 2.7.5 & Numpy 1.14.0 which were not present in ROOT 5 with Python 2.6.6 & Numpy 1.11.3:
/afs/cern.ch/user/b/benjamin/gemdaq/gem-plotting-tools/anaUltraScurve.py:287: RuntimeWarning: invalid value encountered in greater
reason[effectivePedestals[vfat] > 50] |= MaskReason.HighEffPed
/afs/cern.ch/user/b/benjamin/gemdaq/gem-plotting-tools/anaUltraScurve.py:295: RuntimeWarning: invalid value encountered in greater
np.count_nonzero(effectivePedestals[vfat] > 50))
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
That the warnings do not appear
Current Behavior
Warnings appear - however the final output looks to be ok.
Steps to Reproduce (for bugs)
- Use ROOT 6, eg on gem904qc8daq
- Run:
anaUltraScurve.py -i /afs/cern.ch/user/b/benjamin/public/GEM/Data/SCurveTestFile.root --type=short -f - Watch the text output
Possible Solution (for bugs)
Context (for feature requests)
Your Environment
- Version used: ROOT 6 (Python 2.7.5 & Numpy 1.14.0) on gem904qc8daq
- Shell used: bash
Move Analysis Tools from vfatqc-python-scripts to this repo
Based on discussion we had in GEM Operations & Online SW Meeting on July 18th we decided to migrate all analysis based tools from vfatqc-python-scripts to this repo.
This impacts issues:
cms-gem-daq-project/vfatqc-python-scripts#84
cms-gem-daq-project/vfatqc-python-scripts#97
cms-gem-daq-project/vfatqc-python-scripts#99
cms-gem-daq-project/vfatqc-python-scripts#101
And also pull request:
cms-gem-daq-project/vfatqc-python-scripts#98
Bug Report: anaUltraLatency.py Does Not Use TDirectories Correctly
Brief summary of issue
Looking at the produced ROOT file latencyAna.root that anaUltraLatency.py produces the TObjects that are produced should be stored in TDirectories with names VFATN for N in [0,23]. But instead the TObjects are stored outside of the directories.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
TObjects for VFAT N should be stored in the VFATN TDirectory.
Current Behavior
TObjects for VFAT N are not stored in the VFATN TDirectory
Steps to Reproduce (for bugs)
- Take a latency scan with
vfatqc-python-scripts/ultraLatency.py - Analyze the output with
anaUltraLatency.py - Open the produced file in
ROOT
Possible Solution (for bugs)
At Line 98 a TDirectory is created but not called as the active directory. Change this from:
outF.mkdir("VFAT%i"%vfat)
to:
dirVFAT = outF.mkdir("VFAT%i"%vfat)
dirVFAT.cd()
Your Environment
- Version used: def88f9
- Shell used: /bin/bash
Bug Report: Problems generating full list of scan dates for plotTimeSeries.py
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
When I type the following command:
python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01
for each detector defined in chamber_config.values() the listOfScanDAtes.txt file should be found at $DATA_PATH//scurve/
Current Behavior
Instead, it halts the process and outputs the following error:
Traceback (most recent call last):
File "plotTimeSeries.py", line 26, in
from gempython.utils.wrappers import envCheck
ImportError: No module named gempython.utils.wrappers
Steps to Reproduce (for bugs)
- python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01
Possible Solution (for bugs)
Context (for feature requests)
I am trying to generate the full list of scan dates since 2017 installation for the various detectors to study the presence of burned VFAT inputs in time-series SCurve plots.
Your Environment
- Version used:
- Shell used: Mac terminal
anaUltraScurve doesn't take the -o option into account
anaUltraScurve.py has a -o option to specify the output filename, but will always use the default.
Brief summary of issue
The -o option is defined in anaoptions.py and not used by anaUltraScurve. It can be useful e.g. when willing to compare S-curve analysis at different z-score or when debugging changes to anaUltraScurve itself.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Passing a value to the -o option should change the output filename, or the -o option shouldn't appear in the help.
Current Behavior
Steps to Reproduce (for bugs)
- Take an S-curve
- Analyze the data using
-o test - The analysis output is written to directory
SCurveData/
Possible Solution (for bugs)
When specified, let -o override the output directory.
Context (for feature requests)
Was trying to find out if my code changes caused some weird behaviour in http://cmsonline.cern.ch/cms-elog/1005623 by comparing the anaUltraScurve output with and without my changes.
Your Environment
- Version used: master and custom
- Shell used: bash
Bug Report: parseCalFile() has issue with root_numpy...?
Brief summary of issue
Parsing of a calibration file causes an exception. This was not an issue pre-packaging so I suspect perhaps there is still an issue with how root_numpy is installed on the machine.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Parsing a calibration file should not cause an exception.
Current Behavior
(py2.7) [dorney@gem904qc8daq]/data/bigdisk/GEM-Data-Taking/GE11_QC8/GE11-X-S-CERN-0003/scurve% python
Python 2.7.5 (default, Apr 11 2018, 07:36:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from gempython.gemplotting.utils.anautilities import parseCalFile
>>> parseCalFile("/data/bigdisk/GEM-Data-Taking/GE11_QC8/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/cc7/py2.7/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 429, in parseCalFile
array_CalData = rp.tree2array(tree=calTree, branches=list_bNames)
AttributeError: 'module' object has no attribute 'tree2array'
Doesn't throw an import error which is strange. The tree2array function was definitely in 4.7.2 my venv has 4.7.3 installed:
(py2.7) % which pip
~/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/cc7/py2.7/bin/pip
(py2.7) % pip list | grep "root-numpy"
root-numpy 4.7.3
(py2.7) % pip list --user | grep "root-numpy"
(py2.7) %
This does not appear to be a problem with my venv since the user account on the qc8 machine gives the same problem:
[gemuser@gem904qc8daq 2018.06.01.04.39]$ anaUltraScurve.py -i SCurveData_trimdac0.root -c -f --calFile=/data/bigdisk/GEM-Data-Taking/GE11_QC8/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt
Analyzing: 'SCurveData_trimdac0.root'
mkdir: cannot create directory ‘SCurveData_trimdac0’: File exists
SysError in <TFile::TFile>: could not delete SCurveData_trimdac0/SCurveFitData.root (errno: 13) (Permission denied)
Traceback (most recent call last):
File "/opt/cmsgemos/bin/anaUltraScurve.py", line 191, in <module>
tuple_calInfo = parseCalFile(options.calFile)
File "/usr/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 429, in parseCalFile
array_CalData = rp.tree2array(tree=calTree, branches=list_bNames)
AttributeError: 'module' object has no attribute 'tree2array'
Steps to Reproduce (for bugs)
- Call
anaUltraScurve.py -i SomeSCurveFile.root -f --calFile=$DATA_PATH/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt
Possible Solution (for bugs)
The internet seems to think this is due to multiple top-level imports as described here. However I could not find an instance in which root_numpy was imported at top level in anaUltraScurve.py or any of it's imports.
Context (for feature requests)
Cannot run calibrated analysis.
Your Environment
- Version used:
1.0.0 - Shell used:
/bin/zsh
Bug Report: macros lack executable permissions post release install
Brief summary of issue
Following instructions developer or user instructions for setting up a scripts placed under:
$VENV_ROOT/lib/python2.7/site-packages/gempython/gemplotting/macros
Lack executable permissions. Originally reported by @lmoureaux at here
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
The following scripts need executable permissions:
% ll venv/slc6/py27_postpkg/lib/python2.7/site-packages/gempython/gemplotting/macros
total 182K
-rw-r--r--. 1 zh 2.8K May 30 22:20 summary_plots.py
-rw-r--r--. 1 zh 296 May 30 22:20 plot_vfat_summary.py
-rw-r--r--. 1 zh 420 May 30 22:20 plot_vfat_and_channel_Scurve.py
-rw-r--r--. 1 zh 4.2K May 30 22:20 plotTimeSeries.py
-rw-r--r--. 1 zh 1.1K May 30 22:20 plot_scurves_by_thresh.py
-rw-r--r--. 1 zh 20K May 30 22:20 plotSCurveFitResults.py
-rw-r--r--. 1 zh 434 May 30 22:20 plot_noise_vs_trim.py
-rw-r--r--. 1 zh 7.5K May 30 22:20 plot_eff.py
-rw-r--r--. 1 zh 16K May 30 22:20 gemTreeDrawWrapper.py
-rw-r--r--. 1 zh 6.2K May 30 22:20 gemSCurveAnaToolkit.py
-rw-r--r--. 1 zh 22K May 30 22:20 gemPlotter.py
-rw-r--r--. 1 zh 12K May 30 22:20 clusterAnaScurve.py
Current Behavior
Scripts found at:
$VENV_ROOT/lib/python2.7/site-packages/gempython/gemplotting/macros
lack executable permissions
Steps to Reproduce (for bugs)
- Install packages following instructions above.
Context (for feature requests)
After the venv is setup for a batch user or a developer scripts here need to be executable to be called from path independent location
Your Environment
- Version used: https://github.com/cms-gem-daq-project/gem-plotting-tools/releases/tag/v1.0.0-dev5
- Shell used:
/bin/zsh
Remove `cd` usage from paths.sh
Following:
cms-gem-daq-project/vfatqc-python-scripts#109
Sourcing the setup/paths.sh script leaves the user in a specific directory, which is not good practice, especially when used in a profile init script
For paths.sh here we should modify blocks at:
Generic plotting tool calibration
Currently all of the plots generated by the generic plotting tool gives all results in DAC units.
We should give this tool access to calibration parameters so plots can be reported in physical
units, rather than digital units. These plots ultimately should be in physical units so it is easier
for the user to compare results from VFAT to VFAT.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Current Behavior
Plots are in simple units kept in the data format, and we need a way of building these
top level plots in physical units.
Possible Solution (for bugs)
It is possible we need to have this script query a database of calibration parameters.
Context (for feature requests)
Producing plots which have physical units allows the user to make meaningful comparisons
within the system. As is, the plots made by this tool can themselves not even be compared
to each other in a sensible way.
Feature Request: anaUltraScurve.py uses TGraphAsymmErrors::BayesDivide()
Brief summary of issue
Based on the suggestion of @cbravo135 in #37:
We should change the scurve object to be a TGraphAsymmErrors made by dividing 2 histograms so C-P confidence intervals are used. Then all fits will be done with an efficiency of 0 to 1 and have the proper error bars. This would also mean several scurves could be statistically summed together and the analysis would be robust. It would also allow the number of pulses per Vcal bin to be unequal and the fit could still work.
I think the best way to implement this would be to change anaUltraScurve.py to fit a distribution that is made with the TGraphAsymmErrors::Divide() method following the BayesDivide() approach described here, or more aptly as the documentation suggests e.g.:
TGraphAsymmErrors::Divide(pass,total,"cl=0.683 b(1,1) mode")
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
scurve distributions made and fit by anaUltraScurve.py should go from 0 to 1 instead of 0 to options.Nev and then their error bars should reflect appropriate confidence intervals.
Current Behavior
scurve distributions made and fit by anaUltraScurve.py should go from 0 to options.Nev which require knowledge of the number of triggers per VCAL value that were used during the scan. Error bars at each point are simply sqrt(N) where N is the value of the function at a given VCAL value.
Feature Request: Function that plots an Arbitrary TBranch by scandate
Brief summary of issue
Right now macros/plot_eff.py will be given a tab deliminted file which looks like:
ChamberName scandate <IndepVarName>
GEMINIm27L1 2017.08.29.18.11 IndepVarVal1
GEMINIm27L1 2017.08.29.18.19 IndepVarVal2
GEMINIm27L1 2017.08.29.18.33 IndepVarVal3
GEMINIm27L1 2017.08.29.18.06 IndepVarVal4
GEMINIm27L1 2017.08.30.08.22 IndepVarVal5
And it will analyze the latTree found in the LatencyScanData.root file from each of the listed scandates given.
To allow the DOC additional tools we should provide an additional macros like plot_eff.py which can plot an arbitrary observable stored in our output TTrees e.g. vth1, trimDAC, trimRange, scurve mu, scurve sigma, etc... versus an independent variable. This independent variable could also be day of the week to allow for time series plotting.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
This new macros should follow the same style of syntax that plot_eff.py uses for simplicity. We could add additional parameters:
anaTypethis would define the type of root file to be analyzed and the tree name (see the proposed dicttree_namebelow).branchNamethe name of the branch we want plotted from the input tree vs the independent variable
Then the syntax could be:
gemPlotter.py --anaType=trim --infilename=listOfScanDates.txt --vfat=12 --strip=49 --treeName=scurveTree --branchName=trimDAC
Then it would produce a plot of trimDAC for strip 49 of VFAT 12 vs. the indep var defined in listOfScanDates.txt. The syntax could be changed but this would be the general idea.
Additionally I would propose in anaInfo.py we add an additional dict. Right now ana_config has key values that link a scan type to the python script that should analyze it. I would propose an additional dictionary tree_name that uses the same key values as ana_config but maps these keys to tuples (or pairs) where first element is the root file name and the second is the tree name within this root file:
tree_name = {
"latency":("LatencyScanData.root","latTree"),
"scurve":("SCurveData.root","scurveTree"),
"threshold":("ThresholdScanData.root",thrTree"),
"trim":("SCurveData_Trimmed.root","scurveTree")
}
This would tell gemPlotter.py what file to look at similiarly to how ana_scans.py knows where to look. The implementation should be that gemPlotter.py has a function declared such that other scripts could make use of this. In the example above this function has the ability to go down to the strip level per VFAT. Then this function could elsewhere such that summary plots for an entire chamber (or all chambers) could be made (perhaps by integrating this into ana_scans.py!!!)
I think this wouldn't be too difficult to implement and would give the DOC/community a nice tool to cover any analysis of interest.
Current Behavior
We have to make plots by hand...
Context (for feature requests)
Implementing this would somewhat automatize making plots from gem data for a given VFAT and could be expanded to making plots from the entire system.
Bug Report: ana_scans.py series mode crashes
Brief summary of issue
When trying to launch ana_scans.py with the option --series it crashes.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
Should not crash and launch analysis one after another.
Current Behavior
Crashes:
ana_scans.py --anaType=trim --scandate=current --ztrim=1 --series
Running jobs in serial mode
Traceback (most recent call last):
File "/home/GE11COSMICS/Software/DAQ/gem-plotting-tools/ana_scans.py", line 220, in <module>
options.latSigMaskRange
TypeError: launchAna() takes exactly 1 argument (12 given)
Steps to Reproduce (for bugs)
- Call
ana_scans.pywith the--seriesoption
Possible Solution (for bugs)
Looks like a typo at L209:
launchAna(options.anaType,
chamber,
GEB,
options.scandate,
options.scandatetrim,
Instead of a call of launchAna(...) here it should be launchAnaArgs(...).
Your Environment
- Version used: 97ab0f2
- Shell used: /bin/bash
Feature Request: ana_scans.py moves latency plots to $ELOG_PATH
Right now ana_scans.py does not move plots produced by anaUltraLatency.py to the $ELOG_PATH after the analysis has finished. Propose to add this functionality.
Bug Report: ScanDataFitter Generates Null Pointer For Empty TH1 in ROOT v6.X
Brief summary of issue
Attempting to fit an scurve with 0 entries causes a null pointer TFitResultPtr object to be returned even when supplying the fit option S.
Previously it looks like in ROOT v5.34 a TH1 object with 0 entries was being fit by a TF1 object using the TH1::Fit(...) method it would return a non-null pointer to a TFitResultPtr object when the fit option S. However, now in ROOT v6.X this returns a null pointer which crashes the program.
Looks like again default behavior of ROOT has changed between v5.34 and v.6.X.
Types of issue
- Bug report (report an issue with the code)
- Feature request (request for change which adds functionality)
Expected Behavior
If an scurve is empty it should just be skipped.
Current Behavior
At Line 80 of fitScanData.py the object fitResult will be a null-pointer in ROOT v6.X if the TH1 object being fit is empty even though the fit option S is used.
This then generates an error condition.
Steps to Reproduce (for bugs)
- Using ROOT v6.X
- Mask a channel or use a VFAT with a dead channel
- Run
ultraScurve.py - Try to analyze the output with the fitting option
-f
Possible Solution (for bugs)
Since this is generated from a projection of a 2D histogram the TH1::GetEntries() method is not reliable to determine if it's empty or not.
One should instead rely on the Integral (since it will always be greater than 0 for an S-curve).
Specifically I'm proposing to change Lines 59-60 to the following:
if self.isDead[vfat][ch]:
continue # Don't try to fit dead channels
elif not (self.scanHistos[vfat][ch].Integral() > 0):
continue
Context (for feature requests)
Fitting s-curve runs with empty s-curves will cause a crash on machines using ROOT v6.X.
Your Environment
- Version used: probably any
- Shell used:
zsh
Add repo structure
Following from cmsgemos and vfatqc-python-scripts:
cms-gem-daq-project/vfatqc-python-scripts#95
Add in contributing guidelines and templates (mostly copied from cmsgemos).
Suggest to copy the files from .github of vfatqc-python-scripts and place them here:
CONTRIBUTING.mdISSUE_TEMPLATE.mdPULL_REQUEST_TEMPALTE.md
And make the corresponding updates to .gitignore e.g.:
# Don't ignore special files
!*.md
!*.rst
# Don't ignore subdirectories
!*/
Update clusterAnaScurve.py for new computing environment.
Brief summary of issue
Since gem-plotting-tools has been updated to be more of a package that is installed or downloaded as a standalone release the clusterAnaScurve.py tool no longer properly sets up the computing env on the worker node of lxplus.
Additionally CERN IT is moving lxplus away from using the scheduler bsub to moving HTCondor:
http://batchdocs.web.cern.ch/batchdocs/
Now might be a good time to migrate to HTCondor.
Expected Behavior
The worker node on lxplus should:
- Download a given release and all supporting software,
- Unpack it and install it,
- Set the env
- Launch the analysis
- Return the results
You may have to adapt this procedure to instead download all tarballs/zip files, and then send them to the worker node instead of having the worker node do this.
Presently right now the worker node is using AFS directories this typically is "frowned" upon and angry emails may be sent from cluster administers. Perhaps we should alter this.
Current Behavior
The computing env setup by clusterAnaScurve.py is no longer compatible with how the package is installed as either a rpm, zip, or tarball. Specifically the following lines need to be updated:
gem-plotting-tools/macros/clusterAnaScurve.py
Lines 150 to 163 in dd3c00f
Additionally the example shown in the README.txt needs to be updated:
https://github.com/cms-gem-daq-project/gem-plotting-tools/#full-example-for-p5-s-curve-data
Context (for feature requests)
Need cluster computing abilities to re-analyze all s-curve data taken in P5.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.