jpata / cmssw Goto Github PK

In order to reproduce the TDR jet response plot, we need to make sure the jet response in ParticleFlowDQM.cc is implemented in the same way as in the JetMET code.
I started to dig through the JetMET code but so far didn't find the exact defition.

As a start, I changed the jet matching from reco->gen to gen->reco here:

Likely also the matching deltaR has to be changed to conform to hald the cone radius: https://github.com/cms-jet/JetMETAnalysis/blob/master/JetAnalyzers/python/Defaults_cff.py#L33

@juska @hatakeyamak

@hatakeyamak @juska
Now we used my personal CMSSW repo to develop the PFvalidation CMSSW code, but if we are generally happy with the direction, how about we move future developments to a new fork that is managed by the CMS PF group, not just me? We could set up a github organization (if it doesn't exist yet) similar as the BTV POG has done: https://github.com/cms-btv-pog

After choosing an existing version of CMSSW and fixed overseas file reading with 'voms-proxy-init -voms cms', I successfully proceeded to the final plotting step but there I face the following error:

[juska@lxplus014 RecoParticleFlow]$ make plots
rm -Rf plots
python test/compare.py --sample FlatQCD_noPU:tmp/QCD/DQMTotal.root:tmp/QCD/DQMTotal.root
--sample FlatQCD_PU25ns:tmp/QCDPU/DQMTotal.root:tmp/QCDPU/DQMTotal.root
--sample FlatQCD:tmp/QCD/DQMTotal.root:tmp/QCDPU/DQMTotal.root
--sample MinBias:tmp/MinBias/DQMTotal.root:tmp/MinBias/DQMTotal.root
--doResponsePlots
Trying to open DQM file tmp/QCD/DQMTotal.root for sample FlatQCD_noPU
Trying to open DQM file tmp/QCD/DQMTotal.root for sample FlatQCD_noPU
Trying to open DQM file tmp/QCDPU/DQMTotal.root for sample FlatQCD_PU25ns
Traceback (most recent call last):
File "test/compare.py", line 139, in
main()
File "test/compare.py", line 115, in main
samples, plots = parse_args()
File "test/compare.py", line 80, in parse_args
name, files = parse_sample_string(ss)
File "test/compare.py", line 29, in parse_sample_string
raise Exception("Could not read DQM file {0}, it does not exist".format(fi))
Exception: Could not read DQM file tmp/QCDPU/DQMTotal.root, it does not exist
make: *** [Makefile:31: plots] Error 1
[juska@lxplus014 RecoParticleFlow]$

As @hatakeyamak suggested, we should do the validation on the CHS jets.

I think it is better to deal with ak4 chs jets with default L1L2L3 corrections applied so that the response is first order already corrected.
Then, the resolution plot will be more sensitive to actual resolution effects, and less-dependent on non-uniformity of the response.

Now, we have some on-going discussion to reoptimize PFEGammaFilters parameters to find an optimized balance between tau and egamma needs. Then, when I looked at relatively long particleFlow_cfi.py, I was wondering if we can re-organize to make it easier to work with.

For example, introduce PSets such as:
PFEGammaFiltersParam
PFMuonAlgoParam
possibly also BadHcalTrackParam PSets
and put relevant parameter sets under them.

I think this will help egamma or muon people to focus only on parameters relevant for them.

Does this make sense? Is this something you can possibly look into? @guitargeek @jpata

Cc: @bendavid

Regarding the current post-processing step such as those performed by:
https://github.com/jpata/cmssw/blob/pfvalidation-10_5_0_pre1-master/Validation/RecoParticleFlow/test/addResponse.py
https://github.com/jpata/cmssw/blob/pfvalidation-10_5_0_pre1-master/Validation/RecoParticleFlow/test/addResponseAndResolution.py

I wonder if it can be run via DQM client.
Example:
https://github.com/jpata/cmssw/blob/pfvalidation-10_5_0_pre1-master/Validation/CaloTowers/src/CaloTowersClient.cc

The benefit of this is that, then, this can be included in the central relval sequence.

Response & resolution plot, add additional bin of 3<|eta|<4
so that we can test changes in the forward region which we anticipate to make at least for phase 2 with very forward tracks.
@juska @jpata

I made this issue to track the work on PFAlgo refactoring.

I'm now done with the first iteration of PF refactoring, PFAlgo::processBlock is broken up into subfunctions.

• Factorize PFAlgo::processBlock to smaller subfunctions (cms-sw#27153 and cms-sw#26894)

The next steps are:

• Follow up with a clang-format pass on the whole PFAlgo package
• Modernize logging and make it thread-safe (cout -> MessageLogger)
• Sanitize the inputs and outputs of the functions and making sure they rely less on the PFAlgo state, each function ideally having a single purpose and clear inputs and outputs.
• Given the cleaned-up implementation from above, revisit the logic/physics of the algo to rewrite it from with Run3/HL-LHC in mind
• Compare to an eventual ML PF algo that arrives in CMS

I appreciate your inputs to this todo list.

One thing to note, I will be offline June 14 - 26. Therefore, if someone else such as @guitargeek want to do a few rounds of optimization, from my point of view this is a good time to avoid conflicts.

cc @hatakeyamak @bendavid

I have the crab configs prepared, however, CMSSW_10_5_0_pre1 is not available on grid.

The latest we have there is CMSSW_10_4_0, see here, but we would need to rebase or backport the DQM changes, which may not be worth it at the moment.

@juska would you be able to update the script https://github.com/jpata/cmssw/blob/pfvalidation/Validation/RecoParticleFlow/test/validor.py such that I can include it in a workflow that runs on top of an existing DQM_V0001_R000000001__Global__CMSSW_X_Y_Z__RECO.root file and updates it with the response scan plot? I believe you also need to commit the Jet class and other dependencies.
Thank you!

In order to prepare for a offical-cmssw PR and to access CERN RElVal samples, need to move pfvalidation to 10_4_X.

Via @hatakeyamak

Do you think you can make an equivalent response vs pt plot in addition to resol vs pt? I think both response and resol plots should look similar to what Juska makes from nanoAOD. If we can confirm that, I think we can move on to L1-offset plots as we talked?

We use the script https://github.com/jpata/cmssw/blob/pfvalidation/Validation/RecoParticleFlow/test/compare.py
to create A vs B type comparison plots from DQM*.root files.

I'm working on creating a more convenient command line interface for that script, something along the lines of

./test/compare.py --sample QCD:DQMfile1.root:DQMfile2.root:DQMfile3.root --sample QCDNoPU:DQMfile4.root:DQMfile5.root:DQMfile6.root --plots JetResponse:reso_dist_10_24:reso_dist_10_24_eta05:reso_dist_10_24_eta13

would compare 3 versions of the sample QCD (DQMFile[1-3].root) and 3 versions of the sample QCDNoPu (DQMFile[4-6].root) and plot the histograms reso_dist_10_24_eta05:reso_dist_10_24_eta13 from the folder JetResponse on a canvas reso_dist_10_24.

@hatakeyamak @juska would appreciate your thoughts.

Via @juska, I'm working on adding this:

Joosep, if you have the time during coffee breaks, could you please add 
gen jet pT spectrum histogram production to the DQM step? Just need to 
fill a TH1D (with the used pT binning) with gen jet pT before any 
matching requirement etc.

I need this histogram as input to the addResponse.py in the future (I 
will read the number of gen jets per bin from this). This needs to be 
stored separately in all the different pseudorapidity bins that we use. 
Don't worry if you're busy and don't use your skiing breaks for this, I 
probably manage to do this myself too!

Also, I think its safer for the future to use a more fine-grained pT binning in all of the plots, for example this:

   ptbins = ar.array('d',[1, 5, 6, 8, 10, 12, 15, 18, 21, 24, 28, 32, 37, 43,
      49, 56, 64, 74, 84, 97, 114, 133, 153, 174, 196, 220, 245, 272, 300, 330,
      362, 395, 430, 468, 507, 548, 592, 638, 686, 737, 790, 846, 905, 967,
      1032, 1101, 1172, 1248, 1327, 1410, 1497, 1588, 1684, 1784, 1890, 2000,
      2116, 2238, 2366, 2500, 2640, 2787, 2941, 3103, 3273, 3450, 3637, 3832,
      4037, 4252, 4477, 4713, 4961, 5220, 5492, 5777, 6076, 6389, 6717, 7000])

https://twiki.cern.ch/twiki/bin/view/CMS/PFlowDQM
@jpata @juska

The updated README.md crashes at the second command:

[juska@lxplus014 feb12]$ export SCRAM_ARCH=slc6_amd64_gcc700
[juska@lxplus014 feb12]$ cmsrel CMSSW_10_5_X_2019-01-31-1100
ERROR: Unable to find release area for "CMSSW" version "CMSSW_10_5_X_2019-01-31-1100" for arch slc6_amd64_gcc700.
Please make sure you have used the correct name/version.
You can run "scram list CMSSW" to get the list of available versions.
[juska@lxplus014 feb12]$

I wanted to modify plot appearance and check the new error calculation, but I cannot run the latest version of the code due to file reading issues. Do you know if this is a temporary glitch or something else?

----- Begin Fatal Exception 12-Feb-2019 18:40:50 CET-----------------------
An exception of category 'FileOpenError' occurred while
[0] Constructing the EventProcessor
[1] Constructing input source of type PoolSource
[2] Calling RootFileSequenceBase::initTheFile()
[3] Calling StorageFactory::open()
[4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://cmsxrootd.fnal.gov//store/relval/CMSSW_10_4_0_pre4/RelValQCD_FlatPt_15_3000HS_13/GEN-SIM-DIGI-RAW/103X_mc2017_realistic_v2-v1/20000/E6912801-7DEE-6A4C-97F6-4A2FC7B15673.root'
Additional Info:
[a] Input file root://cmsxrootd.fnal.gov///store/relval/CMSSW_10_4_0_pre4/RelValQCD_FlatPt_15_3000HS_13/GEN-SIM-DIGI-RAW/103X_mc2017_realistic_v2-v1/20000/E6912801-7DEE-6A4C-97F6-4A2FC7B15673.root could not be opened.
[b] XrdCl::File::Open(name='root://cmsxrootd.fnal.gov//store/relval/CMSSW_10_4_0_pre4/RelValQCD_FlatPt_15_3000HS_13/GEN-SIM-DIGI-RAW/103X_mc2017_realistic_v2-v1/20000/E6912801-7DEE-6A4C-97F6-4A2FC7B15673.root', flags=0x10, permissions=0660) => error '[FATAL] Redirect limit has been reached' (errno=0, code=306). No additional data servers were found.
[c] Last URL tried: root://cmsxrootd.fnal.gov:1094//store/relval/CMSSW_10_4_0_pre4/RelValQCD_FlatPt_15_3000HS_13/GEN-SIM-DIGI-RAW/103X_mc2017_realistic_v2-v1/20000/E6912801-7DEE-6A4C-97F6-4A2FC7B15673.root?tried=
[d] Problematic data server: cmsxrootd.fnal.gov:1094
[e] Disabled source: cmsxrootd.fnal.gov:1094
----- End Fatal Exception -------------------------------------------------

At least the following changes would be necessary to make our plots look remotely professional:

• Split multi-panel plots more cleverly - now it's only two columns each which make them too tall
• Use "SetMoreLogLabels()" and "SetNoExponent()" for logarithmic pT axes
• Place legends so that they do not overlap with the rest of the plot
• Add space between top and bottom (ratio) panels
• Consider disabling plotting the grid on the background

I plan to work on these but need first to figure out how.

So far we were testing with a FlatQCD sample. Now need to add a few additional ones (PU & no PU, Zmm, MinBias). One issue is making sure we can access these samples from CERN, even if they are stored at FNAL.