The cms_fulllsimulation's discuss from ram1123

use xrdcp to copy in the condor script instead of cp

CMS_FulllSimulation/GetFullSimScriptsFromMCCM.py

Lines 383 to 384 in 10e2c55

 script_content += "echo \"cp -r HIG-RunIISummer20UL17NanoAODv9-03735.root $3/nanoAOD_$1_$2.root\"\n" # FIXME: Hardcoded nanoAOD output file name 

 script_content += "cp -r $6 $3/nanoAOD_$1_$2.root\n".format() # FIXME: Hardcoded nanoAOD output file name

Improve the condor script

This patch can be improved. Once can just keep all the text inside:

"""
a
b
b
"""

CMS_FulllSimulation/GetFullSimScriptsFromMCCM.py

Lines 349 to 385 in 10e2c55

 step = f"step{i}" 

 script_content += 'echo "###################################################"\n' 

 script_content += f"echo \"Running {step}...\"\n" 

 script_content += f"export SCRAM_ARCH={scramArch[step]}\n" 

 script_content += f"if [ -r ${{{step}}}/src ] ; then\n" 

 script_content += f" echo release ${{{step}}} already exists\n" 

 script_content += f" echo deleting release ${{{step}}}\n" 

 script_content += f" rm -rf ${{{step}}}\n" 

 script_content += f" scram p CMSSW ${{{step}}}\n" 

 script_content += f"else\n" 

 script_content += f" scram p CMSSW ${{{step}}}\n" 

 script_content += f"fi\n" 

 script_content += f"echo list files inside ${{{step}}}\n" 

 script_content += f"ls ${{{step}}}\n" 

 script_content += f'echo "--------"\n' 

 script_content += f"cd ${{{step}}}/src\n" 

 script_content += f"eval `scram runtime -sh`\n" 

 if args.UseCustomNanoAOD and step == 'step7': # FIXME: This is a temporary fix 

 script_content += f"git cms-merge-topic -u ram1123:CMSSW_10_6_30_HHWWgg_nanoV9\n" 

 script_content += f"./PhysicsTools/NanoTuples/scripts/install_onnxruntime.sh\n" 

 script_content += f"scram b\n" 

 script_content += f"cd -\n" 

 if i == 1: 

 # Special handling for step 1 with seed value 

 script_content += f"cmsRun ${{{step}_cfg}} seedval=${{seed}} maxEvents=${{5}} gridpack=${{4}}\n" 

 else: 

 script_content += f"cmsRun ${{{step}_cfg}}\n" 

 script_content += "echo \"list all files\"\n" 

 script_content += "ls -ltrh\n" 

 # copy output nanoAOD file to output directory with cluster ID and process ID as suffix of the root file 

 script_content += "\n# Copy output nanoAOD file to output directory\n" 

 script_content += "echo \"Copying output nanoAOD file to output directory\"\n" 

 script_content += "ls -ltrh\n" 

 script_content += "echo \"cp -r HIG-RunIISummer20UL17NanoAODv9-03735.root $3/nanoAOD_$1_$2.root\"\n" # FIXME: Hardcoded nanoAOD output file name 

 script_content += "cp -r $6 $3/nanoAOD_$1_$2.root\n".format() # FIXME: Hardcoded nanoAOD output file name 

 script_content += "echo \"Job finished on \" $(date)\n"

Add self checkpoints to the condor script

Add self-checkpoints to the condor script as mentioned here on slide 90:

https://indico.cern.ch/event/611296/contributions/2604376/attachments/1471164/2276521/TannenbaumT_UserTutorial.pdf

This will help if a couple of steps run and it fails afterwards. Then instead of starting the condor job from scratch, it will start from where it failed.

ram1123 / cms_fulllsimulation Goto Github PK

cms_fulllsimulation's Issues

use xrdcp to copy in the condor script instead of cp

Improve the condor script

Add self checkpoints to the condor script

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Jobs

	script_content += "echo \"cp -r HIG-RunIISummer20UL17NanoAODv9-03735.root $3/nanoAOD_$1_$2.root\"\n" # FIXME: Hardcoded nanoAOD output file name
	script_content += "cp -r $6 $3/nanoAOD_$1_$2.root\n".format() # FIXME: Hardcoded nanoAOD output file name

	step = f"step{i}"
	script_content += 'echo "###################################################"\n'
	script_content += f"echo \"Running {step}...\"\n"
	script_content += f"export SCRAM_ARCH={scramArch[step]}\n"
	script_content += f"if [ -r ${{{step}}}/src ] ; then\n"
	script_content += f" echo release ${{{step}}} already exists\n"
	script_content += f" echo deleting release ${{{step}}}\n"
	script_content += f" rm -rf ${{{step}}}\n"
	script_content += f" scram p CMSSW ${{{step}}}\n"
	script_content += f"else\n"
	script_content += f" scram p CMSSW ${{{step}}}\n"
	script_content += f"fi\n"
	script_content += f"echo list files inside ${{{step}}}\n"
	script_content += f"ls ${{{step}}}\n"
	script_content += f'echo "--------"\n'
	script_content += f"cd ${{{step}}}/src\n"
	script_content += f"eval `scram runtime -sh`\n"
	if args.UseCustomNanoAOD and step == 'step7': # FIXME: This is a temporary fix
	script_content += f"git cms-merge-topic -u ram1123:CMSSW_10_6_30_HHWWgg_nanoV9\n"
	script_content += f"./PhysicsTools/NanoTuples/scripts/install_onnxruntime.sh\n"
	script_content += f"scram b\n"
	script_content += f"cd -\n"
	if i == 1:
	# Special handling for step 1 with seed value
	script_content += f"cmsRun ${{{step}_cfg}} seedval=${{seed}} maxEvents=${{5}} gridpack=${{4}}\n"
	else:
	script_content += f"cmsRun ${{{step}_cfg}}\n"
	script_content += "echo \"list all files\"\n"
	script_content += "ls -ltrh\n"

	# copy output nanoAOD file to output directory with cluster ID and process ID as suffix of the root file
	script_content += "\n# Copy output nanoAOD file to output directory\n"
	script_content += "echo \"Copying output nanoAOD file to output directory\"\n"
	script_content += "ls -ltrh\n"
	script_content += "echo \"cp -r HIG-RunIISummer20UL17NanoAODv9-03735.root $3/nanoAOD_$1_$2.root\"\n" # FIXME: Hardcoded nanoAOD output file name
	script_content += "cp -r $6 $3/nanoAOD_$1_$2.root\n".format() # FIXME: Hardcoded nanoAOD output file name
	script_content += "echo \"Job finished on \" $(date)\n"