agroce / tstl Goto Github PK
View Code? Open in Web Editor NEWTemplate Scripting Testing Language tool: automated test generation for Python
License: Other
Template Scripting Testing Language tool: automated test generation for Python
License: Other
What code exists in experimental/src? Is it still needed in the repository?
--generateLOC
without --noCover
Running the example command in the README:
$ tstl_rt --generateLOC sut.loc --timeout 120
Random testing using config=Config(seed=None, timeout=120, maxTests=-1, depth=100, output='failure.12380.test', multiple=False, running=False, quickTests=False, noCover=False, verbose=False, silentFail=False, silentSUT=False, normalize=False, generalize=False, swarm=False, generateLOC='sut.loc', biasLOC=None, exploit=None, reducePool=False, Pmutate=0.0, Pcrossover=0.2, profile=False, profileProbs=False, trackStates=False, stopSaturated=False, sessions=4, postCover=False, html=None, replayable=False, total=False, noCheck=False, uncaught=False, checkDeterminism=False, determinismTries=1, determinismDelay=0, checkProcessDeterminism=False, processDetTries=1, processDetDelay=0, quickPrefix='quick', readQuick=False, localize=False, localizeTop=20, full=False, logging=None, failedLogging=None, progress=False, timedProgress=30, ddmin=False, useDependencies=False, enumerateEnabled=False, noEnumerateEnabled=False, trackUsed=False, keepLast=False, noPruneGuards=False, swarmP=0.5, computeFeatureStats=False, saveSwarmCoverage=None, highLowSwarm=None, swarmProbs=None, swarmFromTest=None, swarmSwitch=None, swarmLength=None, probs=None, equalProbs=False, LOCBaseline=0.2, LOCProbs=False, markov=None, markovP=1.0, sequencesFromTests=None, sequenceP=1.0, sequenceSize=3, useQuickSequences=False, savePool=None, readPool=None, startExploit=0.0, startExploitStall=0, verboseExploit=False, exploitCeiling=0.5, useHints=False, noCoverageExploit=False, exploitBestHint=1, internal=False, fullCoverage=None, coverFile='coverage.out', noExceptionMatch=False, throughput=False, stopWhenBranches=None, stopWhenStatements=None, stopWhenNoCoverage=None, stopTestWhenNoCoverage=None, stopTestWhenThroughputBelow=None, stopWhenHitBranch=None, stopWhenHitStatement=None, trackMaxCoverage=None, maxMustHitBranch=None, maxMustHitStatement=None, verboseActions=False, hideOpaque=False, showActions=False, noAlphaConvert=False, compareFails=False, noSwarmDependencies=False, noSwarmForceParent=False, genDepth=None, stutter=None, greedyStutter=False, essentials=False, quickAnalysis=False, uniqueValuesAnalysis=False, fastQuickAnalysis=False, speed='FAST', noReassign=False, relax=False)
ERROR: cannot use --generateLOC without --noCover, instrumentations interfere with each other
ERROR: cannot use --generateLOC without --noCover, instrumentations interfere with each other
This is with coverage==4.5.2
installed at the specific version tstl==1.2.38
Note
coverage.py is at version +7.0.0
Additional msgs:
Warning: `bitwise_or()` is deprecated! Please use the | operator instead.
Warning: `bitwise_xor()` is deprecated! Please use the ^ operator instead.
Warning: `bitwise_and()` is deprecated! Please use the & operator instead.
I checked in latest version of source. Harnessmaker (tstl) generates the sut.py. While running the randomtester.py on the sut.py
when there is a bug in the library being tested, this error is observed. It is not observed when there is no bug in the library being tested. (Though TSTL catches the bug successfully). Adding the error traceback:
pranjal:~/projects/temp/tstlproj$ python /Users/pranjal/projects/tstl/generators/randomtester.py
Random testing using config=Config(verbose=False, failedLogging=None, maxtests=-1, greedyStutter=False, seed=None, generalize=False, uncaught=False, speed='FAST', internal=False, normalize=False, replayable=False, essentials=False, quickTests=False, coverfile='coverage.out', ignoreprops=False, total=False, noreassign=False, full=False, multiple=False, stutter=None, running=False, nocover=False, gendepth=None, logging=None, html=None, keep=False, depth=100, timeout=3600, output=None)
Coverage.py warning: No data was collected.
UNCAUGHT EXCEPTION
ERROR: (<type 'exceptions.ZeroDivisionError'>, ZeroDivisionError('integer division or modulo by zero',), <traceback object at 0x101732680>)
TRACEBACK:
File "/Users/pranjal/projects/temp/tstlproj/sut.py", line 897, in safely
act[2]()
File "/Users/pranjal/projects/temp/tstlproj/sut.py", line 503, in act21
mylib.hello(self.p_int[0])
File "/Users/pranjal/projects/temp/tstlproj/mylib.py", line 2, in hello
return 1/n
Original test has 90 steps
REDUCING...
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Reduced test has 2 steps
REDUCED IN 0.0253660678864 SECONDS
Traceback (most recent call last):
File "/Users/pranjal/projects/tstl/generators/randomtester.py", line 469, in <module>
main()
File "/Users/pranjal/projects/tstl/generators/randomtester.py", line 363, in main
handle_failure(test, "UNCAUGHT EXCEPTION", False)
File "/Users/pranjal/projects/tstl/generators/randomtester.py", line 146, in handle_failure
t.prettyPrintTest(test)
AttributeError: 'sut' object has no attribute 'prettyPrintTest'
def hello(n):
return 1/n
# If n is 0, then there will be ZeroDivisionError --> So this is a bug
The instructions to run tstl in the given [usage][1] in readme is pretty unclear.
tstl mytstlfile.tstl
python <tstl-root>/generators/randomtester.py
When the 1st line is run, the sut.py
is generated in the current directory. But when you run the 2nd line, randomtester.py
tries to import the sut
from sys.path
on which the current directory is not by default. I had to copy the randomtester.py
to the current directory to be able to run that.
It wanted to give TSTL a try, but it doesn't work for me on the nutshell example:
$ tstl tstl/examples/nutshell/nutshell.tstl
Generating harness core using config=Config(tstl='tstl/examples/nutshell/nutshell.tstl', stats=False, checkFailureDeterminism=False, pylib=False, defaultReplay=False, forceRefExceptionMatch=False, enumerateEnabled=False, forceStrongRefExceptionMatch=False, classname='sut', version=False, noCover=False, debug=False, output='sut.py', noReload=False, ignoreAngles=False, coverReload=False, coverInit=False)
$ tstl_rt
Random testing using config=Config(noExceptionMatch=False, generalize=False, localize=False, speed='FAST', probs=None, LOCBaseline=0.2, noAlphaConvert=False, markovP=1.0, essentials=False, noCoverageExploit=False, quickTests=False, html=None, noReassign=False, stopWhenNoCoverage=None, generateLOC=None, processDetDelay=0, computeFeatureStats=False, stopTestWhenNoCoverage=None, Pmutate=0.0, timeout=3600, output='failure.1658.test', markov=None, noSwarmDependencies=False, startExploit=0.0, maxMustHitBranch=None, useHints=False, silentFail=False, highLowSwarm=None, maxMustHitStatement=None, equalProbs=False, progress=False, postCover=False, swarmLength=None, processDetTries=1, noCover=False, noEnumerateEnabled=False, full=False, swarmP=0.5, coverFile='coverage.out', replayable=False, swarmProbs=None, genDepth=None, exploitCeiling=0.5, logging=None, noCheck=False, throughput=False, uncaught=False, swarmSwitch=None, verbose=False, reducePool=False, exploit=None, fastQuickAnalysis=False, determinismTries=1, total=False, verboseExploit=False, swarm=False, sequenceP=1.0, biasLOC=None, timedProgress=30, relax=False, ddmin=False, stutter=None, readQuick=False, verboseActions=False, quickAnalysis=False, keepLast=False, sequenceSize=3, LOCProbs=False, sequencesFromTests=None, hideOpaque=False, maxTests=-1, trackMaxCoverage=None, useQuickSequences=False, exploitBestHint=1, failedLogging=None, stopWhenStatements=None, greedyStutter=False, seed=None, stopWhenHitStatement=None, checkDeterminism=False, uniqueValuesAnalysis=False, normalize=False, startExploitStall=0, enumerateEnabled=False, stopTestWhenThroughputBelow=None, internal=False, profile=False, multiple=False, stopWhenBranches=None, running=False, compareFails=False, Pcrossover=0.2, stopWhenHitBranch=None, checkProcessDeterminism=False, depth=100, localizeTop=20, showActions=False)
Traceback (most recent call last):
File "/home/jwilk/.local/bin/tstl_rt", line 11, in <module>
load_entry_point('tstl==1.2.12', 'console_scripts', 'tstl_rt')()
File "/home/jwilk/.local/lib/python2.7/site-packages/src/randomtester.py", line 729, in main
sut = SUT.sut()
File "/home/jwilk/sut.py", line 6156, in __init__
self.__cov = coverage.coverage(branch=True, source=self.moduleLocations(), omit='sut.py' )
File "/home/jwilk/sut.py", line 9918, in moduleLocations
f = m.__file__
AttributeError: 'module' object has no attribute '__file__'
Apparently it trips over the math
module, which indeed doesn't have the __file__
attribute.
Tested with git master (1dfa1ee).
It would be good to have a README file added to each subdirectory, like examples, generators, etc providing an overview of what the code in that directory does.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.