GithubHelp home page GithubHelp logo

agroce / tstl Goto Github PK

View Code? Open in Web Editor NEW
103.0 9.0 24.0 13.78 MB

Template Scripting Testing Language tool: automated test generation for Python

License: Other

Python 100.00%
testing-tools python tstl test-automation testing random-testing test-generation swarm-testing test-reduction delta-debugging

tstl's Issues

Issue: instrumentation interferes with each other cannot run `generateLOC` with coverage

Error: cannot use --generateLOC without --noCover

Running the example command in the README:

$ tstl_rt --generateLOC sut.loc --timeout 120

Error Output

Random testing using config=Config(seed=None, timeout=120, maxTests=-1, depth=100, output='failure.12380.test', multiple=False, running=False, quickTests=False, noCover=False, verbose=False, silentFail=False, silentSUT=False, normalize=False, generalize=False, swarm=False, generateLOC='sut.loc', biasLOC=None, exploit=None, reducePool=False, Pmutate=0.0, Pcrossover=0.2, profile=False, profileProbs=False, trackStates=False, stopSaturated=False, sessions=4, postCover=False, html=None, replayable=False, total=False, noCheck=False, uncaught=False, checkDeterminism=False, determinismTries=1, determinismDelay=0, checkProcessDeterminism=False, processDetTries=1, processDetDelay=0, quickPrefix='quick', readQuick=False, localize=False, localizeTop=20, full=False, logging=None, failedLogging=None, progress=False, timedProgress=30, ddmin=False, useDependencies=False, enumerateEnabled=False, noEnumerateEnabled=False, trackUsed=False, keepLast=False, noPruneGuards=False, swarmP=0.5, computeFeatureStats=False, saveSwarmCoverage=None, highLowSwarm=None, swarmProbs=None, swarmFromTest=None, swarmSwitch=None, swarmLength=None, probs=None, equalProbs=False, LOCBaseline=0.2, LOCProbs=False, markov=None, markovP=1.0, sequencesFromTests=None, sequenceP=1.0, sequenceSize=3, useQuickSequences=False, savePool=None, readPool=None, startExploit=0.0, startExploitStall=0, verboseExploit=False, exploitCeiling=0.5, useHints=False, noCoverageExploit=False, exploitBestHint=1, internal=False, fullCoverage=None, coverFile='coverage.out', noExceptionMatch=False, throughput=False, stopWhenBranches=None, stopWhenStatements=None, stopWhenNoCoverage=None, stopTestWhenNoCoverage=None, stopTestWhenThroughputBelow=None, stopWhenHitBranch=None, stopWhenHitStatement=None, trackMaxCoverage=None, maxMustHitBranch=None, maxMustHitStatement=None, verboseActions=False, hideOpaque=False, showActions=False, noAlphaConvert=False, compareFails=False, noSwarmDependencies=False, noSwarmForceParent=False, genDepth=None, stutter=None, greedyStutter=False, essentials=False, quickAnalysis=False, uniqueValuesAnalysis=False, fastQuickAnalysis=False, speed='FAST', noReassign=False, relax=False)

ERROR: cannot use --generateLOC without --noCover, instrumentations interfere with each other
ERROR: cannot use --generateLOC without --noCover, instrumentations interfere with each other

This is with coverage==4.5.2 installed at the specific version tstl==1.2.38

Note
coverage.py is at version +7.0.0

Additional msgs:

Warning: `bitwise_or()` is deprecated! Please use the | operator instead.
Warning: `bitwise_xor()` is deprecated! Please use the ^ operator instead.
Warning: `bitwise_and()` is deprecated! Please use the & operator instead.

AttributeError: 'sut' object has no attribute 'prettyPrintTest'

I checked in latest version of source. Harnessmaker (tstl) generates the sut.py. While running the randomtester.py on the sut.py when there is a bug in the library being tested, this error is observed. It is not observed when there is no bug in the library being tested. (Though TSTL catches the bug successfully). Adding the error traceback:

Traceback

pranjal:~/projects/temp/tstlproj$ python /Users/pranjal/projects/tstl/generators/randomtester.py
Random testing using config=Config(verbose=False, failedLogging=None, maxtests=-1, greedyStutter=False, seed=None, generalize=False, uncaught=False, speed='FAST', internal=False, normalize=False, replayable=False, essentials=False, quickTests=False, coverfile='coverage.out', ignoreprops=False, total=False, noreassign=False, full=False, multiple=False, stutter=None, running=False, nocover=False, gendepth=None, logging=None, html=None, keep=False, depth=100, timeout=3600, output=None)
Coverage.py warning: No data was collected.
UNCAUGHT EXCEPTION
ERROR: (<type 'exceptions.ZeroDivisionError'>, ZeroDivisionError('integer division or modulo by zero',), <traceback object at 0x101732680>)
TRACEBACK:
  File "/Users/pranjal/projects/temp/tstlproj/sut.py", line 897, in safely
    act[2]()
  File "/Users/pranjal/projects/temp/tstlproj/sut.py", line 503, in act21
    mylib.hello(self.p_int[0])
  File "/Users/pranjal/projects/temp/tstlproj/mylib.py", line 2, in hello
    return 1/n
Original test has 90 steps
REDUCING...
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Coverage.py warning: No data was collected.
Reduced test has 2 steps
REDUCED IN 0.0253660678864 SECONDS
Traceback (most recent call last):
  File "/Users/pranjal/projects/tstl/generators/randomtester.py", line 469, in <module>
    main()
  File "/Users/pranjal/projects/tstl/generators/randomtester.py", line 363, in main
    handle_failure(test, "UNCAUGHT EXCEPTION", False)
  File "/Users/pranjal/projects/tstl/generators/randomtester.py", line 146, in handle_failure
    t.prettyPrintTest(test)
AttributeError: 'sut' object has no attribute 'prettyPrintTest'

mylib.py: Input system

def hello(n):
    return 1/n

# If n is 0, then there will be ZeroDivisionError --> So this is a bug

Unclear instructions on running tstl

The instructions to run tstl in the given [usage][1] in readme is pretty unclear.

tstl mytstlfile.tstl
python <tstl-root>/generators/randomtester.py

When the 1st line is run, the sut.py is generated in the current directory. But when you run the 2nd line, randomtester.py tries to import the sut from sys.path on which the current directory is not by default. I had to copy the randomtester.py to the current directory to be able to run that.

AttributeError: 'module' object has no attribute '__file__'

It wanted to give TSTL a try, but it doesn't work for me on the nutshell example:

$ tstl tstl/examples/nutshell/nutshell.tstl 
Generating harness core using config=Config(tstl='tstl/examples/nutshell/nutshell.tstl', stats=False, checkFailureDeterminism=False, pylib=False, defaultReplay=False, forceRefExceptionMatch=False, enumerateEnabled=False, forceStrongRefExceptionMatch=False, classname='sut', version=False, noCover=False, debug=False, output='sut.py', noReload=False, ignoreAngles=False, coverReload=False, coverInit=False)

$ tstl_rt
Random testing using config=Config(noExceptionMatch=False, generalize=False, localize=False, speed='FAST', probs=None, LOCBaseline=0.2, noAlphaConvert=False, markovP=1.0, essentials=False, noCoverageExploit=False, quickTests=False, html=None, noReassign=False, stopWhenNoCoverage=None, generateLOC=None, processDetDelay=0, computeFeatureStats=False, stopTestWhenNoCoverage=None, Pmutate=0.0, timeout=3600, output='failure.1658.test', markov=None, noSwarmDependencies=False, startExploit=0.0, maxMustHitBranch=None, useHints=False, silentFail=False, highLowSwarm=None, maxMustHitStatement=None, equalProbs=False, progress=False, postCover=False, swarmLength=None, processDetTries=1, noCover=False, noEnumerateEnabled=False, full=False, swarmP=0.5, coverFile='coverage.out', replayable=False, swarmProbs=None, genDepth=None, exploitCeiling=0.5, logging=None, noCheck=False, throughput=False, uncaught=False, swarmSwitch=None, verbose=False, reducePool=False, exploit=None, fastQuickAnalysis=False, determinismTries=1, total=False, verboseExploit=False, swarm=False, sequenceP=1.0, biasLOC=None, timedProgress=30, relax=False, ddmin=False, stutter=None, readQuick=False, verboseActions=False, quickAnalysis=False, keepLast=False, sequenceSize=3, LOCProbs=False, sequencesFromTests=None, hideOpaque=False, maxTests=-1, trackMaxCoverage=None, useQuickSequences=False, exploitBestHint=1, failedLogging=None, stopWhenStatements=None, greedyStutter=False, seed=None, stopWhenHitStatement=None, checkDeterminism=False, uniqueValuesAnalysis=False, normalize=False, startExploitStall=0, enumerateEnabled=False, stopTestWhenThroughputBelow=None, internal=False, profile=False, multiple=False, stopWhenBranches=None, running=False, compareFails=False, Pcrossover=0.2, stopWhenHitBranch=None, checkProcessDeterminism=False, depth=100, localizeTop=20, showActions=False)
Traceback (most recent call last):
  File "/home/jwilk/.local/bin/tstl_rt", line 11, in <module>
    load_entry_point('tstl==1.2.12', 'console_scripts', 'tstl_rt')()
  File "/home/jwilk/.local/lib/python2.7/site-packages/src/randomtester.py", line 729, in main
    sut = SUT.sut()
  File "/home/jwilk/sut.py", line 6156, in __init__
    self.__cov = coverage.coverage(branch=True, source=self.moduleLocations(),    omit='sut.py'    )
  File "/home/jwilk/sut.py", line 9918, in moduleLocations
    f = m.__file__
AttributeError: 'module' object has no attribute '__file__'

Apparently it trips over the math module, which indeed doesn't have the __file__ attribute.

Tested with git master (1dfa1ee).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.