rohanpadhye / jqf Goto Github PK
View Code? Open in Web Editor NEWJQF + Zest: Coverage-guided semantic fuzzing for Java.
License: BSD 2-Clause "Simplified" License
JQF + Zest: Coverage-guided semantic fuzzing for Java.
License: BSD 2-Clause "Simplified" License
Currently, the names /tmp/a2j
and /tmp/j2a
are used statically for communicating between AFL and Java. Make this an auto-generated name so that multiple instances can be run simultaneously.
Make a script scripts/count_branches.sh [TRACE_FILE]
to count branches in a given trace file generated by jqf-repro
(defaulting to main.log
).
It doesn't usually make sense to try bit flips and integer arithmetic for the inputs to quickcheck's PRNG. Therefore, let's force the -d
option to AFL by default.
jqf-ei
, which backs the Maven plugin, now supports a number of system properties to configure its behavior.
Need to document these somewhere (probably in the Maven plugin wiki page).
If Janala can't instrument something, ignore it with a simple warning instead of a blaring stacktrace.
This is mainly useful for include/exclude lists, and therefore should probably go in https://github.com/rohanpadhye/jqf/wiki/Writing-a-JQF-test
I think for JQF to be successful, it needs to aim for as many developers as possible. Using JUnit is probably a very good starting idea. However, I think JQF should it make as easy as possible to create a fuzzing run. There are different aspects, I open a new issue for each of them.
First of all, I think Zest needs a tutorial that does not require writing a generator and just use the JUnit built-in generators. Because maybe a lot of developers only have trivial types they use for the things they want to fuzz (such as parsers that take InputStream and String).
This would make the current 101 tutorial to be 102 and this be the new 101. It works exactly as the current 101 tutorial, but does not require writing a generator class. It will also print Fuzzing stopped due to guidance exception: Assumption is too strong; too many inputs discarded
when running JUnit, and Zest will eventually find a failure where java.lang.AssertionError: yZiou"8no should be a YES
.
This tutorial introduces a trivial bug in a program and shows how to find it with Zest. We require a method to always return true if a String starts with "y". But it includes an artificial bug: even if the String starts with "y" but has the characters "no" in it, it will return false.
public class TrivialLogic{
public static boolean isYes(String in) {
if (in.contains("no"))
return false;
else if (in.startsWith("y"))
return true;
else
return false;
}
}
import static org.junit.Assert.*;
import static org.junit.Assume.*;
public class TrivialTest {
public void testTrivial(String in) {
assumeFalse(in.contains("may"); //special case we don't want our logic to test
assumeTrue(in.startsWith("y"));
assertTrue(in + " should be a YES", TrivialLogic.isYes(in));
}
}
import static org.junit.Assert.*;
import static org.junit.Assume.*;
import org.junit.runner.RunWith;
import com.pholser.junit.quickcheck.*;
import com.pholser.junit.quickcheck.generator.*;
import edu.berkeley.cs.jqf.fuzz.*;
@RunWith(JQF.class)
public class TrivialTest {
@Fuzz
public void testTrivial(String in) {
assumeFalse(in.contains("may"));
assumeTrue(in.startsWith("y"));
assertTrue(in + " should be a YES", TrivialLogic.isYes(in));
}
}
The input corpus is the most important success factor of a properly configured fuzzer. Therefore, corpus has to be distilled, a minimized corpus can provide good code coverage while not including unnecessary files not resulting in new code paths being taken.
afl-cmin is very useful in this regard. I had situations where I ran afl-cmin for several days, but then the fuzzer found issues within a day.
Would it be possible to provide afl-cmin for jqf? And maybe also afl-tmin?
Probably missing timeout specification in #26 is a blocker for this one...
we added the EXIT_ON_CRASH feature which works good but the exit code is always 0 which is a bit weird and I can rely on in a script. I'm not sure were to put this change. should I throw an exception here:
I think I've encountered a bug or either I'm doing something wrong.
I've added a simple fuzzing test to JSqlParser library.
The execution of the test via mvn -Dtest=CCJSqlParserFuzz test
(i.e quickcheck) works fine.
but when I use mvn jqf:fuzz -Dclass=CCJSqlParserFuzz -Dmethod=testParseExpression
I get the following error:
[ERROR] Failed to execute goal edu.berkeley.cs.jqf:jqf-maven-plugin:1.2:fuzz (default-cli) on project jsqlparser: Could not load test class: Cannot find class CCJSqlParserFuzz -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Here are the exact steps to reproduce
git clone https://github.com/fuzzitdev/JSqlParser
cd JSqlParser
git checkout fuzzing
docker run -v `pwd`:/jsql -it maven:3.6.1-jdk-8 /bin/bash
cd /jsql
## This works
mvn -Dtest=CCJSqlParserFuzz test
## this doesn't work
mvn jqf:fuzz -Dclass=CCJSqlParserFuzz -Dmethod=testParseExpression
cc @rohanpadhye
Hi Rohan,
I'm currently running into an issue:
AFL_DIR="/data1/fuzzing/tools/afl-2.52b" /data1/fuzzing/tools/jqf-zip/bin/jqf-afl-showmap -m 200 -t 30000 -i /data1/fuzzing/documents/test1.txt -c $(/data1/fuzzing/tools/jqf-zip/scripts/examples_classpath.sh) edu.berkeley.cs.jqf.examples.tika.TikaParserTest fuzz
afl-showmap 2.52b by <[email protected]>
[*] Executing '/data1/fuzzing/tools/jqf-zip/bin/jqf-afl-target'...
-- Program output begins --
Error occurred during initialization of VM
Could not reserve enough space for code cache
I guess that error is from the JVM start in scripts/jqf-driver.sh? I tried to add several JVM options there, for example -Xmx2000m -Xms2000m -XX:ReservedCodeCacheSize=2048m
but no luck so far. Any idea?
In "Analyzing crashes (and other fuzzed inputs)" section of https://github.com/rohanpadhye/jqf/wiki/Fuzzing-with-AFL
You can reproduce one or more inputs found by AFL using the program jqf-repro:
jqf/bin/jqf-afl-repro -c tests MyTest fuzzThis fuzz-results/crashes/*
The example for the command should be jqf-repro
too, as jqf-afl-repro
does not exist
I am using zest fuzzing with the maven plugin.
The fuzzing is directly prompting after the first iteration with this IllegalArgumentException:
Fuzzing stopped due to guidance exception: java.lang.IllegalArgumentException: bound must be positive
edu.berkeley.cs.jqf.fuzz.guidance.GuidanceException: java.lang.IllegalArgumentException: bound must be positive
at edu.berkeley.cs.jqf.fuzz.junit.quickcheck.FuzzStatement.evaluate(FuzzStatement.java:159)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at edu.berkeley.cs.jqf.fuzz.junit.GuidedFuzzing.run(GuidedFuzzing.java:184)
at edu.berkeley.cs.jqf.fuzz.junit.GuidedFuzzing.run(GuidedFuzzing.java:126)
at edu.berkeley.cs.jqf.plugin.FuzzGoal.execute(FuzzGoal.java:207)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
Caused by: java.lang.IllegalArgumentException: bound must be positive
at java.base/java.util.Random.nextInt(Random.java:388)
at edu.berkeley.cs.jqf.fuzz.ei.ZestGuidance$LinearInput.fuzz(ZestGuidance.java:1137)
at edu.berkeley.cs.jqf.fuzz.ei.ZestGuidance.getInput(ZestGuidance.java:556)
at edu.berkeley.cs.jqf.fuzz.junit.quickcheck.FuzzStatement.evaluate(FuzzStatement.java:136)
... 38 more
the error is occuring in the java.util.Random#nextInt(int) method as the passed bound is equal to zero because the values is empty.
basically it's the line 1137, in edu.berkeley.cs.jqf.fuzz.ei.ZestGuidance.LinearInput#fuzz which is causing the crash:
int offset = random.nextInt(newInput.values.size());
thank you very much for your time and support!! :)
Currently, the tests only test internal logic of various guidance engines without actually verifying that the code-coverage feedback is working.
We need a way to test that the instrumentation works, and that coverage-guided fuzzing yields expected results (perhaps by using a fixed seed?) every time a change is made. This is to prevent introducing regressions like the ones that kept creeping in for #26.
This is going to be tricky, though, since we can't easily run on-the-fly bytecode instrumentation from within JUnit tests.
Following the tutorial on https://github.com/rohanpadhye/jqf/wiki/Fuzzing-with-AFL on macOS 10.14.5 (Mojave), I was unable to get the example to run successfully.
I'm not sure if this is a problem with jqf or my install of afl. The jqf -v option just created an empty jqf.log file so no insight there. The jqf -t option did not help.
$ jqf/bin/jqf-afl-fuzz -i afl/testcases/images/png/ -c $(jqf/scripts/classpath.sh):<classpath> com.foo.PngTest testRead
Performing pilot run.... Pilot run success! Launching AFL now...
afl-fuzz 2.52b by <[email protected]>
[+] You have 4 CPU cores and 3 runnable tasks (utilization: 75%).
[+] Try parallel jobs - see docs/parallel_fuzzing.txt.
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning 'afl/testcases/images/png/'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:not_kitty.png'...
[!] WARNING: Test case results in a timeout (skipping)
[*] Attempting dry run with 'id:000001,orig:not_kitty_alpha.png'...
[!] WARNING: Test case results in a timeout (skipping)
[*] Attempting dry run with 'id:000002,orig:not_kitty_gamma.png'...
[!] WARNING: Test case results in a timeout (skipping)
[*] Attempting dry run with 'id:000003,orig:not_kitty_icc.png'...
[!] WARNING: Test case results in a timeout (skipping)
[-] PROGRAM ABORT : All test cases time out, giving up!
Location : perform_dry_run(), afl-fuzz.c:2883
Perhaps we need a new wiki page?
I have build errors again. First trying to run mvn package
:
[INFO] Reactor Summary:
[INFO]
[INFO] jqf ............................................... SUCCESS [0.006s]
[INFO] jqf-instrument .................................... SUCCESS [6.658s]
[INFO] jqf-fuzz .......................................... SUCCESS [28.155s]
[INFO] jqf-maven-plugin .................................. FAILURE [3.624s]
[INFO] jqf-examples ...................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 39.056s
[INFO] Finished at: Tue May 08 01:26:35 ACST 2018
[INFO] Final Memory: 16M/40M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.5.1:descriptor (default-descriptor) on project jqf-maven-plugin: Error extracting plugin descriptor: 'No mojo definitions were found for plugin: edu.berkeley.cs.jqf:jqf-maven-plugin.' -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.5.1:descriptor (default-descriptor) on project jqf-maven-plugin: Error extracting plugin descriptor: 'No mojo definitions were found for plugin: edu.berkeley.cs.jqf:jqf-maven-plugin.'
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error extracting plugin descriptor: 'No mojo definitions were found for plugin: edu.berkeley.cs.jqf:jqf-maven-plugin.'
at org.apache.maven.plugin.plugin.AbstractGeneratorMojo.execute(AbstractGeneratorMojo.java:271)
at org.apache.maven.plugin.plugin.DescriptorGeneratorMojo.execute(DescriptorGeneratorMojo.java:90)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
Caused by: org.apache.maven.plugin.descriptor.InvalidPluginDescriptorException: No mojo definitions were found for plugin: edu.berkeley.cs.jqf:jqf-maven-plugin.
at org.apache.maven.tools.plugin.scanner.DefaultMojoScanner.populatePluginDescriptor(DefaultMojoScanner.java:114)
at org.apache.maven.plugin.plugin.AbstractGeneratorMojo.execute(AbstractGeneratorMojo.java:259)
... 22 more
Then when trying mvn package -e -pl examples
:
[ERROR] Failed to execute goal on project jqf-examples: Could not resolve dependencies for project edu.berkeley.cs.jqf:jqf-examples:jar:1.0-alpha-3-SNAPSHOT: The following artifacts could not be resolved: edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT, edu.berkeley.cs.jqf:jqf-instrument:jar:1.0-alpha-3-SNAPSHOT: Could not find artifact edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT in lila-maven (https://raw.githubusercontent.com/ornicar/lila-maven/master) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project jqf-examples: Could not resolve dependencies for project edu.berkeley.cs.jqf:jqf-examples:jar:1.0-alpha-3-SNAPSHOT: The following artifacts could not be resolved: edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT, edu.berkeley.cs.jqf:jqf-instrument:jar:1.0-alpha-3-SNAPSHOT: Could not find artifact edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT in lila-maven (https://raw.githubusercontent.com/ornicar/lila-maven/master)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:210)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:117)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:258)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:201)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project edu.berkeley.cs.jqf:jqf-examples:jar:1.0-alpha-3-SNAPSHOT: The following artifacts could not be resolved: edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT, edu.berkeley.cs.jqf:jqf-instrument:jar:1.0-alpha-3-SNAPSHOT: Could not find artifact edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT in lila-maven (https://raw.githubusercontent.com/ornicar/lila-maven/master)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:189)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:185)
... 22 more
Caused by: org.sonatype.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT, edu.berkeley.cs.jqf:jqf-instrument:jar:1.0-alpha-3-SNAPSHOT: Could not find artifact edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT in lila-maven (https://raw.githubusercontent.com/ornicar/lila-maven/master)
at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:375)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:183)
... 23 more
Caused by: org.sonatype.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT, edu.berkeley.cs.jqf:jqf-instrument:jar:1.0-alpha-3-SNAPSHOT: Could not find artifact edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT in lila-maven (https://raw.githubusercontent.com/ornicar/lila-maven/master)
at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:538)
at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:216)
at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:358)
... 24 more
Caused by: org.sonatype.aether.transfer.ArtifactNotFoundException: Could not find artifact edu.berkeley.cs.jqf:jqf-fuzz:jar:1.0-alpha-3-SNAPSHOT in lila-maven (https://raw.githubusercontent.com/ornicar/lila-maven/master)
at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:947)
at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:939)
at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:669)
at org.sonatype.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:60)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
What am I doing wrong?
Hello !
I am trying to adapt some existing junit tests to turn them into fuzz-testing with jqf. I tried to code a generator for my data type. For that I used the SourceOfRandomness random
object to read the data byte per byte with the method nextBytes
, as you did in this example. However, I have the feeling that the return values are random and do not depend on the mutated inputs fed by afl.
I think there is an interface implementation problem between StreamBackedRandom and SourceOfRandomness.
Next bytes ask its delegate
to "generate" the byte. As far as I have understood, the delegate is an instance of StreamBackedRandom and generates data from the input file. But as it does not override the nextBytes
method, it can't be called from SourceOfRandomness.
Moreover, I could use the nextByte method but it does not seem to rely on its delegate
attribute to generate the value, so I can't use it.
To investigate, I tried to log the data fed to the tested method, I wrote to a file the arguments passed the the targeted method during the first dry run of afl, with the -i
option set and a corpus made of a file full of aaaaaaaaaaaaa
. I expected to see arguments based on aaaa...
, but I had different values each time I restarted and they seemed random. This was tried with the DateFormatterTest example.
If you think that's not a bug, how can I build a generator that parses the data fed by afl ?
PS : congrats for this project, I really enjoy the way you approach adapting AFL fuzzing style to java. 👍
Hi there,
I seem to be stuck when trying to do a fuzzing run. My class to fuzz:
import java.io.ByteArrayInputStream;
import java.io.IOException;
import some.import.other.SomeParser;
import some.import.other.CustomExceptionOne;
import some.import.other.CustomExceptionTwo;
import org.junit.runner.RunWith;
import edu.berkeley.cs.jqf.fuzz.junit.Fuzz;
import edu.berkeley.cs.jqf.fuzz.junit.quickcheck.JQF;
@RunWith(JQF.class)
public class SilentDriver {
@Fuzz
public void main(byte[] arg) throws IOException, CustomExceptionOne, CustomExceptionTwo {
SomeParser parser = new SomeParser();
final ByteArrayInputStream stream = new ByteArrayInputStream(arg);
parser.parse(stream);
stream.close();
parser.toString();
}
}
How it gets stuck:
$ /opt/jqf/bin/jqf-afl-fuzz -c /opt/jqf/example/other/bin/:/opt/jqf/examples/target/classes/:/opt/jqf/examples/target/test-classes:/opt/jqf/fuzz/target/jqf-fuzz-1.0-SNAPSHOT.jar:/opt/jqf/examples/target/dependency/*:someCwdJar.jar -i in_dir/ -o out_dir/ SilentDriver main
Performing pilot run....
I know that the SomeParser isn't the fastest, so it might take a while (a second or two per invocation). However, it seems stuck.
afl-fuzz is by default very verbose and talks to me what it's trying to do, while jqf-afl-fuzz is very quiet. "Performing pilot run" doesn't mean anything to me. Can I enable logging? Can I run it verbosely? What is happening?
We should be able to work with a system install of afl-fuzz
.
This means that the AFL proxy cannot use config.h
from AFL_DIR
. We will have to hard code the MAP_SIZE
.
in the wiki, the zest comands are written without the parameters prefixs.
i.e. to fuzz a method of a class the comand should be:
mvn jqf:fuzz -Dclass=com.tuto.zest.CalendarLogicTest -Dmethod=testLeapYear
but is written:
mvn jqf:fuzz CalendarLogicTest testLeapYear
Very good library!
Best regards
Any chance we could support Java Security Policy with JQF? An example is here: https://github.com/floyd-fuh/kelinci/tree/master/examples/commons-imaging . It basically means starting Java with java -Djava.security.manager -Djava.security.policy=java-security-policy.txt
, which would then throw an exception if the Java program tries to write to a file that was not specified in the policy (whitelist). In theory it should allow finding vulnerabilities such as Server Side Request Forgery, as the code would throw an exception when the Java program tries to create a socket.
I use jqf to fuzz generic functions, and the result is good. But when i try to fuzz web application functions written in Spring format, some problems occur. How can i use autowire and mock in JQF, is it possible to extend jqf to fuzz wen application?
@rohanpadhye
While writing a basic example for Fuzzit (an example I always use for other fuzzers) https://github.com/fuzzitdev/example-java#understanding-the-bug
I saw that it is having hard time finding this bug. maybe we can take some mutation heuristics from go-fuzz or libfuzzer so it will work better with stream of bytes. go-fuzz also use sonar heuristics which is quite powerful and might work well for jqf. but for a start I think this should be detected just by coverage guided and the right mutations. do you have an idea why it is not detected?/what are the current mutations in the case of my example?
Hi there,
I wanted to try this project but the mvn package fails:
Downloaded: http://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.7.4/byte-buddy-1.7.4.jar (2793 KB at 4017.9 KB/sec)
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ jqf-fuzz ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /opt/jqf/fuzz/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ jqf-fuzz ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 29 source files to /opt/jqf/fuzz/target/classes
[INFO] /opt/jqf/fuzz/src/main/java/edu/berkeley/cs/jqf/fuzz/util/ProducerTreeMap.java: Some input files use unchecked or unsafe operations.
[INFO] /opt/jqf/fuzz/src/main/java/edu/berkeley/cs/jqf/fuzz/util/ProducerTreeMap.java: Recompile with -Xlint:unchecked for details.
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /opt/jqf/fuzz/src/main/java/edu/berkeley/cs/jqf/fuzz/junit/quickcheck/FuzzStatement.java:[140,37] incompatible types: cannot infer type-variable(s) R
(argument mismatch; java.util.function.Function<capture#1 of ? super com.pholser.junit.quickcheck.generator.Generator<?>,capture#2 of ? extends capture#3 of ?> cannot be converted to java.util.function.Function<? super com.pholser.junit.quickcheck.generator.Generator<?>,? extends capture#3 of ?>)
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] jqf ............................................... SUCCESS [0.005s]
[INFO] jqf-instrument .................................... SUCCESS [30.904s]
[INFO] jqf-fuzz .......................................... FAILURE [3.553s]
[INFO] jqf-examples ...................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35.059s
[INFO] Finished at: Wed Feb 21 03:12:41 CST 2018
[INFO] Final Memory: 15M/37M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project jqf-fuzz: Compilation failure
[ERROR] /opt/jqf/fuzz/src/main/java/edu/berkeley/cs/jqf/fuzz/junit/quickcheck/FuzzStatement.java:[140,37] incompatible types: cannot infer type-variable(s) R
[ERROR] (argument mismatch; java.util.function.Function<capture#1 of ? super com.pholser.junit.quickcheck.generator.Generator<?>,capture#2 of ? extends capture#3 of ?> cannot be converted to java.util.function.Function<? super com.pholser.junit.quickcheck.generator.Generator<?>,? extends capture#3 of ?>)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :jqf-fuzz
I don't really understand what is going on as I'm not that familiar with quickcheck. Any ideas?
Would it be possible to get the -t switch to specify a timeout for a testcase? Fuzzing is only efficient when the inputs do not take too long. I'm currently trying to do another run with Tika, but alredy the fourth input dry run is just stuck for several minutes...
We would like to evaluate the following types of feedback:
Now that we have a brand new plugin, we should document mvn jqf:fuzz
and mvn jqf:repro
somewhere.
Currently, JQF scripts create files and directories in /tmp
for intermediate stuff.
However, on some machines the users running JQF may not have permission to write to /tmp
. Therefore, let's make this a configurable location with default as /tmp
and overridable via the env var JQF_TMP_DIR
.
When building with $ jqf/setup.sh
I got this error:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] jqf ................................................ SUCCESS [ 0.003 s]
[INFO] jqf-instrument ..................................... SUCCESS [ 5.486 s]
[INFO] jqf-fuzz ........................................... SUCCESS [ 7.941 s]
[INFO] jqf-maven-plugin ................................... SUCCESS [ 2.638 s]
[INFO] jqf-examples ....................................... FAILURE [01:10 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:27 min
[INFO] Finished at: 2018-09-18T14:27:25+00:00
[INFO] Final Memory: 37M/530M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project jqf-examples: Could not resolve dependencies for project edu.berkeley.cs.jqf:jqf-examples:jar:1.0-beta-1-SNAPSHOT: Could not find artifact org.lichess:scalachess_2.12:jar:8.6.8 in mirror (https://uk.maven.org/maven2/content/groups/public) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :jqf-examples
Currently, we start with an empty seed but that sometimes leads to fuzzing getting stuck in boring places (because the first few bytes are often interpreted as the length of something and AFL trims an input down if there wasn't anything generated).
Instead, let's make jqf-afl-fuzz
start with random inputs as seeds. We can either generate these once and store them in examples/src/seeds
or generate them fresh every time the script is run (for non-deterministic results).
I made a jqf-afl-cmin-py.sh which is very similar to jqf-afl-cmin.sh, but uses the afl-cmin.py which supports multiple workers/threads, so that an afl-cmin run is faster. afl-kit which includes afl-cmin.py can be found at https://github.com/kcwu/afl-kit
And here's the script:
#!/bin/bash
# Figure out script absolute path
pushd `dirname $0` > /dev/null
BIN_DIR=`pwd`
popd > /dev/null
ROOT_DIR=`dirname $BIN_DIR`
print_usage() {
echo "Usage: $0 [options] TEST_CLASS TEST_METHOD"
echo "Options: "
echo " -c JAVA_CLASSPATH Classpath used to find your test classes (default is '.')"
echo " -i INPUT_DIR Directory containing original corpus (default is 'fuzz-results')"
echo " -o OUTPUT_DIR Directory where minimized corpus will be written (default is 'fuzz-min')"
echo " -m MEM_LIMIT Set a memory limit in MB (default is 8192)"
echo " -t TIMEOUT Set a single-run timeout in milliseconds (default is 10000)"
echo " -v Enable verbose logging (in file 'jqf.log')"
}
# Ensure that afl-cmin.py can be found
if [ -n "$AFL_CMIN_PY_DIR" ]; then
AFL_CMIN_PY="$AFL_CMIN_PY_DIR/afl-cmin.py"
else
AFL_CMIN_PY=$(which afl-cmin.py)
fi
if [ ! -x "$AFL_CMIN_PY" ]; then
echo "The program 'afl-cmin.py' cannot be found" >&2
echo "Fix this in one of two ways:" >&2
echo " 1. Make sure 'afl-cmin.py' is in your PATH" >&2
echo " 2. Set the env var AFL_CMIN_PY_DIR to point to where afl-kit is installed" >&2
exit 2
fi
# Ensure that AFL proxy is built
if [ ! -f "$ROOT_DIR/bin/afl-proxy" ]; then
echo "The JQF-AFL proxy has not been built! Make sure to run scripts/setup.sh or run 'make'" >&2
exit 3
fi
# Set temp directory
if [ -z "$JQF_TMP_DIR" ]; then
JQF_TMP_DIR="/tmp"
fi
# Build AFL command-line
afl_options=""
classpath="."
mem_limit="8192"
target_options=""
in_dir="fuzz-results"
out_dir="fuzz-min"
workers="2"
timeout="10000"
while getopts ":c:i:o:m:vw:t:" opt; do
case $opt in
/?)
echo "Invalid option: -$OPTARG" >&2
print_usage >&1
exit 1
;;
c)
classpath="$OPTARG"
;;
i)
in_dir="$OPTARG"
;;
o)
out_dir="$OPTARG"
;;
m)
mem_limit="$OPTARG"
;;
v)
target_options="$target_options -v"
export JVM_OPTS="$JVM_OPTS -Djanala.verbose=true"
;;
w)
workers="$OPTARG"
;;
t)
timeout="$OPTARG"
;;
esac
done
shift $((OPTIND-1))
# Check positional arguments
if [ $# -lt 2 ]; then
print_usage >&2
exit 1
fi
class="$1"
method="$2"
target="$BIN_DIR/jqf-afl-target"
# Set AFL options
afl_options="$afl_options -i $in_dir -o $out_dir"
afl_options="$afl_options -m $mem_limit"
afl_options="$afl_options -t $timeout"
afl_options="$afl_options -w $workers"
# Set classpath for JQF
export CLASSPATH="$classpath"
# Set environment variables for AFL
export AFL_SKIP_BIN_CHECK=1
export AFL_NO_AFFINITY=1
export AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES=1
export AFL_SKIP_CPUFREQ=1
# Ready to go
echo "$AFL_CMIN_PY" $afl_options "$target" $target_options "$class" "$method" @@
exec "$AFL_CMIN_PY" $afl_options "$target" $target_options "$class" "$method" @@
Currently, FileBackedRandom
falls back to a (fixed-seed) pseduo-random number generator when the input file runs out of bytes.
Change this to throwing an AssumptionViolated
exception when reading after EOF so that the fuzzer finds small inputs to be invalid.
Do you think is sensible to update the dependency to 6.2.1?
Show how to add a maven dependency to build with JQF
Write generator for JGraphT graphs using the Erdős–Rényi model, and run analysis on various graph algorithms.
Would be useful for debugging/evaluation.
I have noticed this warning trace in my fuzzing log. It said "please report it", so that's what I am doing. I have no idea how to reproduce it...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by edu.berkeley.cs.jqf.instrument.tracing.SingleSnoop (.../.m2/repository/edu/berkeley/cs/jqf/jqf-instrument/1.1/jqf-instrument-1.1.jar) to field java.lang.Thread.target
WARNING: Please consider reporting this to the maintainers of edu.berkeley.cs.jqf.instrument.tracing.SingleSnoop
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Exception in thread "myThread" java.lang.IllegalStateException: class edu.berkeley.cs.jqf.fuzz.ei.ZestGuidance only supports single-threaded apps at the moment
at edu.berkeley.cs.jqf.fuzz.ei.ZestGuidance.generateCallBack(ZestGuidance.java:880)
at edu.berkeley.cs.jqf.instrument.tracing.ThreadTracer.spawn(ThreadTracer.java:107)
at edu.berkeley.cs.jqf.instrument.tracing.TraceLogger.lambda$new$0(TraceLogger.java:39)
at java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
at edu.berkeley.cs.jqf.instrument.tracing.TraceLogger.log(TraceLogger.java:43)
at janala.logger.AbstractLogger.METHOD_BEGIN(AbstractLogger.java:698)
at edu.berkeley.cs.jqf.instrument.tracing.SingleSnoop.METHOD_BEGIN(SingleSnoop.java:999)
at randoop.util.ProgressDisplay.run(ProgressDisplay.java)
Hi Rohan,
When I was running the AFL tutorial of JQF and simply replaced the PNG parsing with GIF parsing, feeding afl/testcases/images/gif/not_kitty.gif, I found several uncaught exceptions (attached, haven't reported them yet, feel free if you feel like it). So far so good, here are the 4 gif images:
However, at one point AFLs UI got a little stuck and soon after the hang counter went up 1. A little later another hang was found.
However, after running the hangs with either jqf-repro or a simple Java program that does the same as the driver, both did not result in a hang. One of both resulted in a "crash" (exception), the other was ok and did not result in any result (just ran through).
Do you have any idea what could cause it that hangs really occur (the UI is frozen for a short period) but the hang directory contains the wrong files? Could you try to reproduce?
In general I think the "hang" findings are very important for Java programs. While exceptions ("crashes") are good for code robustness, hangs usually indicate a more severe problem that might lead to denial of service issues.
cheers,
floyd
The pattern:
class T extends S {
public T() { super(new S()); }
}
Is quite troublesome because it is difficult for JQF to distinguish between the call to new S()
and super()
when performing instrumentation in a single pass.
Detecting which expression is the call to super()
is important for JQF because super()
calls cannot be surrounded in a try-catch block, as is done by JQF's instrumentation for almost all other method invocations to correctly track where in the call stack we are in the presence of run-time exceptions.
Determining call stack position is important for performance reasons, in order to tracking code coverage outside the test program, e.g. JVM class loading activity.
super()
and this()
calls cannot be surrounded in a try-catch block because of the JVM's infamous uninitialized this
object security vulnerability. The JVM performs a data-flow analysis when loading every class file to verify that constructors call super
/this
properly before returning.
In JQF, we do not want to perform a full-scale data-flow analysis (because we are lazy), so we instead try to guess which is the call to super()
or this()
in a single pass by looking for the first call to a constructor of the super-class or declaring-class respectively. This of course fails like in the case above when the first such call is an argument to super
/this
itself.
This pattern was found in the wild in Apache Tika's ContentHandlerDecorator, reported by @floyd-fuh in #20.
A suggested fix is to track all NEW
bytecode instructions executed inside a constructor before super
/this
, and to remember them so that we avoid mistaking arguments to super
/this
as calls to super
/this
themselves.
We need a utility script, say jqf_classpath.sh
that expands to the classpath required by programs that want to compile with JQF. Example usage:
javac -cp src:$(jqf/scripts/jqf_classpath.sh) Test.java
I'm getting this error when trying to use jqf-afl-fuzz
:
[me@computer validation-commons]$ ~/tools/jqf/bin/jqf-afl-fuzz -c ./target/test-classes/ com.my-company.products.AA.validation.commons.BB.DecoderFuzzerTest fuzzWhenDecoding
Performing pilot run.... ERROR: Pilot run failed!!!
java.lang.IllegalArgumentException: Could not instantiate a Junit runner for method class com.my-company.products.AA.validation.commons.BB.DecoderFuzzerTest#fuzzWhenDecoding.
at edu.berkeley.cs.jqf.fuzz.junit.GuidedFuzzing.run(GuidedFuzzing.java:172)
at edu.berkeley.cs.jqf.fuzz.junit.GuidedFuzzing.run(GuidedFuzzing.java:126)
at edu.berkeley.cs.jqf.fuzz.junit.GuidedFuzzing.run(GuidedFuzzing.java:96)
at edu.berkeley.cs.jqf.fuzz.repro.ReproDriver.main(ReproDriver.java:64)
The method works fine using the goal jqf:fuzz
of the maven plugin.
It is quite wasteful to run the instrumentation for scripts like jqf-repro
. Allow drivers to turn it off.
I just wanted to let you know that JQF was able to find a bug in Java, that is fixed in version 10: https://bugs.openjdk.java.net/browse/JDK-8135160
I think a common use-case for fuzzers is to run them on remote machines and not just locally from IDE like unit-tests. The current solution is to checkout out the entire code on the remote machine or bring the main jar + test class + jqf jar to a remote machine and use the ZestDriver directly or via the ./jqf-zest
which involves quite a lot of steps.
I thought about adding something like mvn jqf:package -Dclass=classname -Dmethod=method
that would output a jar that can just be run via java -jar fuzzer.jar
what do you think? how feasible this is and what obstacles you think I'll go into if I would add such capability.
In case of 0 coverage the method edu.berkeley.cs.jqf.fuzz.util.Coverage#updateBits
is anyway executing the full for-loop from 0 to 65535.
@rohanpadhye , may be if a 0 check is added before the for loop the computing performance would be improved in the long run
i.e. if (that.counter.nonZeroCount != 0){ for ... }
Latest version available in maven.org is alpha2
:
Shouldn't it be updated to alpha3
?
I have created a very small generator with a very simple test to test the library.
I have tried to follow the existing examples.
I am getting an issue after some fuzzing iterations.
Could you please help me knowing if this is an internal known issue? Or am I doing any mistake in my generator and test implementation?
Please find below the source code of my Generator and my test, followed by the exception stacktrace.
Many thanks again for your help and support!!!
Generator
`
public class SeedsGenerator extends ArrayListGenerator {
public SeedsGenerator() {
super();
}
@Override
public ArrayList<Object> generate(SourceOfRandomness random, GenerationStatus status) {
return getObjectSeeds(random,status);
}
private static final int SHORT_MEAN_NUMBER = 5;
private static final int INT_MEAN_NUMBER = 5;
private static final int LONG_MEAN_NUMBER = 5;
private static final int DOUBLE_MEAN_NUMBER = 5;
private static final int FLOAT_MEAN_NUMBER = 5;
private static final int CHAR_MEAN_NUMBER = 4;
private static final int BOOLEAN_MEAN_NUMBER = 2;
private static final int STRING_MEAN_NUMBER = 2;
private static Generator<String> stringGenerator = new StringGenerator();
private static GeometricDistribution geometricDistribution =
new GeometricDistribution();
private static String makeString(SourceOfRandomness random, GenerationStatus status) {
return stringGenerator.generate(random, status);
}
private SeedsGenerator() {
throw new IllegalAccessError("no instance creation. static access only.");
}
public static ArrayList<Object> getObjectSeeds(SourceOfRandomness randomness, GenerationStatus generationStatus) {
ArrayList<Object> seeds;
seeds = new ArrayList<>();
int numShorts = Math.max(1, geometricDistribution.sampleWithMean(SeedsGenerator.SHORT_MEAN_NUMBER, randomness));
int numInts = Math.max(1, geometricDistribution.sampleWithMean(SeedsGenerator.INT_MEAN_NUMBER, randomness));
int numLongs = Math.max(1, geometricDistribution.sampleWithMean(SeedsGenerator.LONG_MEAN_NUMBER, randomness));
int numStrings = Math.max(1, geometricDistribution.sampleWithMean(SeedsGenerator.STRING_MEAN_NUMBER, randomness));
int numChars = Math.max(1, geometricDistribution.sampleWithMean(SeedsGenerator.CHAR_MEAN_NUMBER, randomness));
int numDoubles = Math.max(1, geometricDistribution.sampleWithMean(SeedsGenerator.DOUBLE_MEAN_NUMBER, randomness));
int numFloats = Math.max(1, geometricDistribution.sampleWithMean(SeedsGenerator.FLOAT_MEAN_NUMBER, randomness));
int numBooleans = Math.max(1, geometricDistribution.sampleWithMean(SeedsGenerator.BOOLEAN_MEAN_NUMBER, randomness));
// in some cases the same object will be generated twice. we don't care about this repeated seed and leave zest and do the magic.
for (int i = 0; i < numShorts; i++) {
seeds.add((short) randomness.nextInt());
}
for (int i = 0; i < numInts; i++) {
seeds.add(randomness.nextInt());
}
for (int i = 0; i < numLongs; i++) {
seeds.add(randomness.nextLong());
}
for (int i = 0; i < numStrings; i++) {
makeString(randomness, generationStatus);
}
for (int i = 0; i < numChars; i++) {
String str = makeString(randomness, generationStatus);
int charPos = randomness.nextInt(0, str.length() - 1);
seeds.add(str.charAt(charPos));
}
for (int i = 0; i < numDoubles; i++) {
seeds.add(randomness.nextDouble());
}
for (int i = 0; i < numFloats; i++) {
seeds.add(randomness.nextFloat());
}
for (int i = 0; i < numBooleans; i++) {
seeds.add(randomness.nextBoolean());
}
return seeds;
}
}
`
test
`import com.pholser.junit.quickcheck.From;
import edu.berkeley.cs.jqf.fuzz.Fuzz;
import edu.berkeley.cs.jqf.fuzz.JQF;
import org.junit.runner.RunWith;
import java.util.List;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
@RunWith(JQF.class)
public class SeedsGeneratorTest {
private static int fuzzIteration = 0;
@Fuzz
public void testSeedsGeneration(@From(SeedsGenerator.class) List<Object> seeds) {
++fuzzIteration;
System.out.println("fuzzIteration = " + fuzzIteration);
assertTrue("seeds should be not null", seeds != null);
assertFalse("seeds list should not be empty", seeds.isEmpty());
assertTrue("seeds list should be more than 4", seeds.size() > 4); // I guiding the generator to generate Lists with more than 4 items.
}
}
`
exception stack trace
Fuzzing stopped due to guidance exception: java.lang.IllegalArgumentException: bad range, 0 > -1 edu.berkeley.cs.jqf.fuzz.guidance.GuidanceException: java.lang.IllegalArgumentException: bad range, 0 > -1 at edu.berkeley.cs.jqf.fuzz.junit.quickcheck.FuzzStatement.evaluate(FuzzStatement.java:159) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at edu.berkeley.cs.jqf.fuzz.junit.GuidedFuzzing.run(GuidedFuzzing.java:184) at edu.berkeley.cs.jqf.fuzz.junit.GuidedFuzzing.run(GuidedFuzzing.java:126) at edu.berkeley.cs.jqf.plugin.FuzzGoal.execute(FuzzGoal.java:207) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) at org.codehaus.classworlds.Launcher.main(Launcher.java:47) Caused by: java.lang.IllegalArgumentException: bad range, 0 > -1 at com.pholser.junit.quickcheck.internal.Ranges.checkRange(Ranges.java:62) at com.pholser.junit.quickcheck.internal.Ranges.choose(Ranges.java:87) at edu.berkeley.cs.jqf.fuzz.junit.quickcheck.FastSourceOfRandomness.fastChooseIntInRange(FastSourceOfRandomness.java:123) at edu.berkeley.cs.jqf.fuzz.junit.quickcheck.FastSourceOfRandomness.nextInt(FastSourceOfRandomness.java:98)
Find suitable macro-benchmarks and write generators and tests for them.
Support an argument for the @Fuzz
annotation so that fuzz results can be reproduced in an IDE without having to run the command-line scripts.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.