Comments (14)
I just noticed that JqwikStreamSupport.concat
has a comment mentioning that using flatMap
breaks in "recursive scenarios, e.g. ArbitrariesRecursiveTests.ShrinkingTests.complexShrinking
", and this does appear to be true. With flatMap
ArbitrariesRecursiveTests.ShrinkingTests.complexShrinking
seems to get stuck shrinking indefinitely. However, I can't figure out why that is. What's the difference between flatMap
and concat
that causes flatMap
, but not concat
to break this test?
from jqwik.
It's been a while that I went into the rabbit hole of this problem. What I think I remember is that the JDK (and this may depend on the exact version) is doing quite some optimization for flat mapping. One of this optimizations breaks the recursive use case - maybe because some state is being cached that shouldn't be.
As for the SOE: Does it show up independent of available memory?
from jqwik.
Yes, it does show up regardless of available memory. I believe JVM stack size is fixed (the default is 1MB iirc), unless you manually configure your VM to increase stack size.
from jqwik.
Assuming that the flatMap issue is too arcane to solve directly I propose something like this:
public static <T> Stream<T> concat(List<Stream<T>> streams) {
return concat(new ArrayList<>(streams), 0, streams.size());
}
private static <T> Stream<T> concat(List<Stream<T>> streams, int i, int j) {
if (j - i == 0) {
return Stream.empty();
}
if (j - i == 1) {
return streams.get(i);
}
int midpoint = (i + j) / 2;
return Stream.concat(concat(streams, i, midpoint), concat(streams, midpoint, j));
}
This splits the list approximately in half with every recursion, which should reduce recursion depth to approximately log2(n), which grows much slower than the current algorithm.
EDIT: Fixed bug with empty list not working.
from jqwik.
For reference, there was a link clarifying the current implementation: #305 (comment)
I believe it makes sense to add the link to the sources
from jqwik.
That linked article seems to recommend against the current implementation of JqwikStreamSupport.concat
because of the same stack overflow issue that I have encountered.
from jqwik.
For example, it interacts poorly with infinite streams. Calling findAny on the concatenated stream may cause the program to enter an infinite loop
In other words, "balanced" implementation behaves badly with infinite streams, so it is not usable either.
The article recommends https://github.com/TechEmpower/misc/blob/master/concat/src/main/java/rnd/StreamConcatenation.java at the very end
It is a bit much to inline in a blog article, so take a look at StreamConcatenation.java for the source code.
This implementation is similar to Stream.concat(a, b) in that it uses a custom Spliterator, except this implementation handles any number of input streams.
It performs quite well. It does not outperform every other solution in every scenario (flatMap is generally better for very small input streams), but it never performs much worse and it scales nicely with the number and size of the input streams.
@jlink , WDYT of pulling StreamConcatenation.java?
It is not the first time users run into StackOverflowError triggered by concat
.
from jqwik.
For example, it interacts poorly with infinite streams. Calling findAny on the concatenated stream may cause the program to enter an infinite loop
That quote is talking about flatMap, not the balanced approach.
I do agree that StreamConcatenation.java is probably the best option, provided that we trust the author of that article that it is actually correct.
from jqwik.
Also fwiw I ran complexShrinking with the balanced approach and the test passed.
from jqwik.
FYI, I have disabled shrinking because I am using stateful testing, and virtually all my operations modify state, so the shrinker does not work.
At the same time, when the shrinker was active it did fail with OOM and SOE from time to time. So improving concat seems helpful.
I wonder what the odds are for .concat
improvement to be accepted in OpenJDK.
from jqwik.
WDYT of pulling StreamConcatenation.java?
Think I will do just that. License seems to allow it.
from jqwik.
Fixed in 5a2a12a
from jqwik.
I wonder what the odds are for
.concat
improvement to be accepted in OpenJDK.
Have you ever tried to do a PR there? How's the process anyway?
from jqwik.
Released in 1.8.2
from jqwik.
Related Issues (20)
- Shrinking of large arrays uses a very large amount of memory HOT 11
- question: any way to imperatively summon a pre-configured arbitrary? HOT 15
- Make nested groups work for JDK >= 18 HOT 31
- Combinators.combine.as nullability of arguments HOT 4
- Can't figure out how to create an Arbitrary I want HOT 10
- Bug: Arbitraries.strings().uniqueChars() will sometimes shrink to values with duplicate chars HOT 5
- Bug: Uniform Distribution of Integers does not work if range is >= Integer.MAX_VALUE HOT 2
- Kotlin K2 Support HOT 23
- Time Module: Support generating java.sql.Timestamp
- adding `jqwik-kotlin` causes existing tests written in java to fail with NPE HOT 18
- Potential idea for running `SAMPLE_FIRST`/`SAMPLE_ONLY` for stateful tests HOT 2
- Lifecycle method called twice when subclassing HOT 4
- Cannot use Arbitrary.generator() outside jqwik HOT 6
- Is Arbitrary<@Nullable T> correct/intended under JSpecify semantics? HOT 32
- TypeUsage example from user guide fails in 1.8.x HOT 8
- Right Way to Integrate jqwik with DataFaker HOT 23
- Generic @Provide methods are sensitive to the type variables' names HOT 3
- Allowing Sampling of Arbitraries with Injected Seed HOT 4
- TestEngine with ID 'jqwik' failed to discover tests [BUG] HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jqwik.