Exceptions when sampling discrete distribution about powerlaw

[jeffalstott]

Thanks for using powerlaw! You said "sometimes" you get the exception. Does your specific example *always *cause the problem, or only sometimes?

…

On Mon, Feb 18, 2019 at 5:20 AM matejgroma ***@***.***> wrote: I am having a problem obtaining random samples from a discrete power law distribution. Sometimes, I get an exception instead of the desired value. Code to reproduce the issue: import powerlaw xmax=100 exponent=1.1 dist=powerlaw.Power_Law(xmin=1,xmax=xmax,discrete=True,parameters=[exponent],discrete_approximation="xmax") print(dist.generate_random(n=1,estimate_discrete=False)) The exception: Traceback (most recent call last): File "sample_ips.py", line 7, in <module> print(dist.generate_random(n=1,estimate_discrete=False)) File "/usr/lib/python3.7/site-packages/powerlaw.py", line 1094, in generate_random x = array([self._double_search_discrete(R) for R in r], File "/usr/lib/python3.7/site-packages/powerlaw.py", line 1094, in <listcomp> x = array([self._double_search_discrete(R) for R in r], File "/usr/lib/python3.7/site-packages/powerlaw.py", line 1101, in _double_search_discrete while self.ccdf(data=[x2]) >= (1 - r): File "/usr/lib/python3.7/site-packages/powerlaw.py", line 727, in ccdf return self.cdf(data=data, survival=survival) File "/usr/lib/python3.7/site-packages/powerlaw.py", line 782, in cdf if isnan(min(CDF)): File "/usr/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2618, in amin initial=initial) File "/usr/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction return ufunc.reduce(obj, axis, dtype, out, **passkwargs) ValueError: zero-size array to reduction operation minimum which has no identity — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#69>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6_r33UPlHYj6AKXfQlLmCEAPmCiXMZks5vOn6EgaJpZM4bAifc> .

When I run the example in a while loop, like this:

import powerlaw
xmax=100
exponent=1.1
dist=powerlaw.Power_Law(xmin=1,xmax=xmax,discrete=True,parameters=[exponent],discrete_approximation="xmax")
while True:
    print(dist.generate_random(n=1,estimate_discrete=False))

it crashes after a while. Alternatively, more samples also (almost always) cause the problem:
print(dist.generate_random(n=100,estimate_discrete=False))

[jeffalstott]

I don't have a computer at the moment with powerlaw installed/installable, but reading through the code it looks like the doc string for .generate_random states "If xmax is present, it is currently ignored." So, in principle this function shouldn't even generate the thing that you want, anyway. I would just generate data from a Power_Law instance without an xmax, then cut out all the data above your desired xmax.

It appears the bug is that .generate_random calls on _double_search_discrete, which does binary search across values of x to match a random number in [0,1] with the CCDF of the Power_Law instance (thus producing x proportional to its probability in the Power_Law). My guess is that the problem arises when there's xmax and the binary search overshoots past xmax, yielding a candidate x value the cdf function can't handle; it trims out everything that's outside of xmax and xmin with trim_to_range, yielding an empty list and later producing the error you see.

So, the problem is that we don't actually know how to directly generate synthetic data from a discrete power law with an xmax. If there's an elegant solution, I'll happily implement it. In the meantime, please confirm that my suggestion above actually works for you. Thanks!

If I understood correctly, it is not possible to use this library to sample the distribution with given xmax?
Thank you for your help.

[jeffalstott]

I would just generate data from a Power_Law instance without an xmax, then cut out all the data above your desired xmax.
That'll get you what you want.

Okay, thank you very much.