ewangilligan / tdapnrg

TDAPNRG

Dependancies:

• numpy
• scipy
• Cython
• scitkit-tda
• matplotlib.
• scikit-learn
• plotly
• GF2Matrix

Cython should be the only one required before installation. Due to setuptools not specifying order of installation, you may need to install Cython before setup, which can be done by running

pip install Cython

This may also occur for numpy.

Windows users: If you are using a Windows machine, you will also need to install MinGW on your system.

To setup for development use:

# TDAPNRG/python-implementation/
python setup.py develop

Or for install:

# TDAPNRG/python-implementation/
python setup.py install

Running with Docker

To build:

#path/to/TDAPNRG
sudo docker build --tag=randology .

To run as a single container, the run.sh shell script is provided for convenience. This handles the locations that files need to be mounted.

sudo bash ./run.sh relative/path/to/output_directory relative/path/to/config_file.json

For infomation on writing config files, see config-syntax.md

The output of the program will then be stored in the output directory mounted.

The random sequence files for running can be found here.

tdapnrg's Issues

Pipeline Specification

I am opening this issue following our email so we can all define the spec we want for the PRNG evaluation pipeline and discuss each architecture.

Ideally, initially @EwanGilligan and @zaklogician can give a rough overview of the desired/expected input/outputs and then maybe myself and Rodrigo can chip in and comment as well.

Docker Build Error

Steps to reproduce:

1. Clone the repository with git clone https://github.com/EwanGilligan/TDAPNRG.git
2. Issue sudo docker build --tag=randology . directly or via shell script in the repository directory.

What happens?

The build process runs until Step 7/17 : RUN pip install numpy, then fails with a socket timeout
exception (see stderr below).

What did you expect to happen?

The build process terminates successfully.

What is your environment?

Docker version 18.09.7 build 2d0083d, installed from the official apt repositories on Linux VM 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux; running on a Oracle VirtualBox 6 host.

Terminal output (stdio/stderr):

Step 7/17 : RUN pip install numpy
 ---> Running in 0e0eb2432660
Collecting numpy
  Downloading https://files.pythonhosted.org/packages/ac/36/325b27ef698684c38b1fe2e546e2e7ef9cecd7037bcdb35c87efec4356af/numpy-1.17.2.zip (6.5MB)
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 397, in _error_catcher
    yield
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 479, in read
    data = self._fp.read(amt)
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 62, in read
    data = self.__fp.read(amt)
  File "/usr/local/lib/python3.7/http/client.py", line 457, in read
    n = self.readinto(b)
  File "/usr/local/lib/python3.7/http/client.py", line 501, in readinto
    n = self.fp.readinto(b)
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 345, in run
    resolver.resolve(requirement_set)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 199, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/download.py", line 1064, in unpack_url
    progress_bar=progress_bar
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/download.py", line 924, in unpack_http_url
    progress_bar)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/download.py", line 1152, in _download_http_url
    _download_url(resp, link, content_file, hashes, progress_bar)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/download.py", line 861, in _download_url
    hashes.check_against_chunks(downloaded_chunks)
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/utils/hashes.py", line 75, in check_against_chunks
    for chunk in chunks:
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/download.py", line 829, in written_chunks
    for chunk in chunks:
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/utils/ui.py", line 156, in iter
    for x in it:
  File "/usr/local/lib/python3.7/site-packages/pip/_internal/download.py", line 818, in resp_read
    decode_content=False):
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 531, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 496, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/pip/_vendor/urllib3/response.py", line 402, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
The command '/bin/sh -c pip install numpy' returned a non-zero code: 2

RFC: Can we identify generators based on test output?

In our discussion last Friday, we have agreed to proceed by attempting to answer the titular question.

We had the following sub-questions. How many times do we need to invoke a specific generator before the hypercube test can tell it apart from a source of true random numbers (w.h.p.)? Can we use the number of invocations to identify the generator, or the family of the generator?

In the hypercube test, we use the generator to sample points from the unit hypercube, and we calculate the persistent homology of those points that fall into a smaller hypercube inside the unit hypercube. Recall that he hypercube test takes the following three test parameters:

• dimension: The dimension d of the hypercube to sample points from. Each point of a d-dimensional unit hypercube consists of d coordinates taking values in the interval [0,1], so generating one point requires d invocations of the generator.

• scale: The scale s is the side length of the smaller hypercube whose persistent homology we calculate.

• number of points: The number of points n that we sample from the smaller (!) hypercube.

The scale parameter s is required because the calculation of persistent homology has poor asymptotic behavior with respect to n. If we want large sample sizes, we need to calculate in a smaller cube. Fortunately, a source of true random numbers (a sequence of independent, identically, uniformly distributed random variables) is scale-invariant in this sense: the Betti numbers should have the same distribution in the entire hypercube and in any subhypercube.

It's not straightforward to determine how many times we'll invoke a given generator before running the test. We can estimate this for a true source of random numbers: by using the fact that a random point falls into a unit subhypercube with probability s^d, we get that generating n points should require about nd/(s^d) invocations of a true random number generator.

Idea: instead of counting the number of invocations, we could track the distribution of Betti numbers returned by a fixed test. We can easily compare these distributions using the two-sample Kolmogorov–Smirnov test, and with some luck we shall find that generator from some families have characteristic distributions.