perone / pyevolve Goto Github PK

Hi,

I have some trouble using pyevolve's multiprocessing. Specifically, I have issue with threads. This is the error that I get :

...
File "build/bdist.linux-x86_64/egg/pyevolve/GSimpleGA.py", line 824, in evolve
File "build/bdist.linux-x86_64/egg/pyevolve/GSimpleGA.py", line 668, in step
File "build/bdist.linux-x86_64/egg/pyevolve/GPopulation.py", line 365, in evaluate
File "/usr/lib64/python2.7/multiprocessing/init.py", line 232, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild)
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 184, in init
self._result_handler.start()
File "/usr/lib64/python2.7/threading.py", line 746, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread

I don't think that my code is involved in this error, but just in case :

eval func =>
def eval_gen(seq):

len_seq = len(seq)
j = 0
state = [0,0,0,0]
pmin = float("+inf")
while j < len_seq and not(is_terminal(state)) :
    a = 5 if seq[j] == 0 else -5 
    state = compute_next_state(a,state)
    cost = d(state)
    if cost <= pmin:
        pmin = cost
    j += 1
return pmin + j

GA =>

ga = GSimpleGA.GSimpleGA(genome,interactiveMode=False)
ga.setGenerations(50)
ga.setMinimax(Consts.minimaxType["minimize"])
ga.setCrossoverRate(0.5)
ga.setMutationRate(0.1)
ga.setPopulationSize(20)
ga.setMultiProcessing(True)
genome.evaluator.set(eval_gen)
ga.evolve()
seq = ga.bestIndividual()

Thanks for you help!

The documentation page http://pyevolve.sourceforge.net/intro.html#glossary-concepts reads

Pyevolve have an interactive mode, you can ...

Instead, it should red

Pyevolve has an interactive mode, you can ...

Note the have/has change

Add Lorenzo Riano to the contributors list in the Sphinx documentation.

Looking at the code of GTournamentSelector, it seems like it is not considering if maximizing or minimizing when selecting from the candidates.

if population.sortType == Consts.sortType["scaled"]:
choosen = max(tournament_pool, key=key_fitness_score)
else:
choosen = max(tournament_pool, key=key_raw_score)

Since in GASimpleGA.py to use elitism it considers the min/max param, it seems like this one should do it as well and use min instead if minimizing.

Here is sample code:

        mom = G1DList(3)
        mom.genomeList = [1, 2, 3]
        dad = G1DList(3)
        dad.genomeList = [4, 5, 6]
        Crossovers.G1DListCrossoverOX(None, mom=mom, dad=dad, count=1)

It fails with AssertionError in pyevolve/Crossovers.py, line 216, assert listSize == len(sister) (first is 3, second is 4).

Hi,
it would help a lot to tell the user community about production quality releases if you would consider tagging released versions. In Debian we would observe what you consider release quality code.
Thanks a lot for considering, Andreas.

I want to use the Pyevolve in python3, and the I download the package follow the Pull request instruction #79 then I find the bug, what can I do ?

Traceback (most recent call last):
File "setup.py", line 2, in
from pyevolve import version, author
File "/mnt/e/sub/Pyevolve-master/Pyevolve-master/pyevolve/init.py", line 22, in
import pyevolve.Consts
File "/mnt/e/sub/Pyevolve-master/Pyevolve-master/pyevolve/Consts.py", line 369, in
import Scaling
ModuleNotFoundError: No module named 'Scaling`

How does the elitism work in this project ? The elite subject is copied to the next generation or is used as a parent? I'm asking because when I set the elitism in True, I expected to see always the same minium fitnes over the generations, but is not the case.

Now all classes in the project are old-style classes:

class ClassName:
    pass

Better option is to use new-style classes:

class ClassName(object):
    pass

Old style classes has different MRO, different syntax and so on.
Actually, there is no reason to use old-style classes (http://stackoverflow.com/questions/54867/)

pyevolve.Mutators:766

if genome[i][j] == 0: genome.setItem(which_y, which_x, 1)

Variables i and j are undefined.

We need to run http://svn.python.org/projects/python/trunk/Tools/scripts/reindent.py into the entire Pyevolve source code in order to convert the 3-space tab to the 4-space tab.

Report by Billy Earney:

I'm trying to execute my evaluation functions in parallel, but several when I do this, pyevolve creates many processes that are not terminated automatically. I have to kill them myself. A few test runs produced over 200 zombie processes..

Below is a URL to a script I downloaded, and when I execute it, I get the same result (many processes are created). I've also downloaded the the source from the git repository, just in case this was a bug that has been fixed recently.

http://pyevolve.sourceforge.net/wordpress/?tag=parallel

Here's the basic info about the version of python and pyevolve I'm using:
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import pyevolve
pyevolve.version
'0.6'

It would be great if python 3 is supported. I made some changes as a first attempt

https://github.com/greole/Pyevolve/tree/master
maybe it could be merged into a python3-development branch.

There is a lot of copy-paste since all mutators works pretty much the same way.
Since mutators are covered with tests, they can (and must) be refactored.

it would be nice to get automated tests

if there is no-one against it' i would like to start using pyt.est on my own changes

Currently, Pyevolve doesn't have a resume mechanism where the user can stop the algorithm and re-start again later from the point it stopped.

The code can certain use some very small performance changes, which after summing up all changes it should improve performance slightly.
I'm talking about thinks like, use correct structures, more pytonic code, etc..

For example:

Using a list for just TWO elements: cuts = [rand_randint(1, len(gMom)-1), rand_randint(1, len(gMom)-1)]
Might be better just having two separate vars or a tuple.
This is in Crossovers

return True if rand_random() <= p else False in Util.py. Like really? Could be just return rand_random() <= p

Do not re calculate lengths every time, if you are planning on using len(something) many times, assign it to a variable first and then use it.

Define functions that are used in big loops, as a local variable, since doing var1.var2.somefunc on a loop is expensive due to look ups every time.

Having the following:

ga = GSimpleGA.GSimpleGA(genome)
ga.selector.set(Selectors.GTournamentSelector)

There 'tournamentPool' is not working.

I have added it on both, the genome:
genome = G1DList.G1DList(N)
genome.setParams(rangemin=0,
rangemax=2,
tournamentPool = 5
)

And ga.setParams(tournamentPool = 5)

However when GTournamentSelector tries to get that param, it can't find anywhere. I have debugged the code and it is quite messy, the same get/set operations are in GenomeBase, GPopulation, and GSimpleGA, some params are fetched from one class, some from others, you don't really know where to set each one of them.

GTreeNodeBase.__init__ method accepts childs argument, but it's inheritor GTreeNode overloads __init__ without this argument.
GTreeNode.__init__ method must be changed to allow this.

Now they accept one argument mandatory argument - genome - and args, but genome is not used. It should explicitly take mom and dad arguments.

The elitism implementation seems a little strange to me. It looks like it's comparing the best in the current generation with the best in the previous generation, the 2nd best in the current generation with the 2nd best in the previous generation, etc. one-by-one. Thus it seems possible that the top candidate from the previous generation is not carried over (not better than the best in the current generation) but the second best candidate from the previous is carried over (better than the 2nd best in the current generation). But, I might also be reading the code incorrectly.

There are some commented code in the project and it shouldn't be there since we are using git. It should be deleted.

We need to change the MigrationScheme to make it more abstract than it is today.

See comment in the pull request:
#11 (comment)

You might consider providing an option for using random.sample() to choose candidates for tournament selection. In my experience it's much faster, plus it won't pick duplicate candidates.

The code would look like:
tournament_pool = random.sample(population, pool_size)

Add a list of papers citing the main SIGEVOlution paper to the documentation. People can use it to know how PyEvolve is being used in research. Also links to code/research groups that uses PyEvolve will be useful, so people will know who's using PyEvolve and how.

Prepare for v.0.6 final milestone release.

Sphinx docs are great, but there are to many of them. I mean, look: 10 lines of code and 20 lines of docs - ordinary case in the project.
This dramatically reduces maintainability - to many scrolling per action rate.
Good idea is to keep small doc strings where it's required and move other info to docs/ folder.
Doctests are not a problem also since we have unit tests now.

The version of pyevolve available through pip doesn't join or close the processing pool in GPopulation.evaluate(). This results in an EXPLOSION of RAM usage, so much so that it becomes almost unusable when the multiprocessing flag is True.

The current code on the master branch appears to fix this issue, all that needs to be done is to update the code available in the repos. I think that this is a big enough issue that it merits a new patch.

-- Eric

I'm running PyEvolve on a computer with 72 CPU cores. I've got a relatively complex objective function, that runs without issue without multiprocessing. The objective function uses the Threading module to run an external Fortran program three times for each individual of the population (the individual is a set of options for the program, and I'm trying to find options that will yield a certain result). The score is then based on results from the Fortran program, which are parsed in the objective function from an output log file. The Pyevolve GA settings themselves are relatively standard.

Here's the problem: if I run the GA with a population that is either close to 72 or larger than 72, PyEvolve crashes when evaluating the first finished population, giving the following message:

  File "/home/PERSONAL/raskar/programs/pyevolve_git/pyevolve/GSimpleGA.py", line 777, in evolve
    self.internalPop.evaluate()
  File "/home/PERSONAL/raskar/programs/pyevolve_git/pyevolve/GPopulation.py", line 393, in evaluate
    results = proc_pool.map(multiprocessing_eval_full, self.internalPop)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
IndexError: list index out of range

Unfortunately, I have a large space to search, so it is probably necessary to use a large population.

The error seems similar to the one reported previously here:
https://groups.google.com/forum/#!topic/pyevolve/lPWq_rHEe8M

A fix was suggested there, by Simon Liddington:

I have found that if I modify evaluate() in GPopulation.py to use a
chunksize that means each CPU is given one chunk then it works:
proc_pool.map(multiprocessing_eval, self.internalPop, 1 +
(len(self.internalPop) / CPU_COUNT))

However, making this change doesn't fix my problem.

I'm using the latest git version of pyevolve.

Does anyone have any idea about what might be causing this problem, and what can be done to fix it?

It's very easy to end up at the sourceforge site, especially since the Pyevolve-project on PyPI still points there. A simple link and explanation on the front saying the project has moved to github would be very helpful.

The Mutators.G1DListMutatorIntegerRange only seems to accept one mutation probability, which seems like it is the one that decides if a value on the chromosome should be mutated or not.

However, shouldn't be there also another probability that defines if the mutation should be applied at all or not? I think that's how other libraries work but I have no idea if that's an improvement or not.

We need to fix the examples in the trunk of the development version to reflect the new 0.6 version.
The TSP really needs an entire refactoring.

ERROR: Command errored out with exit status 1:
 command: 'C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\acarlton\\AppData\\Local\\Temp\\pip-install-vow2i4bj\\pyevolve\\setup.py'"'"'; __file__='"'"'C:\\Users\\acarlton\\AppData\\Local\\Temp\\pip-install-vow2i4bj\\pyevolve\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\acarlton\AppData\Local\Temp\pip-install-vow2i4bj\pyevolve\pip-egg-info'
     cwd: C:\Users\acarlton\AppData\Local\Temp\pip-install-vow2i4bj\pyevolve\
Complete output (9 lines):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\acarlton\AppData\Local\Temp\pip-install-vow2i4bj\pyevolve\setup.py", line 2, in <module>
    from pyevolve import __version__, __author__
  File "C:\Users\acarlton\AppData\Local\Temp\pip-install-vow2i4bj\pyevolve\pyevolve\__init__.py", line 22, in <module>
    import pyevolve.Consts
  File "C:\Users\acarlton\AppData\Local\Temp\pip-install-vow2i4bj\pyevolve\pyevolve\Consts.py", line 369, in <module>
    import Scaling
ModuleNotFoundError: No module named 'Scaling'
----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

FunctionSlot.clear() does not clear function weights list

Since we have tests now, it make sense for robots tot launch them, not humans.
Its free and highly recommended since everyone will see that the project is stable (or not).
Add coveralls (https://coveralls.io) to CI cycle is great idea to - we'll be able to monitor coverage of the project (and pull requests)!

Any thoughts on why the default number for number of individuals to be replaced is set so "low" at 1 individual (out of a default population of 80?). Wouldn't a higher default number lead to improved performance for most cases "out-of-the-box"?

Need to clear the weights list in clear() ~ line 116 in FunctionSlot.py, ex.

del self.funcWeights[:]

Selectors.GTournamentSelectorAlternative causes a crash.

"Gen. 0 (0.00%): Max/Min/Avg Fitness(Raw) [0.00(0.00)/0.00(0.00)/0.00(0.00)]
Traceback (most recent call last):
File "C:\Users\Cristiano\Desktop\ej1\ej1.py", line 143, in
ga.evolve(freq_stats=GENERATIONS / 10)
File "C:\Python27\lib\site-packages\pyevolve\GSimpleGA.py", line 824, in evolv
e
if self.step(): break #exit if the number of generations is equal to the max
. number of gens.
File "C:\Python27\lib\site-packages\pyevolve\GSimpleGA.py", line 630, in step
crossover_empty = self.select(popID=self.currentGeneration).crossover.isEmpt
y()
AttributeError: 'NoneType' object has no attribute 'crossover'
"