standardgalactic / forkoff Goto Github PK
View Code? Open in Web Editor NEWThis project forked from ahoward/forkoff
brain-dead simple parallel processing for ruby
This project forked from ahoward/forkoff
brain-dead simple parallel processing for ruby
NAME forkoff SYNOPSIS brain-dead simple parallel processing for ruby URI http://rubyforge.org/projects/codeforpeople http://github.com/ahoward/forkoff INSTALL gem install forkoff DESCRIPTION forkoff works for any enumerable object, iterating a code block to run in a child process and collecting the results. forkoff can limit the number of child processes which is, by default, 2. SAMPLES <========< samples/a.rb >========> ~ > cat samples/a.rb # forkoff makes it trivial to do parallel processing with ruby, the following # prints out each word in a separate process # require 'forkoff' %w( hey you ).forkoff!{|word| puts "#{ word } from #{ Process.pid }"} ~ > ruby samples/a.rb hey from 7907 you from 7908 <========< samples/b.rb >========> ~ > cat samples/b.rb # for example, this takes only 4 seconds or so to complete (8 iterations # running in two processes = twice as fast) # require 'forkoff' a = Time.now.to_f results = (0..7).forkoff do |i| sleep 1 i ** 2 end b = Time.now.to_f elapsed = b - a puts "elapsed: #{ elapsed }" puts "results: #{ results.inspect }" ~ > ruby samples/b.rb elapsed: 4.19184589385986 results: [0, 1, 4, 9, 16, 25, 36, 49] <========< samples/c.rb >========> ~ > cat samples/c.rb # forkoff does *NOT* spawn processes in batches, waiting for each batch to # complete. rather, it keeps a certain number of processes busy until all # results have been gathered. in otherwords the following will ensure that 3 # processes are running at all times, until the list is complete. note that # the following will take about 3 seconds to run (3 sets of 3 @ 1 second). # require 'forkoff' pid = Process.pid a = Time.now.to_f pstrees = %w( a b c d e f g h i ).forkoff! :processes => 3 do |letter| sleep 1 { letter => ` pstree -l 2 #{ pid } ` } end b = Time.now.to_f puts puts "pid: #{ pid }" puts "elapsed: #{ b - a }" puts require 'yaml' pstrees.each do |pstree| y pstree end ~ > ruby samples/c.rb pid: 7922 elapsed: 3.37899208068848 --- a: | -+- 07922 ahoward ruby -Ilib samples/c.rb |-+- 07923 ahoward ruby -Ilib samples/c.rb |-+- 07924 ahoward (ruby) \-+- 07925 ahoward ruby -Ilib samples/c.rb --- b: | -+- 07922 ahoward ruby -Ilib samples/c.rb |-+- 07923 ahoward ruby -Ilib samples/c.rb |-+- 07924 ahoward ruby -Ilib samples/c.rb \-+- 07925 ahoward ruby -Ilib samples/c.rb --- c: | -+- 07922 ahoward ruby -Ilib samples/c.rb |-+- 07923 ahoward ruby -Ilib samples/c.rb |-+- 07924 ahoward (ruby) \-+- 07925 ahoward ruby -Ilib samples/c.rb --- d: | -+- 07922 ahoward ruby -Ilib samples/c.rb |-+- 07932 ahoward ruby -Ilib samples/c.rb |--- 07933 ahoward ruby -Ilib samples/c.rb \--- 07934 ahoward ruby -Ilib samples/c.rb --- e: | -+- 07922 ahoward ruby -Ilib samples/c.rb |--- 07932 ahoward (ruby) |-+- 07933 ahoward ruby -Ilib samples/c.rb \-+- 07934 ahoward (ruby) --- f: | -+- 07922 ahoward ruby -Ilib samples/c.rb |--- 07932 ahoward (ruby) |-+- 07933 ahoward ruby -Ilib samples/c.rb \-+- 07934 ahoward ruby -Ilib samples/c.rb --- g: | -+- 07922 ahoward ruby -Ilib samples/c.rb |-+- 07941 ahoward ruby -Ilib samples/c.rb |--- 07942 ahoward ruby -Ilib samples/c.rb \--- 07943 ahoward ruby -Ilib samples/c.rb --- h: | -+- 07922 ahoward ruby -Ilib samples/c.rb |-+- 07941 ahoward (ruby) |-+- 07942 ahoward ruby -Ilib samples/c.rb \--- 07943 ahoward ruby -Ilib samples/c.rb --- i: | -+- 07922 ahoward ruby -Ilib samples/c.rb |--- 07942 ahoward (ruby) \-+- 07943 ahoward ruby -Ilib samples/c.rb <========< samples/d.rb >========> ~ > cat samples/d.rb # forkoff supports two strategies of reading the result from the child: via # pipe (the default) or via file. you can select which to use using the # :strategy option. # require 'forkoff' %w( hey you guys ).forkoff :strategy => :file do |word| puts "#{ word } from #{ Process.pid }" end ~ > ruby samples/d.rb hey from 7953 you from 7954 guys from 7955 HISTORY 1.1.0 - move to a model with one work queue and signals sent from consumers to producer to noitify ready state. this let's smaller jobs race through a single process even while a larger job may have one sub-process bound up. incorporates a fix from http://github.com/fredrikj/forkoff which meant some processes would lag behind when jobs didn't have similar execution times. 1.0.0 - move to github 0.0.4 - code re-org - add :strategy option - default number of processes is 2, not 8 0.0.1 - updated to use producer threds pushing onto a SizedQueue for each consumer channel. in this way the producers do not build up a massize parllel data structure but provide data to the consumers only as fast as they can fork and proccess it. basically for a 4 process run you'll end up with 4 channels of size 1 between 4 produces and 4 consumers, each consumer is a thread popping of jobs, forking, and yielding results. - removed use of Queue for capturing the output. now it's simply an array of arrays which removed some sync overhead. - you can configure the number of processes globally with Forkoff.default['proccess'] = 4 - you can now pass either an options hash forkoff( :processes => 2 ) ... or plain vanilla number forkoff( 2 ) ... to the forkoff call - default number of processes is 8, not 2 0.0.0 initial version
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.