Comments (13)
That's it, thank you.
Basically, Ruby is using vfork
natively now.
I just modified and ran a benchmark of POSIX::Spawn
on my computer using ruby:3-slim
Docker image on an Ubuntu 20.04 host. Here's the result:
$ bundle exec rake benchmark
cp tmp/x86_64-linux/posix_spawn_ext/3.1.2/posix_spawn_ext.so lib/posix_spawn_ext.so
/usr/local/bin/ruby -Ilib bin/posix-spawn-benchmark
benchmarking fork/exec vs. posix_spawn over 1000 runs at 100M res
user system total real
fspawn (fork/exec): 0.097599 2.031621 6.810560 ( 6.756999)
pspawn (posix_spawn): 0.006890 0.025629 0.251916 ( 0.244938)
spawn (native): 0.003529 0.032049 0.195628 ( 0.188914)
Child (posix_spawn): 0.037435 0.037484 0.294511 ( 0.275843)
open3 (native): 0.058668 0.037705 0.262761 ( 0.226236)
The open3 implementation is using similar IO::select
as POSIX::Spawn::Child
.
The posix-spawn
gem is worse than native, time-wise.
The gem should not be used in new applications because it doesn't provide advantages over native Ruby implementation anymore. And GitHub + the gem maintainers have moved on.
For the Child
API, I think it is trivial to implement something similar with Open3
.
from posix-spawn.
Writing a simple C program confirms that for some reason, stdout is set to non-blocking on Ruby 3.0, but not on Ruby 2.7:
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
int main(void) {
printf("%d\n", fcntl(STDOUT_FILENO, F_GETFL) & O_NONBLOCK);
return 0;
}
Indeed, unlike Ruby 2.7, Ruby 3.0 makes all pipes non-blocking by default, and as far as I understand it, manually undoes that when passing stdout etc. to child processes with built-in popen/exec etc.
This means that posix-spawn probably has to do the same thing, not only for POSIX::Spawn.popen4
, but probably for POSIX::Spawn.spawn
as well.
from posix-spawn.
Hey, thanks a bunch for the explanation; that answered most of my questions.
However,
from a couple of dialogs ago, it seems that GitHub didn't use the gem anymore and switched back to Process::spawn
.
Made me wonder what's different from now and then when POSIX::Spawn
was created; what was changed in Process::spawn
?
How does it make a comeback?
I don't quite understand low-level details of OS kinds of stuff, and since you considered switching to Process::spawn
too, so I asked in succession because you seem to understand that one too.
from posix-spawn.
does anyone know where I can find discussion about the changes with Process::spawn in ruby 3?
Searching the web, ruby codebase, and ruby changelogs for vfork doesn't come up with anything
- https://www.google.com/search?q=ruby+3+spawn
- https://www.google.com/search?q=ruby+3+vfork
- https://ml.ruby-lang.org/mailman3/hyperkitty/search?count=200&mlist=ruby-core%40ml.ruby-lang.org&q=vfork
This code was committed 9 years ago
https://github.com/ruby/ruby/blame/7b27ad9ad36dbfd2ec6571b0ed15fbc4aa65d460/process.c#L58
If anyone has better search terms for me to use, much appreciate, thanks!
from posix-spawn.
For reference this fork apparently solves the problem for spawn
: jhawthorn@529e214
@jhawthorn Is that patch used at GitHub? If so is there any chance to upstream it?
Regarding popen4
it can be patched directly into the posix-spawn Ruby code.
from posix-spawn.
@catwell we are no longer using this gem. Ruby has had a built-in and efficient Process.spawn
for a long time so I don't think there's any reason to use this gem anymore.
from posix-spawn.
OK, thanks for the answer! I was still using it in part because I prefer its API - see this blobpost by @jnunemaker for an example - but I can switch to native Process.spawn in this case.
I also suggest the maintainers of posix-spawn mark it as officially deprecated in the README.
from posix-spawn.
@catwell
I was late to the party and could not quite follow.
Why did you suggest that the gem should be deprecated? What changed in the built-in Process.spawn
?
What do you suggest in place of POSIX::Spawn::Child
?
from posix-spawn.
@midnight-wonderer I only suggest the gem should be deprecated if the maintainers don't use it anymore and don't intend to fix its bugs.
Process.spawn is not what changed, it's IO (and pipes in particular) that is non-blocking by default. But pipes used for stdin / stderr / stdout should not be non-blocking, so they should be made blocking explicitly.
from posix-spawn.
Not entirely sure that is what he was referring to but Ruby now uses vfork when it can. Like I said that wasn't my reason for using this gem anyway.
from posix-spawn.
Yes, that is probably what I will do now. In the meantime we monkey patched this gem to have it work on Ruby 3 since we wanted to avoid such changes at the same time as our Ruby 3 upgrade.
from posix-spawn.
@jjb Yes the vfork change is older than Ruby 3, it shipped with Ruby 2.2.
POSIX::Spawn
is much older though, it was created for Ruby 1.9, back when fork performance was bad, as said at the top of the README:
So since Ruby 2.2 it is no longer necessary to use POSIX:Spawn to get good fork performance, but it still worked well until Ruby 3.
The breaking change in Ruby 3 is that pipes, used internally by POSIX:Spawn, became non-blocking by default, as explained earlier in the thread. If you're looking for the commit, that change happened here.
from posix-spawn.
Thanks for that info! Wow, interesting - I didn't discover posix-spawn until far after 2.2 and used it everywhere I could.
I did understand that breaking change prompted the discussion in this ticket - what I was looking for was info about the change in ruby which made posix-spawn irrelevant, so I guess back in 2.2. Looks like the release notes cover it! https://www.ruby-lang.org/en/news/2014/12/25/ruby-2-2-0-released/
Experimental support for using vfork(2)....
and looks like it's never used on macos and solaris: https://github.com/ruby/ruby/blob/master/configure.ac
from posix-spawn.
Related Issues (20)
- Windows - 'which' is not recognized as an internal or external command HOT 2
- posix-spawn 0.3.9 is not compatible with ruby 1.8 HOT 1
- Why switch interpreters for cygwin? HOT 2
- Posix::Spawn::Child gets stuck when restarting service
- POSIX::Spawn.` slower than Kernel.` on MRI 2.2 HOT 3
- coompiling posix-spawn on mac failed HOT 5
- gcc error while installing gem on 32bit OS HOT 1
- deprecation warning in ruby 2.4 HOT 4
- posix_spawn_ext.so: undefined symbol: rb_fix2int HOT 6
- Bug with out encoding in Cygwin HOT 2
- gem check posix-spawn has problems HOT 1
- POSIX::Spawn.system wrong return values and stale $? for missing commands HOT 2
- [macOS] POSIX::Spawn.system stale $? for missing commands HOT 1
- Get an error when compiling posix-spawn 0.3.13 HOT 10
- release 0.3.14 containing #83 HOT 4
- Release 0.3.15 for Ruby 2.7 HOT 2
- Unable to install on Termux HOT 5
- Cannot Build on M3 HOT 9
- Is posix-spawn still maintained?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from posix-spawn.