Concept for production deployment?

#46 · open · 8 comments

View on GitHub ↗

jensb

Hi, I have setup ruby-clock successfully in one project and am now ready to deploy it to a production environment - which, in my case, is an Apache Passenger webserver. I am using 'mina' for deployment, which is easier and faster than Capistrano (which I used before), but that shouldn't matter. For production environments, what is the concept to also run ruby-clock when the Passenger server boots up my application? How can I automate this, and automate setting it up during deployment? Or do I need to write (possibly system-wide) systemd init scripts to run my cronjobs? Any ideas?

Comments

jjb

I’m unfamiliar with how passenger does this stuff. Do they have documentation about running things like sidekick or other background job processors? For ruby-clock it would be very analogous. If you can point me to those docs, then I can suggest something.

jensb

Most other gems like Sidekiq assume they are running as a systemd job - not necessarily on the same host as the webserver. We could document how to set this up. Passenger has startup hooks which we could use. I did not (yet) test this but I asked ChatGPT to imagine a Rails bootup script for me which demonstrates the concept. :-) Here's the idea: ```ruby # config/environment.rb require_relative 'application' Rails.application.initialize! CLOCK_LOCK_FILE = "/tmp/clock_process.lock" CLOCK_LOG_FILE = "log/ruby-clock.log" if defined?(PhusionPassenger) PhusionPassenger.on_event(:starting_worker_process) do |forked| if forked if File.exist?(CLOCK_LOCK_FILE) clock_pid = File.read(CLOCK_LOCK_FILE).to_i if( clock_pid > 0 && Process.kill(0, clock_pid) rescue false ) next else File.delete(CLOCK_LOCK_FILE) end end File.open(CLOCK_LOCK_FILE, 'w') {|file| file.write(Process.pid) } @clock_pid = spawn("bundle exec clock", out: CLOCK_LOG_FILE, err: CLOCK_LOG_FILE) Process.detach(@clock_pid) end end # Optional. This will attempt clock shutdown whenever a worker exits, so maybe not ideal PhusionPassenger.on_event(:stopping_worker_process) do if @clock_pid && File.exist?(CLOCK_LOCK_FILE) Process.kill("TERM", @clock_pid) @clock_pid = nil File.delete(CLOCK_LOCK_FILE) if File.exist?(CLOCK_LOCK_FILE) end end end ``` The upside is there will be a restart attempt per spawned web worker so if ruby-clock crashes, it'll be restarted. The downside is it's hard to shut down ruby-clock in a controlled way because there's no "the last worker has exited" event in Passenger. Better would probably be an implementation in config.ru, which is started just once, even by Passenger: ```ruby # config.ru require_relative 'config/environment' use Rack::CommonLogger CLOCK_LOCK_FILE = "/tmp/clock_process.lock" CLOCK_LOG_FILE = "log/clock.log" if File.exist?(CLOCK_LOCK_FILE) clock_pid = File.read(CLOCK_LOCK_FILE).to_i if clock_pid > 0 && Process.kill(0, clock_pid) # rescue false puts "Clock process is already running with PID #{clock_pid}" else File.delete(CLOCK_LOCK_FILE) end end unless File.exist?(CLOCK_LOCK_FILE) @clock_pid = spawn("bundle exec clock", out: CLOCK_LOG_FILE, err: CLOCK_LOG_FILE) Process.detach(@clock_pid) File.open(CLOCK_LOCK_FILE, 'w') { |file| file.write(@clock_pid) } puts "Started clock process with PID #{@clock_pid}" end run Rails.application ``` I briefly tested the second approach with the Puma web server and it seems to work: ``` => Booting Puma => Rails 7.2.1.2 application starting in development => Run `bin/rails server --help` for more startup options Started clock process with PID 2483774 Puma starting in single mode... * Puma version: 6.4.3 (ruby 3.3.5-p100) ("The Eagle of Durango") * Min threads: 5 * Max threads: 5 * Environment: development * PID: 2483637 * Listening on http://0.0.0.0:3000 Use Ctrl-C to stop ```

jjb

> Most other gems like Sidekiq assume they are running as a systemd job - not necessarily on the same host as the webserver. ruby-clock assumes to be run analogously as sidekiq - are you avoiding this to save money, or was that not clear? i don't know anything about passenger so i can't give much feedback on the proposed code, other then that you should aim to run ruby-clock before forking and to not run it after forking. something you can do is set up a test job ```ruby every('5 seconds') do puts "hello p#{Process.pid} t#{Thread.current.object_id}" end ``` and then boot up with multiple processes and threads and make sure that there is only once instance of ruby clock running

jensb

It's not about saving money, it's about staying simple and contained, even on production. Also, about having multiple independent Rails apps with different Clockfiles on the same machine, which is perfectly possible with Passenger, without having to resort to complex setups like Docker. I already have a working Clockfile and putting the startup code in the `config.ru` works fine locally. I'll see how this performs on production.

jjb

gotcha! Does your project run a background job processor? How does it run it? `Process.kill(0, clock_pid)` is new to me, happy to learn about it. i usually find myself hacking around with ```ruby `ps aux | grep #{pid}`.strip.blank? ``` etc 😅

jensb

Great :-) I tried to avoid grepping for PIDs since this may result in false positives (e.g. a `grep 239` also matches 2391 and 42396). Right now I'd also like to avoid the complexity of running a background job processor (which again would require more work when deploying and keeping Rails apps separated within the same Apache Passenger setup just by UID). This is why I was asking in the first place. If my app becomes popular enough to warrant multiple instances and possibly virtualization, then I'll consider it.

jensb

Follow-up question: is there a way I can force / trigger a restart of a running ruby-clock instance, e.g. after pushing a code update? Preferably without having to restart the web server?

jjb

No specific restart mechanism, but if you send `TERM` or `INT` to it (typically `TERM` in a prod/staging environment, `INT` is what control-c sends) it will shut down cleanly. Then, you can just start it again the same way you did before. https://github.com/jjb/ruby-clock/blob/e7e83393286561de7ccc0d62822beb6f2d262020/lib/ruby-clock.rb#L27-L46