Re-institute async mode#644
Conversation
|
@joshleblanc thanks for this amazing job! I installed the fork in an app hosted on Heroku that servers webhooks from a live application. I've run it as a separate process though (as a worker). My So far it looks very good: 
I used to run the worker before without recurring jobs because it would consume all RAM. Now, even with recurring jobs it takes ~2x less RAM! Thanks for sharing this work! |
|
I'd like to help get this finished up, so that non-forking implementations like JRuby can use Solid Queue. Is it complete enough that I could try to run on JRuby now and see what's missing? Do you have an example I could try? |
|
See #679 for my recent request to have a non-forking implementation. |
|
Just give me a bit.more time on the JEP-380 socket PR. I can talk to Cruby, C, or any other language using jruby as the layer via Linux Sockets. Separation of concerns. Let both interpreters run as independent processes and communicate via unix sockets and message pack. I should having it ready by this evening/tomorrow morning. |
|
Hey folks! Going to work on this one soon. Thanks for the help so far. |
rosa
force-pushed
the
async-mode
branch
2 times, most recently
from
adb57b2 to
f588105
Compare
|
Hey @joshleblanc, @ka8725, I've pushed a bunch of changes to bring this to the finish line. If you have time, could you try running this in your apps to see if there are any problems? I'll do the same on Friday with one of our apps, as I've refactored a bunch of things that could also affect the regular fork mode. Then, if things are looking good, I'll merge this and release a new version. No rush! Just whenever you have time. |
|
Upgraded the gem in the app, it's not under load now. So need time to collect data. Will respond tomorrow. |
It looks great: |
|
Seems to be working well on my deploy |
|
That's great! Thank you so much 🙏🏻 We're also running this in HEY now and so far we haven't had any problems. I'll update the docs and will merge this next week. |
rosa
force-pushed
the
async-mode
branch
4 times, most recently
from
030cc49 to
ae2d84d
Compare
Which was removed in rails#308 refactor: remove unused supervisor require statements fix: use 6075 instead of $ to get correct process ID in solid_queue plugin fix: don't modify array during iteration chore: remove duplicate code chore: logging refactor: remove async thread creation option from runnable process
The tests were removed in 4b3483a This also restores support for "standalone" async mode, which means the Solid Queue's supervisor running on its own process, via `bin/jobs`, but having all other processes (workers, dispatchers, schedulers) run as threads instead of processes. When Solid Queue is run via the Puma plugin, it won't be considered standalone, and it won't need to handle signals and act on them.
|
Ok, I fixed a few small issues with the async supervisor and made other small changes. Now all tests pass, and I've updated HEY to use the last version of this. I'll run it for a couple of days, and then I'll merge and publish a new version. |
|
FYI I have been running this branch in production at work and at RubyEvents (much smaller scale than Hey) for several days and it works great. In both cases it was almost a 50% memory reduction |
|
@adrienpoly that's awesome to hear! 🥳 Thanks a lot for testing 🙏🏻😊 |
For consistency with the other `bin/jobs` options.
Otherwise `supervise` will block and we won't be able to stop it externally (from the Puma plugin).
Standalone supervisors run this from the signal handler, so for non-standalone we need to make sure this is run after shutting down.
|
Quite I look away for a few weeks and it's all done! Excellent work @rosa and @joshleblanc! Can you recommend benchmarks I can run in JRuby to profile and contribute some additional optimizations? |
This branch's puma.rb adds `solid_queue_mode :async if ENV["SOLID_QUEUE_IN_PUMA"]`, but the lockfile resolved to solid_queue 1.1.4, which has no such DSL method. Booting in production crashes immediately with: config/puma.rb:47:in 'Puma::DSL#_load_from': undefined method 'solid_queue_mode' for an instance of Puma::DSL (NoMethodError) Async supervisor mode was re-introduced in solid_queue 1.3.0 (rails/solid_queue#644); the earlier 0.4 implementation was removed in 0.7. Pinning to ~> 1.3 documents the minimum that matches this branch's puma.rb and updates the lockfile to 1.4.0. Caught by deploying a marketplace smoke-test app built from this branch + #22 + #24 to Render free tier.
* Run Solid Queue in async mode on Render free tier Companion to #22 (WEB_CONCURRENCY=0). Free tier (512MB) can't fit SQ's default fork mode, which spawns supervisor + worker + dispatcher + scheduler subprocesses for ~460MB of RSS overhead on top of Puma. The container OOM-cycles every ~7-15 min — the deploy still looks "live" but HTTP returns 502s during each restart window. `solid_queue_mode :async` (documented switch in lib/puma/plugin/solid_queue.rb of the gem) tells the plugin to run worker/dispatcher/scheduler as threads inside the Puma master process. The 4 SQ subprocesses collapse into ~50MB of thread overhead in Puma. Container RSS drops from ~530MB (cycling into OOM) to a flat ~300MB. Trade-off per SQ's own docs ("Only use async if you know what you're doing and have strong reasons to"): - Less isolation between SQ and Puma. A leaky/hung SQ thread affects request serving. For low-traffic student projects this is the right call; revisit if upgrading to a paid plan. - No inter-process parallelism for jobs. Free tier is already 0.1 vCPU, so this isn't real lost throughput. - Recurring tasks and concurrency controls still work unchanged. The DB pool bump (5 → 8) is non-optional: previously the pool was sized for Puma's 3 request threads only; now the same process also runs 3 SQ worker threads + dispatcher + scheduler. Under any load the old pool=5 would throw ConnectionTimeoutError. The 5 -> 8 default still respects DB_POOL env override for projects that need finer control. Diagnosed and validated on raghubetina/aplace: raghubetina/aplace#58 — RSS metric showed the predicted plateau at ~300MB with zero growth after deploy. * Pin solid_queue ~> 1.3 for solid_queue_mode :async support This branch's puma.rb adds `solid_queue_mode :async if ENV["SOLID_QUEUE_IN_PUMA"]`, but the lockfile resolved to solid_queue 1.1.4, which has no such DSL method. Booting in production crashes immediately with: config/puma.rb:47:in 'Puma::DSL#_load_from': undefined method 'solid_queue_mode' for an instance of Puma::DSL (NoMethodError) Async supervisor mode was re-introduced in solid_queue 1.3.0 (rails/solid_queue#644); the earlier 0.4 implementation was removed in 0.7. Pinning to ~> 1.3 documents the minimum that matches this branch's puma.rb and updates the lockfile to 1.4.0. Caught by deploying a marketplace smoke-test app built from this branch + #22 + #24 to Render free tier.
6 participants
This is less of a "this is ready to be merged", and more of an effort to reinvigorate conversation around this feature. I cobbled this together as a proof of concept because I simply couldn't use the multi-process approach on the server I'm trying to deploy to. This is related to #343, and tries to naively reimplement this
I originally stumbled on this feature because I was testing Kamal on a 1GB VPS. The initial deploy would work, however there wasn't enough resource headroom for the second deploy. I narrowed this down to SQ using ~100MB per process, essentially taking up half the available memory on the system.
After implementing this change, I saw the application memory drop from ~700MB to ~270MB.
The usage is simply placing this in puma.rb
I currently have this deployed to a 1GB droplet - the application is running quickly, background jobs are running, and redeploys are succeeding