Add support for container policies. (#5)

- Introduce default failure tolerance of 5 failures in 60 seconds.

Author: samuel-williams-shopify

async-service.gemspec

@@ -27,6 +27,6 @@ Gem::Specification.new do |spec|
 	spec.required_ruby_version = ">= 3.2"
 
 	spec.add_dependency "async"
-	spec.add_dependency "async-container", "~> 0.29"
+	spec.add_dependency "async-container", "~> 0.33"
 	spec.add_dependency "string-format", "~> 0.2"
 end

bake/async/service/controller.rb

@@ -10,9 +10,6 @@ def initialize(context)
 end
 
 def run
-	# Warm up the Ruby process by preloading gems and running GC.
-	Async::Service::Controller.warmup
-	
 	controller.run
 end
 
@@ -21,5 +18,5 @@ def run
 def controller
 	configuration = context.lookup("async:service:configuration").instance.configuration
 
-	return Async::Service::Controller.new(configuration.services)
+	return configuration.make_controller
 end

context/index.yaml

@@ -10,6 +10,10 @@ files:
   title: Getting Started
   description: This guide explains how to get started with `async-service` to create
     and run services in Ruby.
+- path: policies.md
+  title: Container Policies
+  description: This guide explains how to configure container policies for your services
+    and understand the default failure handling behavior.
 - path: service-architecture.md
   title: Service Architecture
   description: This guide explains the key architectural components of `async-service`

context/policies.md

@@ -0,0 +1,87 @@
+# Container Policies
+
+This guide explains how to configure container policies for your services and understand the default failure handling behavior.
+
+## Default Failure Handling
+
+All services use {ruby Async::Service::Policy::DEFAULT} which monitors failure rates and stops the container when failures exceed a threshold.
+
+**Default threshold:** 6 failures in 60 seconds (0.1 failures per second).
+
+This means:
+- Services can tolerate occasional failures and transient issues.
+- More than 6 failures in any 60-second window stops the container.
+- Prevents services from restart-looping indefinitely when fundamentally broken.
+
+This fail-fast behavior is appropriate for orchestrated environments (Kubernetes, systemd) where the orchestrator will restart the entire service.
+
+### Why This Default?
+
+Without failure monitoring, a broken service with `restart: true` would restart indefinitely, wasting resources. The default policy:
+
+- **Catches problems quickly**: Broken services stop within 10-20 seconds.
+- **Prevents resource waste**: Doesn't keep trying to start services that will never succeed.
+- **Enables orchestrator recovery**: Systemd/Kubernetes can restart the whole process with a clean state.
+- **Detects environmental issues**: Bad hardware, corrupted pre-fork state, or system-level problems can't be fixed by restarting children - the entire service needs to be restarted (potentially on different hardware).
+- **Signals clear failure**: Exit code indicates the service couldn't maintain healthy operation.
+
+## Configuring Policies
+
+Use `container_policy` in your service configuration to customize failure handling:
+
+``` ruby
+# config/service.rb
+
+# More lenient: allow 5 failures per minute:
+container_policy Async::Service::Policy.new(maximum_failures: 5, window: 60)
+
+service "web" do
+	# Your service configuration.
+end
+
+service "worker" do
+	# Also uses the same policy.
+end
+```
+
+The policy applies to **all services** in the configuration file.
+
+### Choosing a Threshold
+
+Consider your service characteristics:
+
+**Strict (catch problems immediately):**
+``` ruby
+container_policy Async::Service::Policy.new(maximum_failures: 1, window: 5)
+```
+
+**Balanced (tolerate transient issues):**
+``` ruby
+container_policy Async::Service::Policy.new(maximum_failures: 5, window: 60)
+```
+
+**Lenient (allow many retries):**
+``` ruby
+container_policy Async::Service::Policy.new(maximum_failures: 20, window: 60)
+```
+
+Factors to consider:
+- **Traffic volume**: High-traffic services may have more absolute failures.
+- **Error types**: Some errors are transient (network timeouts, rate limits).
+- **Dependencies**: Upstream services may need time to recover.
+- **Deployment environment**: Kubernetes/systemd handle restarts, local dev doesn't.
+
+## Per-Container Policy Instances
+
+The `container_policy` method accepts a block that's evaluated **each time a container is created**:
+
+``` ruby
+# config/service.rb
+container_policy do
+	# This block is called for EACH container created
+	# Each container gets its own policy instance with fresh state
+	Async::Service::Policy.new(maximum_failures: 5, window: 60)
+end
+```
+
+If your policy is tracking per-container state, this will ensure each container has new policy with clean state.

guides/links.yaml

@@ -1,8 +1,10 @@
 getting-started:
   order: 1
-service-architecture:
+policies:
   order: 2
-best-practices:
+service-architecture:
   order: 3
-deployment:
+best-practices:
   order: 4
+deployment:
+  order: 5

guides/policies/readme.md

@@ -0,0 +1,87 @@
+# Container Policies
+
+This guide explains how to configure container policies for your services and understand the default failure handling behavior.
+
+## Default Failure Handling
+
+All services use {ruby Async::Service::Policy::DEFAULT} which monitors failure rates and stops the container when failures exceed a threshold.
+
+**Default threshold:** 6 failures in 60 seconds (0.1 failures per second).
+
+This means:
+- Services can tolerate occasional failures and transient issues.
+- More than 6 failures in any 60-second window stops the container.
+- Prevents services from restart-looping indefinitely when fundamentally broken.
+
+This fail-fast behavior is appropriate for orchestrated environments (Kubernetes, systemd) where the orchestrator will restart the entire service.
+
+### Why This Default?
+
+Without failure monitoring, a broken service with `restart: true` would restart indefinitely, wasting resources. The default policy:
+
+- **Catches problems quickly**: Broken services stop within 10-20 seconds.
+- **Prevents resource waste**: Doesn't keep trying to start services that will never succeed.
+- **Enables orchestrator recovery**: Systemd/Kubernetes can restart the whole process with a clean state.
+- **Detects environmental issues**: Bad hardware, corrupted pre-fork state, or system-level problems can't be fixed by restarting children - the entire service needs to be restarted (potentially on different hardware).
+- **Signals clear failure**: Exit code indicates the service couldn't maintain healthy operation.
+
+## Configuring Policies
+
+Use `container_policy` in your service configuration to customize failure handling:
+
+``` ruby
+# config/service.rb
+
+# More lenient: allow 5 failures per minute:
+container_policy Async::Service::Policy.new(maximum_failures: 5, window: 60)
+
+service "web" do
+	# Your service configuration.
+end
+
+service "worker" do
+	# Also uses the same policy.
+end
+```
+
+The policy applies to **all services** in the configuration file.
+
+### Choosing a Threshold
+
+Consider your service characteristics:
+
+**Strict (catch problems immediately):**
+``` ruby
+container_policy Async::Service::Policy.new(maximum_failures: 1, window: 5)
+```
+
+**Balanced (tolerate transient issues):**
+``` ruby
+container_policy Async::Service::Policy.new(maximum_failures: 5, window: 60)
+```
+
+**Lenient (allow many retries):**
+``` ruby
+container_policy Async::Service::Policy.new(maximum_failures: 20, window: 60)
+```
+
+Factors to consider:
+- **Traffic volume**: High-traffic services may have more absolute failures.
+- **Error types**: Some errors are transient (network timeouts, rate limits).
+- **Dependencies**: Upstream services may need time to recover.
+- **Deployment environment**: Kubernetes/systemd handle restarts, local dev doesn't.
+
+## Per-Container Policy Instances
+
+The `container_policy` method accepts a block that's evaluated **each time a container is created**:
+
+``` ruby
+# config/service.rb
+container_policy do
+	# This block is called for EACH container created
+	# Each container gets its own policy instance with fresh state
+	Async::Service::Policy.new(maximum_failures: 5, window: 60)
+end
+```
+
+If your policy is tracking per-container state, this will ensure each container has new policy with clean state.

lib/async/service/configuration.rb

@@ -52,11 +52,15 @@ def self.for(*environments)
 			end
 
 			# Initialize an empty configuration.
-			def initialize(environments = [])
+			# @parameter environments [Array] Environment instances.
+			# @parameter container_policy [Proc] Optional proc that returns a policy for container lifecycle management.
+			def initialize(environments = [], container_policy: nil)
 				@environments = environments
+				@container_policy = container_policy
 			end
 
 			attr :environments
+			attr_accessor :container_policy
 
 			# Check if the configuration is empty.
 			# @returns [Boolean] True if no environments are configured.
@@ -84,11 +88,22 @@ def services(implementing: nil)
 
 			# Create a controller for the configured services.
 			#
+			# @parameter container_policy [Proc] A proc that returns the policy to use for managing child lifecycle events.
+			# @parameter options [Hash] Additional options passed to the controller.
 			# @returns [Controller] A controller that can be used to start/stop services.
-			def controller(**options)
-				Controller.new(self.services(**options).to_a)
+			def make_controller(container_policy: @container_policy, implementing: nil, **options)
+				controller = Controller.new(self.services(implementing: implementing).to_a, **options)
+				
+				if container_policy
+					controller.define_singleton_method(:make_policy, &container_policy)
+				end
+				
+				return controller
 			end
 
+			# Alias for backwards compatibility.
+			alias controller make_controller
+			
 			# Add the environment to the configuration.
 			def add(environment)
 				@environments << environment

lib/async/service/controller.rb

@@ -4,6 +4,7 @@
 # Copyright, 2024-2025, by Samuel Williams.
 
 require "async/container/controller"
+require_relative "policy"
 
 module Async
 	module Service
@@ -13,32 +14,11 @@ module Service
 		# within containers. It extends Async::Container::Controller to provide
 		# service-specific functionality.
 		class Controller < Async::Container::Controller
-			# Warm up the Ruby process by preloading gems and running GC.
-			def self.warmup
-				begin
-					require "bundler"
-					Bundler.require(:preload)
-				rescue Bundler::GemfileNotFound, LoadError
-					# Ignore.
-				end
-				
-				if ::Process.respond_to?(:warmup)
-					::Process.warmup
-				elsif ::GC.respond_to?(:compact)
-					3.times{::GC.start}
-					::GC.compact
-				end
-			end
-			
 			# Run a configuration of services.
 			# @parameter configuration [Configuration] The service configuration to run.
 			# @parameter options [Hash] Additional options for the controller.
 			def self.run(configuration, **options)
-				controller = Async::Service::Controller.new(configuration.services.to_a, **options)
-				
-				self.warmup
-				
-				controller.run
+				configuration.make_controller(**options).run
 			end
 
 			# Create a controller for the given services.
@@ -58,17 +38,43 @@ def initialize(services, **options)
 				@services = services
 			end
 
+			# Warm up the Ruby process by preloading gems, running GC, and compacting memory.
+			# This reduces startup latency and improves copy-on-write efficiency.
+			def warmup
+				begin
+					require "bundler"
+					Bundler.require(:preload)
+				rescue Bundler::GemfileNotFound, LoadError
+					# Ignore.
+				end
+				
+				if ::Process.respond_to?(:warmup)
+					::Process.warmup
+				elsif ::GC.respond_to?(:compact)
+					3.times{::GC.start}
+					::GC.compact
+				end
+			end
+			
 			# All the services associated with this controller.
 			# @attribute [Array(Async::Service::Generic)]
 			attr :services
 
+			# Create a policy for managing child lifecycle events.
+			# @returns [Policy] The service-level policy with failure rate monitoring.
+			def make_policy
+				Policy::DEFAULT
+			end
+			
 			# Start all named services.
 			def start
 				@services.each do |service|
 					service.start
 				end
 
 				super
+				
+				self.warmup
 			end
 
 			# Setup all services into the given container.

lib/async/service/loader.rb

@@ -58,6 +58,18 @@ def service(name = nil, **options, &block)
 
 				@configuration.add(self.environment(**options, &block))
 			end
+			
+			# Set the container policy for all services in this configuration.
+			# Can be called with either an argument or a block.
+			# @parameter value [Async::Container::Policy] The policy to use for managing child lifecycle events.
+			# @parameter block [Proc] A block that returns a policy instance.
+			def container_policy(value = nil, &block)
+				if @configuration.container_policy
+					Console.warn(self, "Container policy is already set, overriding previous value!")
+				end
+				
+				@configuration.container_policy = block_given? ? block : proc{value}
+			end
 		end
 	end
 end