Implement embed=process_instances

Process instances can be embedded into process resources; the parameter
embed=process_instances is available for the following API endpoints:
- GET /v3/processes/:guid?embed=process_instances
- GET /v3/apps/:guid/processes/:type?embed=process_instances
- GET /v3/processes?embed=process_instances
- GET /v3/apps/:guid/processes?embed=process_instances

A new ProcessShowMessage has been introduced; ProcessesListMessage has
been extended. Both messages allow the 'embed' key that is checked with
the EmbedParamValidator.

The process output is enhanced by the EmbedProcessInstancesDecorator
that fetches the required information from Diego and adds a field
'process_instances' to the resource.

Process resources now also contain a link to 'process_instances'.

API docs have been extended with the new process instances
pseudo-resource and a description of the 'embed' parameter.

Author: philippthun

app/controllers/v3/processes_controller.rb

@@ -2,6 +2,7 @@
 require 'presenters/v3/process_presenter'
 require 'presenters/v3/process_stats_presenter'
 require 'presenters/v3/process_instances_presenter'
+require 'decorators/embed_process_instances_decorator'
 require 'cloud_controller/paging/pagination_options'
 require 'actions/process_delete'
 require 'fetchers/process_list_fetcher'
@@ -10,6 +11,7 @@
 require 'actions/process_terminate'
 require 'actions/process_update'
 require 'messages/process_scale_message'
+require 'messages/process_show_message'
 require 'messages/process_update_message'
 require 'messages/processes_list_message'
 require 'controllers/v3/mixins/app_sub_resource'
@@ -41,17 +43,30 @@ def index
                 end
     end
 
+    decorators = []
+    decorators << EmbedProcessInstancesDecorator if EmbedProcessInstancesDecorator.match?(message.embed)
+
     render status: :ok, json: Presenters::V3::PaginatedListPresenter.new(
       presenter: Presenters::V3::ProcessPresenter,
       paginated_result: SequelPaginator.new.get_page(dataset, message.try(:pagination_options)),
       path: base_url(resource: 'processes'),
-      message: message
+      message: message,
+      decorators: decorators
     )
   end
 
   def show
-    # TODO
-    render status: :ok, json: Presenters::V3::ProcessPresenter.new(@process, show_secrets: permission_queryer.can_read_secrets_in_space?(@space.id, @space.organization_id))
+    message = ProcessShowMessage.from_params(query_params)
+    invalid_param!(message.errors.full_messages) unless message.valid?
+
+    decorators = []
+    decorators << EmbedProcessInstancesDecorator if EmbedProcessInstancesDecorator.match?(message.embed)
+
+    render status: :ok, json: Presenters::V3::ProcessPresenter.new(
+      @process,
+      show_secrets: permission_queryer.can_read_secrets_in_space?(@space.id, @space.organization_id),
+      decorators: decorators
+    )
   end
 
   def update

app/decorators/embed_process_instances_decorator.rb

@@ -0,0 +1,48 @@
+module VCAP::CloudController
+  class EmbedProcessInstancesDecorator
+    class << self
+      def match?(embed)
+        embed&.include?('process_instances')
+      end
+
+      def decorate(hash, processes)
+        instances_reporters = CloudController::DependencyLocator.instance.instances_reporters
+        instances = instances_reporters.instances_for_processes(processes)
+
+        if hash.key?(:resources)
+          # Decorate PaginatedListPresenter
+          processes.each do |process|
+            resource_index = hash[:resources].find_index { |resource| resource[:guid] == process.guid }
+            next unless resource_index # Should not happen...
+
+            hash[:resources][resource_index] = embed_process_instances(hash[:resources][resource_index], process_instances(instances, process.guid))
+          end
+        else
+          # Decorate ProcessPresenter
+          hash = embed_process_instances(hash, process_instances(instances, hash[:guid]))
+        end
+
+        hash
+      end
+
+      private
+
+      def process_instances(instances, process_guid)
+        instances[process_guid]&.map do |index, instance|
+          {
+            index: index,
+            state: instance[:state],
+            since: instance[:since]
+          }
+        end || []
+      end
+
+      def embed_process_instances(resource_hash, process_instances)
+        hash_as_array = resource_hash.to_a
+        before_relationships = hash_as_array.index { |k, _| k == :relationships } || hash_as_array.length
+        hash_as_array.insert(before_relationships, [:process_instances, process_instances])
+        hash_as_array.to_h
+      end
+    end
+  end
+end

app/messages/base_message.rb

@@ -139,6 +139,22 @@ def validate(record)
       end
     end
 
+    class EmbedParamValidator < ActiveModel::Validator
+      def validate(record)
+        return unless record.requested?(:embed)
+
+        key_counts = Hash.new(0)
+        record.embed.each do |embed_candidate|
+          if options[:valid_values].member?(embed_candidate)
+            key_counts[embed_candidate] += 1
+            record.errors.add(:base, message: "Duplicate embedded resource: '#{embed_candidate}'") if key_counts[embed_candidate] == 2
+          else
+            record.errors.add(:base, message: "Invalid embedded resource: '#{embed_candidate}'. Valid embedded resources are: '#{options[:valid_values].join("', '")}'")
+          end
+        end
+      end
+    end
+
     class LifecycleTypeParamValidator < ActiveModel::Validator
       def validate(record)
         return unless record.requested?(:lifecycle_type)

app/messages/process_show_message.rb

@@ -0,0 +1,14 @@
+require 'messages/base_message'
+
+module VCAP::CloudController
+  class ProcessShowMessage < BaseMessage
+    register_allowed_keys [:embed]
+
+    validates_with NoAdditionalParamsValidator
+    validates_with EmbedParamValidator, valid_values: ['process_instances']
+
+    def self.from_params(params)
+      super(params, %w[embed])
+    end
+  end
+end

app/messages/processes_list_message.rb

@@ -8,15 +8,19 @@ class ProcessesListMessage < MetadataListMessage
       space_guids
       organization_guids
       app_guids
+      embed
     ]
 
     validates_with NoAdditionalParamsValidator # from BaseMessage
+    validates_with EmbedParamValidator, valid_values: ['process_instances']
 
+    validates :space_guids, array: true, allow_nil: true
+    validates :organization_guids, array: true, allow_nil: true
     validates :app_guids, array: true, allow_nil: true
     validate :app_nested_request, if: -> { app_guid.present? }
 
     def self.from_params(params)
-      super(params, %w[types space_guids organization_guids app_guids])
+      super(params, %w[types space_guids organization_guids app_guids embed])
     end
 
     def to_param_hash

app/presenters/v3/process_presenter.rb

@@ -14,7 +14,8 @@ def to_hash
 
           readiness_health_check_data = { invocation_timeout: process.readiness_health_check_invocation_timeout, interval: process.readiness_health_check_interval }
           readiness_health_check_data[:endpoint] = process.readiness_health_check_http_endpoint if process.readiness_health_check_type == HealthCheckTypes::HTTP
-          {
+
+          hash = {
             guid: process.guid,
             created_at: process.created_at,
             updated_at: process.updated_at,
@@ -44,6 +45,8 @@ def to_hash
             },
             links: build_links
           }
+
+          @decorators.reduce(hash) { |memo, d| d.decorate(memo, [process]) }
         end
 
         private
@@ -66,7 +69,8 @@ def build_links
             scale: { href: url_builder.build_url(path: "/v3/processes/#{process.guid}/actions/scale"), method: 'POST' },
             app: { href: url_builder.build_url(path: "/v3/apps/#{process.app_guid}") },
             space: { href: url_builder.build_url(path: "/v3/spaces/#{process.space_guid}") },
-            stats: { href: url_builder.build_url(path: "/v3/processes/#{process.guid}/stats") }
+            stats: { href: url_builder.build_url(path: "/v3/processes/#{process.guid}/stats") },
+            process_instances: { href: url_builder.build_url(path: "/v3/processes/#{process.guid}/process_instances") }
           }
         end
       end

docs/v3/source/includes/api_resources/_processes.erb

@@ -55,6 +55,9 @@
     },
     "stats": {
       "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82/stats"
+    },
+    "process_instances": {
+      "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82/process_instances"
     }
   }
 }
@@ -136,6 +139,9 @@
         },
         "stats": {
           "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82/stats"
+        },
+        "process_instances": {
+          "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82/process_instances"
         }
       }
     },
@@ -194,6 +200,9 @@
         },
         "stats": {
           "href": "https://api.example.org/v3/processes/3fccacd9-4b02-4b96-8d02-8e865865e9eb/stats"
+        },
+        "process_instances": {
+          "href": "https://api.example.org/v3/processes/3fccacd9-4b02-4b96-8d02-8e865865e9eb/process_instances"
         }
       }
     }
@@ -311,3 +320,123 @@
   "details": null
 }
 <% end %>
+
+<% content_for :single_process_instance do %>
+{
+  "index": 0,
+  "state": "RUNNING",
+  "since": 123456789
+}
+<% end %>
+
+<% content_for :process_instances do %>
+{
+  "resources": [
+    {
+      "index": 0,
+      "state": "RUNNING",
+      "since": 123456789
+    },
+    {
+      "index": 1,
+      "state": "STARTING",
+      "since": 123
+    },
+    {
+      "index": 2,
+      "state": "DOWN",
+      "since": 456
+    }
+  ],
+  "links": {
+    "self": {
+      "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82/process_instances"
+    },
+    "process": {
+      "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82"
+    }
+  }
+}
+<% end %>
+
+<% content_for :single_process_with_embedded_process_instances do %>
+{
+  "guid": "6a901b7c-9417-4dc1-8189-d3234aa0ab82",
+  "type": "web",
+  "command": "rackup",
+  "user": "vcap",
+  "instances": 5,
+  "memory_in_mb": 256,
+  "disk_in_mb": 1024,
+  "log_rate_limit_in_bytes_per_second": 1024,
+  "health_check": {
+    "type": "port",
+    "data": {
+      "timeout": null
+    }
+  },
+  "readiness_health_check": {
+    "type": "process",
+    "data": {
+      "invocation_timeout": null
+    }
+  },
+  "process_instances": [
+    {
+      "index": 0,
+      "state": "RUNNING",
+      "since": 123456789
+    },
+    {
+      "index": 1,
+      "state": "STARTING",
+      "since": 123
+    },
+    {
+      "index": 2,
+      "state": "DOWN",
+      "since": 456
+    }
+  ],
+  "relationships": {
+    "app": {
+      "data": {
+        "guid": "ccc25a0f-c8f4-4b39-9f1b-de9f328d0ee5"
+      }
+    },
+    "revision": {
+      "data": {
+        "guid": "885735b5-aea4-4cf5-8e44-961af0e41920"
+      }
+    }
+  },
+  "metadata": {
+    "labels": {},
+    "annotations": {}
+  },
+  "created_at": "2016-03-23T18:48:22Z",
+  "updated_at": "2016-03-23T18:48:42Z",
+  "version": "e9df685c-0464-4aa7-b5f0-8ed843077c13",
+  "links": {
+    "self": {
+      "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82"
+    },
+    "scale": {
+      "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82/actions/scale",
+      "method": "POST"
+    },
+    "app": {
+      "href": "https://api.example.org/v3/apps/ccc25a0f-c8f4-4b39-9f1b-de9f328d0ee5"
+    },
+    "space": {
+      "href": "https://api.example.org/v3/spaces/2f35885d-0c9d-4423-83ad-fd05066f8576"
+    },
+    "stats": {
+      "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82/stats"
+    },
+    "process_instances": {
+      "href": "https://api.example.org/v3/processes/6a901b7c-9417-4dc1-8189-d3234aa0ab82/process_instances"
+    }
+  }
+}
+<% end %>

docs/v3/source/includes/concepts/_embed.md.erb

@@ -0,0 +1,20 @@
+## Embed
+
+The `embed` parameter allows clients to fetch resources and include information of related pseudo-resources directly in the response. This is different from the [`include`](#include) parameter, which is used for fetching parent resources.
+
+**Important:** The `embed` feature is specifically designed for pseudo-resources that, by design, do not have their own list endpoint. These are resources that are tightly coupled to their parent resource and only make sense in the context of that parent. While they may have a GET endpoint to retrieve them individually for a specific parent, they cannot be listed independently across all parents.
+
+For example, `process_instances` is a pseudo-resource that provides a lightweight view of process instance states. It can be embedded into process resources to avoid making additional API calls, but you cannot list all process instances across all processes in the system.
+
+Developers may choose to use the `embed` feature to reduce the number of API calls. The embed query param can be used with a single resource or a list of resources.
+
+### Resources with embed
+
+The following resources can take an `embed` parameter:
+
+Resource | Allowed values
+-------- | --------------
+**processes** | `process_instances`
+**processes/[:guid]** | `process_instances`
+**apps/[:guid]/processes** | `process_instances`
+**apps/[:guid]/processes/[:type]** | `process_instances`

docs/v3/source/includes/resources/processes/_get.md.erb

@@ -25,6 +25,33 @@ Content-Type: application/json
 `GET /v3/processes/:guid` <br>
 `GET /v3/apps/:guid/processes/:type`
 
+#### Query parameters
+
+Name | Type | Description
+---- | ---- | ------------
+**embed** | _string_ | Comma-delimited list of resources to embed in the response. Valid values are `process_instances`. See [embed](#embed) for more details.
+
+```
+Example Request with Embedded Process Instances
+```
+
+```shell
+curl "https://api.example.org/v3/processes/[guid]?embed=process_instances" \
+  -X GET \
+  -H "Authorization: bearer [token]"
+```
+
+```
+Example Response with Embedded Process Instances
+```
+
+```http
+HTTP/1.1 200 OK
+Content-Type: application/json
+
+<%= yield_content :single_process_with_embedded_process_instances %>
+```
+
 #### Permitted roles
 Role | Notes |
 --- | ---