Skip to content

Commit 54ef682

Browse files
beyhanbeyhan
authored
Merge pull request #1251 from FloThinksPi-Forks/rfc-41
[RFC] Provide custom stacks functionality
2 parents 12eb31d + d1ff32e commit 54ef682

2 files changed

Lines changed: 364 additions & 0 deletions

File tree

‎toc/rfc/rfc-draft-provide-custom-stacks-functionality.md‎

Lines changed: 364 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,364 @@
1+
# Meta
2+
3+
- Name: Provide optional Custom Stacks Functionality in Cloud Foundry
4+
- Start Date: 2025-06-24
5+
- Author(s): @FloThinksPi
6+
- Status: Draft
7+
- RFC Pull Request: [community#1220](https://github.com/cloudfoundry/community/pull/1220)
8+
9+
## Summary
10+
11+
With the deprecation of the CFLinuxfs3 stack it became obvious that
12+
removing unsupported stacks from an existing and heavily used CF
13+
Foundation comes with massive problems. Ultimately a stack cannot be
14+
removed without all depending apps being migrated to a new stack,
15+
otherwise downtimes of applications that rely on the removed stack will
16+
occur. This RFC proposes improvements in CF to shift this unavailability
17+
towards lifecycle operations and not actual app downtime - making it a
18+
more pleasant experience for CF users and operators alike.
19+
This RFC proposes a way for CF Users that cannot follow a stack removal process as described in [RFC-0045](https://github.com/cloudfoundry/community/blob/main/toc/rfc/rfc-0045-enhance-stack-handling.md) to take over the responsibility for their stack and run it as a custom stack, while not needing to mimick the CF staging behaviour and building an own docker application to be able to run with a depricated/locked/disabled/removed stack.
20+
It also opens up possibilities for CF Users to not rely on the CF Foundations stacks but rather use their own stacks to gain independence and release the dependency on the CF Community\'s stack release cadence deliberately on the users choice.
21+
22+
## Table of Contents
23+
24+
- [Meta](#meta)
25+
- [Summary](#summary)
26+
- [Problem](#problem)
27+
- [Usage](#usage)
28+
- [Delivery](#delivery)
29+
- [Adoption Timelines](#adoption-timelines)
30+
- [Motivation](#motivation)
31+
- [Proposal](#proposal)
32+
- [Bring your own Stack](#bring-your-own-stack)
33+
- [CF API Changes](#cf-api-changes)
34+
- [Diego BBS API](#diego-bbs-api)
35+
- [Providing a stack as remote container image](#providing-a-stack-as-remote-container-image)
36+
- [CF Tasks](#cf-tasks)
37+
- [CF Sidecars](#cf-sidecars)
38+
- [Compatibility Documentation](#compatibility-documentation)
39+
- [Wrap Up](#wrap-up)
40+
- [Positive](#positive)
41+
- [Negative](#negative)
42+
43+
## Problem
44+
45+
In the current CF implementation, apps run on so-called Stacks which
46+
provide the base operating system.
47+
48+
Stacks are based on ubuntu and thus bound to ubuntu's [LTS release
49+
cadence and Support](https://ubuntu.com/about/release-cycle). With
50+
Canonical\'s stop of standard security maintenance for ubuntu
51+
18.04(Bionic Beaver) as a consequence also the CFLinuxFS3 stack could
52+
not be maintained anymore and was deprecated. A successor CFLinuxFS4 was
53+
offered based on ubuntu 22.04(Jammy Jellyfish).
54+
55+
Stacks are used for the staging of an application meaning buildpacks run
56+
on the stack to produce the droplet. This in return means that:
57+
58+
- The buildpack is stack specific meaning it can/will break on an
59+
incompatible stack change like a major version bump
60+
61+
- The droplet the buildpack produces is also stack specific meaning it
62+
CAN just be instantiated to become an app instance when combining the droplet
63+
with the same stack it was built with at execution time
64+
65+
![Current Stack Usage](rfc-draft-provide-custom-stacks-functionality/current_stack_usage.png)
66+
Pictured in above diagram is how the stack is brought into a CF
67+
Foundation and used in a CF Foundation. The following problems occur
68+
when trying to remove/deprecate a stack towards users of a CF
69+
Foundation.
70+
71+
Due to [RFC-0039](https://github.com/cloudfoundry/community/blob/main/toc/rfc/rfc-0039-noble-based-cflinuxfs5.md), these struggles with the stack become a more pressing issue for CF Foundation operators and users since many still use CFLinuxFS3 and the migration to CFLinuxFS4 is not yet complete for many users. The current stack handling in CF does not allow for a smooth transition and leads to potential downtimes for applications when a stack is removed from the CF Foundation.
72+
73+
### Usage
74+
75+
The management entry of the stack in the stacks table of the CF API
76+
cannot be removed as long as apps exist which use it as the apps table
77+
has a foreign key relationship to the stacks table. The stack thus is
78+
still visible and still usable to all users of a CF Foundation even if
79+
being insecure, deprecated and SHOULD not be adopted anymore.\
80+
It is a cumbersome process for the users to migrate their workload to a
81+
new stack. Оne has to acknowledge that and give users of Cloud Foundry
82+
time to adapt their workload accordingly without a hard deadline when
83+
their apps will stop working.
84+
85+
### Delivery
86+
87+
By regulations one MAY be forced to stop shipping insecure parts of
88+
CloudFoundry for formal/regulatory reasons. That means removing it/not
89+
deploying it with CF-Deployment at all. In this case, the stack is not
90+
put onto the local filesystem of the Diego cell and every app instance
91+
(Long Running Process(LRP)/Task) start will fail because the runtime did not find the
92+
preloaded stack in its local filesystem. One thus is not able to stop
93+
shipping/delivering the outdated, insecure stack anymore without causing
94+
downtimes to all apps still using it.
95+
96+
### Adoption Timelines
97+
98+
Currently a stack is only shipped ever 4 years skipping one LTS version
99+
of ubuntu entirely. This creates a situation where the old stack e.g.
100+
CFLinuxFS3 is flagged as unsecure at roughly the same time the new stack
101+
CFLinuxFS4 is available. This forces users to adopt it in a very small
102+
timeframe if they still want to receive security updates for their
103+
existing workload. In the last migration from CFLinuxFS3 to CFLinuxFS4
104+
it turned out also the buildpacks need time to adopt -- then the
105+
customers to the new buildpacks for which the time there was far too
106+
short.
107+
108+
## Motivation
109+
110+
1. Enable a productized shipment of CF to come without a deprecated
111+
stack that still is used by apps on the Foundations
112+
113+
2. Without the stack shipped or remove/disable by operators, keep all
114+
apps running without downtime.
115+
116+
3. Give users and buildpacks more time to adopt, ideally in the range
117+
of multiple years.
118+
119+
## Proposal
120+
121+
### Bring your own Stack
122+
123+
Currenlty diego bbs only accepts as rootfs:
124+
125+
`precached:cflinuxfs4` as a stack on the local filesystem
126+
127+
`docker://{URL:PORT}/[Image Namespace]:[Reference]` A container
128+
image from a remote. Without URL defaulting to docker hub. Current
129+
method of passing a container image to CF see [the CF
130+
Docs](https://docs.cloudfoundry.org/devguide/deploy-apps/push-docker.html).
131+
132+
Since container images and stacks are technically identical, we MAY offer to support container images as stacks.
133+
This would allow CF Users to use their own stacks, similar to how they can use their own buildpacks today.
134+
135+
##### CF API Changes
136+
137+
First of all the CF API SHOULD add a new feature flag similar to the `diego_docker` feature flag that allows to enable the use of lifecycle docker container images. This flag SHOULD be called `diego_custom_stacks` and be disabled by default in the CF API.
138+
139+
In the CF API we extend the endpoints of apps and cf
140+
manifest to allow a stack not only to be a precached one with a fixed
141+
name e.g. `cflinuxfs4` but also be a valid container image reference e.g. `docker://docker.io/cloudfoundry/cflinuxfs4:1.268.0`. See
142+
<https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#stack>
143+
144+
The CF API CAN check if it's a system
145+
provided one or a remote one by checking if the stack is an exact match
146+
in the stacks table(it already does this to check validity of the
147+
manifest/request) and if it's not an exact match try to evaluate it as
148+
remote container image reference. If it does not match the container url schema produce a error message.
149+
150+
The apps lifecycle object would not only be enabled to have a hardcoded stack name like:
151+
152+
```json
153+
{
154+
"lifecycle": {
155+
"type": "buildpack",
156+
"data": {
157+
"buildpacks": ["java_buildpack"],
158+
"stack": "cflinuxfs4"
159+
}
160+
},
161+
}
162+
```
163+
164+
but rather also allow a stack to be a container image reference:
165+
166+
```json
167+
{
168+
"lifecycle": {
169+
"type": "buildpack",
170+
"data": {
171+
"buildpacks": ["java_buildpack"],
172+
"stack": "docker://docker.io/cloudfoundry/cflinuxfs4:1.268.0"
173+
},
174+
"credentials": {
175+
"example.org": {
176+
"username": "user",
177+
"password": "****"
178+
},
179+
}
180+
}
181+
}
182+
```
183+
184+
Similar to the [CNB Lifecycle](https://v3-apidocs.cloudfoundry.org/version/3.196.0/index.html#cloud-native-buildpacks-lifecycle-experimental) adding a credentials sections SHOULD be possible in case the stack resides on a private registry and authentication is required to pull the container image. For `Cloud Native Buildpacks` this is already supported as buildpacks are container images. However opposed to the CNB lifecycle, the credential section SHOULD only implement the `username` and `password` fields as the `token` field is not supported by the Diego BBS API. Diego BBS currently needs always a user [when supplying a password](https://github.com/cloudfoundry/bbs/blob/main/docs/031-defining-lrps.md#imagepassword-optional) to pull a container image in its `rootfs` field.
185+
186+
For pulling the stack the first credentials SHALL be passed to the Diego BBS API that matches the URL/Hostname of the stack image URI. The CF API SHOULD then fill the fields [ImageUsername](https://github.com/cloudfoundry/bbs/blob/main/docs/031-defining-lrps.md#imageusername-optional) and [ImagePassword](https://github.com/cloudfoundry/bbs/blob/main/docs/031-defining-lrps.md#imagepassword-optional) when creating `Tasks` or `LRPs` in the Diego BBS API.
187+
188+
Also for CNBs this currently CAN already be used since the credentials are present in the lifecycles data section.
189+
190+
```json
191+
{
192+
"type": "cnb",
193+
"data": {
194+
"buildpacks": [
195+
"docker://example.org/java-buildpack:latest"
196+
"docker://second-example.org/logging-buildpack:latest"
197+
],
198+
"stack": "docker://docker.io/cloudfoundry/cflinuxfs4:1.268.0",
199+
"credentials": {
200+
"example.org": {
201+
"username": "user",
202+
"password": "****"
203+
},
204+
"second-example.org": {
205+
"token": "****"
206+
},
207+
}
208+
}
209+
}
210+
```
211+
212+
In case a `token` is provided in a credential, the CF API SHOULD ignore credentials which supply a `token` as Diego BBS does not support this field and thus the CF API would not be able to pass it to Diego BBS. Only credentials with a `username` and `password` SHOULD be considdered for the stack. Credentials with the `token` field SHOULD be only used for buildpacks.
213+
214+
Furthermore, in case the stack is a remote container image reference also at least
215+
one system buildpack without a pinned stack MUST exist or a custom
216+
buildpack MUST be provided see
217+
<https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#buildpacks>.
218+
219+
Otherwise the request will be denied by the CF API with the message that there is no buildpack
220+
with that name that fits the stack. This is already the current
221+
behaviour since when you try to force a app to stage with a buildpack
222+
that is assigned a specific stack e.g. `CFLinuxFS4` to be staged with
223+
`CFLinuxFS3` as a stack instead this is prevented by the CF API already:
224+
225+
```
226+
For application \'test: Buildpack \"python_buildpack\" for stack
227+
\"cflinuxfs3\" MUST be an existing admin buildpack or a valid git URI\
228+
FAILED
229+
```
230+
231+
The same would apply if the stack would be a remote container reference
232+
and no own buildpack was provided and none of the system buildpacks can
233+
be used.
234+
235+
##### Diego BBS API
236+
237+
If all the URI evaluation succeeds and boundary conditions like a custom buildpack, feature-flag etc. pass,
238+
the CF API creates the LRP/Task in diego bbs in the [rootfs field](https://github.com/cloudfoundry/bbs/blob/main/docs/031-defining-lrps.md#rootfs-required) of the LRP setup definition
239+
240+
Optionally also passing the credentials in the [ImageUsername](https://github.com/cloudfoundry/bbs/blob/main/docs/031-defining-lrps.md#imageusername-optional) and [ImagePassword](https://github.com/cloudfoundry/bbs/blob/main/docs/031-defining-lrps.md#imagepassword-optional) fields.
241+
242+
```json
243+
# Either
244+
"rootfs": "precached:cflinuxfs4"
245+
# or
246+
"rootfs": "docker://docker.io/cloudfoundry/cflinuxfs4:1.268.0"
247+
# or
248+
"rootfs": "docker://myprivateregistry.example.com/cloudfoundry/cflinuxfs4:1.268.0",
249+
"imageusername": "user",
250+
"imagepassword": "123"
251+
```
252+
253+
As the logic what to run in the staging process [resides in the CF API](https://github.com/cloudfoundry/cloud_controller_ng/blob/384b017c2e7cf02a492ccffdb6985348abdbf8bb/lib/cloud_controller/diego/buildpack/staging_action_builder.rb) no code change SHOULD be required in diego bbs to support this.
254+
The same [diego actions](https://github.com/cloudfoundry/bbs/blob/main/docs/053-actions.md#available-actions) SHOULD be used as today when using `prechached` stacks as when using remote container images as stacks.
255+
256+
As they `rootfs` and `action` system is quite flexible in diego no code change in diego MAY be actually required to support this.
257+
258+
##### Providing a stack as remote container image
259+
260+
The CF Community also already uploads the stack and publishes it in
261+
container registries e.g. in docker hub
262+
<https://hub.docker.com/r/cloudfoundry/cflinuxfs4> and
263+
<https://hub.docker.com/r/cloudfoundry/cflinuxfs3>
264+
265+
A comprehensive strategy SHOULD be introduced in which the stack images are distributed on multiple registries for redundancy and vendor independence. Proposing following registries wich offer free public container image hosting:
266+
267+
- Docker Hub
268+
- RedHat`s Quay.io
269+
- GitHub Container Registry
270+
271+
##### CF Tasks
272+
273+
The same stack shall be used also for tasks in respective calls to the
274+
Diego API. When talking to diego BBS the same properties `rootfs`, `imageusername` and `imagepassword` exist in a BBS [Task](https://github.com/cloudfoundry/bbs/blob/main/docs/021-defining-tasks.md#rootfs-required)
275+
276+
##### CF Sidecars
277+
278+
###### User Provided Sidecars
279+
280+
User provided [sidecars](https://docs.cloudfoundry.org/devguide/sidecars.html) executed within the applications container are started in the original app container and thus use the proper stack the user wants beeing it a platform proivided stack or a remote container image.
281+
282+
###### Sidecars provided by CF
283+
284+
Cloud Foundry currently injects binaries like [diego-ssh](https://github.com/cloudfoundry/diego-ssh) automatically into an application container and starts them
285+
as sidecar. Since these sidecars currently work already in arbitrary
286+
containers of lifecycle docker, they are statically linked and thus
287+
self-contained and will also function with any stack version alike as
288+
they do not depend at all on OS functionality.
289+
290+
Other process that are started per app container like the envoy-proxy are started in their own container with a system defined stack by [the diego-release config](https://github.com/cloudfoundry/diego-release/blob/develop/jobs/rep/spec#L52-L56).
291+
292+
With the introduction of custom stacks, these sidecars part beeing it customer provided or CF provided sidecars SHOULD behave and function the same as they do today.
293+
294+
##### Compatibility Documentation
295+
296+
With CF having the scope of running arbitrary stacks and already arbitary container images that contain
297+
unknown software e.g. glibc versions that interact with the host\`s
298+
linux kernel, Cloud Foundry SHOULD start documenting, also for the already existing
299+
docker lifecycle, what kernel versions MUST be supported by the
300+
container and stack. And that the container MUST be able to run with that. On every
301+
stemcell major update a deprication notice for the stack SHOULD be
302+
propagated that informs of a major linux kernel update.
303+
304+
While the linux kernel has outstanding compatibility it SHOULD at least
305+
formally defined to not expect support of the CF community if a
306+
containers software/library is designed to interface against an ancient
307+
old kernel version. Newer kernels might remove certain functionality,
308+
have a braking Application Binary Interface change over a long-time span
309+
and thus break older software when running in a containerized
310+
environment. The [kernel
311+
docs](https://www.kernel.org/doc/Documentation/ABI/README) `ABI ...
312+
backward compatibility for them will be guaranteed for at least 2
313+
years`. Also libraries like glibc drop support for certain kernel
314+
versions e.g. with [glibc
315+
2.24](https://sourceware.org/legacy-ml/libc-alpha/2016-08/msg00212.html)
316+
they bumped the min kernel version from 2.6.32 to 3.2. It is not
317+
guaranteed thus that any arbitrary old stemcell or container image will be functional in CF.
318+
319+
Additionally software compiled to interface against a newer kernel version MAY experience missing functionality, imagine a GLIBC compiled against a newer kernel version that uses a syscall that is not available in the older kernel version. This would lead to a runtime error when the software is executed in a container with an older kernel version provided by a stemcell.
320+
321+
Thus a minimum and maximum(current one in the stemcell) kernel version SHOULD be defined, documented and communicated by CloudFoundry. As a CF User i then know which what kernel versions i can build my software against and be able to run the compiled result in a CF Foundation.
322+
323+
Since most applications are using buildpacks that compile everything in the staging process this is not a problem for most CF Users. However, when using lifecycle docker or custom stacks this should be at least properly documented as constraints as it recieves more relevance when executing precompiled binaries in CF directly.
324+
325+
##### Wrap Up
326+
327+
We would enable CF Users to deliberately use a stack which was removed
328+
as system stack and take over full responsibility and self-support if
329+
they cannot follow the support cadence of CF. Similar how a customers on
330+
e.g. AWS CAN decide to deliberately upload an old/own VM Image ISO on their own
331+
risk and with self-support. The service itself remains in a usable
332+
condition at any time.
333+
334+
We would allow CF Operators to programatically change the stack of all applications to a remote one, restage the applications and thus be able to remove the system stack from the CF Foundation as no usage of the stack exists anymore. This would allow to remove a stack without downtime of applications and without the need to keep the stack in the CF Foundation forever.
335+
336+
#### Positive
337+
338+
- When we would prevent staging with a deprecated/locked/removed stack ([RFC-0045](https://github.com/cloudfoundry/community/blob/main/toc/rfc/rfc-0045-enhance-stack-handling.md))
339+
we still CAN offer the user a way forward in his full responsibility
340+
to be unblocked and to own the whole applications stack end-to-end to
341+
take their own decisions.
342+
343+
- When a CF Foundation operator wants to remove a system stack, he can
344+
programatically change the stack of all applications to a remote one,
345+
restage the applications and thus be able to remove the system stack
346+
from the CF Foundation as no usage of the stack exists anymore. This
347+
would allow to remove a stack without downtime of applications and
348+
without the need to keep the stack in the CF Foundation forever.
349+
350+
- Likely not a high effort to implement on diego level as the container
351+
images currently are handled the same and stacks are also container
352+
images saved as tar file. Also, just minor adoptions to the CF API
353+
logic CAN be expected as its merely accepting container image references
354+
in the stack and passing it through towards Diego bbs.
355+
356+
#### Negative
357+
358+
- Since the issue with container images - the app cannot be started when
359+
registry is unavailable - still exists, it is also applied to custom
360+
stacks. There exist ideas how to cache container images inside CF for
361+
better reliability and this would also benefit custom stacks, however
362+
currently this is not implemented or put into an RFC yet but likely
363+
then needs to be adressed if custom stacks find acceptance to provide the same
364+
high availability qualities as with system stacks.

‎toc/rfc/rfc-draft-provide-custom-stacks-functionality/current_stack_usage.png‎

121 KB
Loading

0 commit comments

Comments
 (0)