devops : add llama in all docker images

[angt]

Overview

This proposal aims to simplify Docker images and prepare for a near future where only the llama binary will be provided, eliminating the need for light and server flavors

See: huggingface/hub-docs#2593

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

[CISC]

This proposal aims to simplify Docker images and prepare for a near future where only the llama binary will be provided, eliminating the need for light and server flavors

I don't see how that would eliminate the need for them? The point is to offer images only containing what you need.

[angt]

This proposal aims to simplify Docker images and prepare for a near future where only the llama binary will be provided, eliminating the need for light and server flavors

I don't see how that would eliminate the need for them? The point is to offer images only containing what you need.

The entrypoint would be only llama at the end, allowing users to run serve, bench, cli, etc directly.
This approach might be simpler (using a single Docker image) without having a bigger image than the server image (though I need to verify that claim haha)

[CISC]

The entrypoint would be only llama at the end, allowing users to run serve, bench, cli, etc directly.

Sure.

This approach might be simpler (using a single Docker image) without having a bigger image than the server image (though I need to verify that claim haha)

I don't think so. :)

[angt]

The entrypoint would be only llama at the end, allowing users to run serve, bench, cli, etc directly.

Sure.

This approach might be simpler (using a single Docker image) without having a bigger image than the server image (though I need to verify that claim haha)

I don't think so. :)

angt at angt-build in ~ on master
$ docker image inspect ghcr.io/ggml-org/llama.cpp:server --format='{{.Size}}'
845948308

angt at angt-build in ~ on master
$ docker image inspect ghcr.io/ggml-org/llama.cpp:light --format='{{.Size}}'
845964276

[angt]

We could actually make the light version lighter, but I don’t believe there is much value in it.
Having one Docker app llama reflecting the llama binary could be very convenient to me.

[pcuenca]

The entrypoint would be only llama at the end, allowing users to run serve, bench, cli, etc directly.

Sure.

This approach might be simpler (using a single Docker image) without having a bigger image than the server image (though I need to verify that claim haha)

I don't think so. :)

angt at angt-build in ~ on master
$ docker image inspect ghcr.io/ggml-org/llama.cpp:server --format='{{.Size}}'
845948308

angt at angt-build in ~ on master
$ docker image inspect ghcr.io/ggml-org/llama.cpp:light --format='{{.Size}}'
845964276

But the full would be larger, wouldn't it?

[angt]

But the full would be larger, wouldn't it?

Not tested, but should be same size to me :)

[CISC]

We could actually make the light version lighter, but I don’t believe there is much value in it. Having one Docker app llama reflecting the llama binary could be very convenient to me.

The problem is that the base image recently got ffmpeg added to it, it should probably only be in server/full. It adds massive bloat.

[pcuenca]

The current full is much larger (2.9 GB vs 850MB for the others):

docker image inspect ghcr.io/ggml-org/llama.cpp:full --format='{{.Size}}'
2886992965

[pcuenca]

Looks good to me, for the goal to standardize llama as the entry point.

[angt]

But the full would be larger, wouldn't it?

Not tested, but should be same size to me :)

I was thinking of the full llama, not the :full haha

[ngxson]

Ignore my last comment, seems like python takes the most space in full image:

root@a946e755ab92:/app# du -h --max-depth=1 /usr
80M	/usr/libexec
153M	/usr/share
28M	/usr/include
8.0M	/usr/sbin
791M	/usr/lib
98M	/usr/bin
1.2G	/usr/local
4.0K	/usr/games
88K	/usr/src
2.4G	/usr
root@a946e755ab92:/app# du -h --max-depth=1 /usr/local/
36K	/usr/local/share
4.0K	/usr/local/include
4.0K	/usr/local/sbin
1.2G	/usr/local/lib
4.0K	/usr/local/etc
120K	/usr/local/bin
4.0K	/usr/local/games
4.0K	/usr/local/src
1.2G	/usr/local/
root@a946e755ab92:/app# du -h --max-depth=1 /usr/local/lib
1.2G	/usr/local/lib/python3.12
1.2G	/usr/local/lib
root@a946e755ab92:/app# du -h --max-depth=1 /usr/local/lib/python3.12/
1.2G	/usr/local/lib/python3.12/dist-packages
1.2G	/usr/local/lib/python3.12/

[CISC]

Edit: not sure why we have fetch-depth: 0 in docker.yml

It's probably a remnant from before adding create_tag when get-tag-name was a step in the main job.

[kannon92]

Will there be a follow up to release these images to dockerhub?

When I was writing this up, my AI / I found that some github actions work are needed to publish these images to dockerHub.

[kannon92]

Ah nvm. I see your plan is to simplify the publishing process but including the app binary rather than a separate one.