Performance benchmarks

[Nyholm]

So I did some testing on a small app. I tested with #17 included (that PR does not have any performance impact).

A normal request: 10ms
First request: 1805ms
First request with BrefKernel: 510ms

BrefKernel spent 88ms preparing cache directory. It spent 77 ms out of those 88ms to copy the "pools" directory.

---

So thanks so #17, I disabled copying "pools" and enabled opcache.preload.

A normal request: 7.5ms
First request: 1144ms
First request with BrefKernel: 160ms

Yes. The first (cold) request is still 20 times slower. (Or 51x slower without preloading) But it is better than 180x slower as we began with.

---

Note: These sample data may not be reliable. I only did 2 samples of each. To get better data I should take 5-10 samples and show the average. However, I did deploy over CI so I know the process of warmup etc is the same.

[Nemo64]

I'm confused as to why preloading improves the first request time.

Preload does load files on boot so they are available within the request right? So shouldn't the first request be slower because it loads stuff it does not need? Or is the preload time within the init time and therefore not counted?

[Nyholm]

Actually no. Preload loads stuff before the first request. Ie, when PHP-Fpm is starting. That does technically happen before we get the "request data" from AWS.

So my "initialisation" should be a little bit slower when using preload. But that was nothing I noticed.

[jenschude]

Preload is btw also faster as the init process has more CPU power: https://medium.com/@hichaelmart/shave-99-93-off-your-lambda-bill-with-this-one-weird-trick-33c0acebb2ea

[markomitranic]

First, awesome job man @Nyholm . Thank you.

Just some future prospects thoughts here.

With every day, and especially with bref, we are moving more and more towards pre-packaged PHP releases. Something that is quite unusual in PHP world - we are packaging compiled assets, compiled/installed vendors, soon compiled code, and one day i hope precompiled container itself. We might as well look into how other ecosystems do it and why. I briefly asked Nikita about the theorethical possibility of precompiling the whole thing into a binary release, but he said that apart from some little CPU benefit he did not see any real value.

Back to topic at hand...

There might be a future possibility of compiling and baking the preloaded opcodes as well as symfony cache into the distribution layer.

• As far as opcodes go, at the moment i do not know of a way to forcibly specify the output location, which would allow us to distribute precompiled code along with the lambda code. Similarly to how we would treat a binary release. But if we look into it, there might be a way to compile on one lambda and bake the whole layer into a new lambda, or something similar. I think the exact same process is called compiler bootstrapping in C-world.

• As for baking and including a precompiled Symfony container, caches and whatever else comes with it - i did do it on non-bref instances a year ago, and it kind of worked. But it was just a quick local experiment, and i never dwelled deeper into it, to prove that it actually worked and if it had any performance benefit. I hope @Nyholm could tell us more about the symfony side of things, as i beleive that the same principle as above could be employed. Just run the first request on a lambda, copy the whole container and caches, and make a new lambda. Unless there are some drawbacks or blockers i do not know about.

[mnapoli]

@markomitranic what I usually do is compile the Symfony cache in a Bref docker image -> the paths are all good then.

[Nemo64]

Out of curiosity I benched my project too with different php versions and preload on/off.

I now used the configuration i mentioned in #21 (comment) even though that is irrelevant for my test request since it uses no validators or property-info. It is therefore similar to @Nyholm's test since his test had not copied the pools directory (what this library currently mainly does)

I always ran 2 requests to check for cold start and warm run.
I didn't repeat runs but the performance in lambda is really consistent so I don't worry too much about it.
The page I requested is a login page so there is a session to dynamodb in there and symfony form is loaded including lots of very small twig templates.

php 7.3

REPORT RequestId: 7cc167fb-5d41-4d59-b301-baf1358b63df	Duration: 1174.78 ms	Billed Duration: 1600 ms	Memory Size: 1024 MB	Max Memory Used: 120 MB	Init Duration: 337.25 ms	
REPORT RequestId: 320c114e-d88a-49b4-9f6c-237ca224eeab	Duration: 56.56 ms	Billed Duration: 100 ms	Memory Size: 1024 MB	Max Memory Used: 120 MB	

php 7.4

REPORT RequestId: b4c1a445-fc15-46d3-a73a-cdb93e5372cd	Duration: 1072.20 ms	Billed Duration: 1500 ms	Memory Size: 1024 MB	Max Memory Used: 125 MB	Init Duration: 332.41 ms	
REPORT RequestId: a8473aa9-0f1d-4800-872d-f9d22f55f9b7	Duration: 56.21 ms	Billed Duration: 100 ms	Memory Size: 1024 MB	Max Memory Used: 125 MB	

php 7.4 with container.dumper.inline_class_loader set to true because it defaults to false in php 7.4 even tough it is needed for the preload feature. It is true by default in 7.3.

REPORT RequestId: 7d9e1292-dadd-4e01-a99f-2e9ce2424c0f	Duration: 1094.20 ms	Billed Duration: 1500 ms	Memory Size: 1024 MB	Max Memory Used: 125 MB	Init Duration: 336.00 ms	
REPORT RequestId: 3e55912c-40ea-4958-a4f3-995ed54963ab	Duration: 49.77 ms	Billed Duration: 100 ms	Memory Size: 1024 MB	Max Memory Used: 125 MB	

php 7.4 with preload configured like opcache.preload=${LAMBDA_TASK_ROOT}/var/cache/prod/srcApp_KernelProdContainer.preload.php

REPORT RequestId: 363c7125-1255-4362-8239-99d5c2865e02	Duration: 609.20 ms	Billed Duration: 1200 ms	Memory Size: 1024 MB	Max Memory Used: 131 MB	Init Duration: 570.07 ms	
REPORT RequestId: cf862c09-ddd9-49f4-8057-a740566138de	Duration: 59.27 ms	Billed Duration: 100 ms	Memory Size: 1024 MB	Max Memory Used: 131 MB	

So php 7.4 is faster in any way. Although you should enable the inline_class_loader option which defaults to false starting with 7.4.
The preload feature does improve cold start time from 1430.20 ms to 1179.27 ms (duration + init time) but it seams to slightly decrease performance for warm requests.

Not this is for symfony 4.4 which is the first version to include the preload feature. There are improvements in 5.1.