[2.1] Upgrader - Performance

[sbulen]

See #9259 for other issues with large DB upgrades

The upgrader has several extremely expensive database operations, all to the messages table:

• Adding likes column
• Adding modified_reason column
• Renaming poster_ip to poster_ip_old
• Adding column poster_ip
• Dropping column poster_ip_old

For each of the above operations, under the covers, ALTER TABLE makes a copy of the table then rebuilds it from scratch.

• Everything gets rebuilt, even fulltext indexes when present
• MySQL implicit table rebuilds can be VERY costly; I've seen them take as long as 10 hours each on a table with 3.3M records

It doesn't makes sense for it to take many hours to do a simple operation, even on 3.3M messages... ALTER TABLE clearly has issues with these implicit table rebuilds; it is much faster to rebuild the table explicitly yourself.

MariaDB handles this much better than MySQL, I believe due to a better implementation of ALGORITHM=INPLACE & ALGORITHM=INSTANT. MySQL has more constraints on these timesavers. HOWEVER... If you restructure the activity a bit, to account for such constraints (e.g., only dropping columns at the end of the column list) MySQL is almost as quick as MariaDB.

This PR:

• Reduces message table rebuilds by making an empty copy of messages, then doing most DB changes to the empty copy.
• The table is copied & repopulated explicitly, which is far faster than ALTER TABLE's implicit copy.
• Removes the fulltext index prior to any DB activity. This had been part of the utf8 conversion; this PR moves it prior to the DB changes. This cleans up many warnings & errors in the MySQL/MariaDB logs when FTS is present. More importantly, the upgrader isn't rebuilding the fulltext index over & over & over...
• Where necessary (because the operation requires populated data), moved column to be dropped to the end of the column list.

The IPv6 & UTF8 conversions remain time-consuming. No way around that.

Still, the time savings are significant. My "before" executions on MySQL were in the ~60 hour range, and varied on a number of factors, e.g., InnoDB vs MyISAM & environment/debug settings. MariaDB "before" executions varied from 5-18 hours. Tests were 2.0 => 2.1 upgrades, DB has 3.3M rows, no attachments.

With this change, MySQL now takes ~1 hour, & MariaDB takes ~45 mins.

If necessary, we could do the same to members. I've never seen members updates go near as long though.

I'm still doing tests on this.

If approved, I'll work on the 3.0 version.

[sbulen]

Without this PR...

Here is an example of an ALTER TABLE, adding the "likes" column. The PROCESSLIST shows that the current state is it is making a tmp copy of the table.

MySQL 8.4.4, an InnoDB table, 3.3M messages, this single ALTER TABLE command is still ongoing at ~7 hours.

This is still early in the upgrade process - there are at least another 4 similar operations to go. I.e., this particular upgrade will take AT LEAST another 28 hours, simply on things like adding & removing columns. Plus it still has the IP conversion & the UTF8 conversion to go.

I believe part of the problem here is the default value; when completed, the default value needs to be on each record, so it cannot take any shortcuts, i.e., there is no opportunity for an ALGORITHM=INSTANT to help out (for MySQL anyway). It MUST update every row in the table...

[sbulen]

OTOH, with this PR...

This upgrade is only 16 minutes in and the DB updates are almost complete. In fact, it's already 40% thru the explicit rebuild of messages. It still has the IPv6, JSON & UTF8 conversion to go, though. Still, it will be done in ~1hr total.

Same setup: MySQL 8.4.4, InnoDB, 3.3M messages.

[sbulen]

Some before & after comparisons... I've always understood that MyISAM is faster than InnoDB. However, in my initial upgrader tests, InnoDB was faster, due to the table copy efficiencies. So it depends on the activity.

MariaDB 11.5.2. MySQL 8.4.4.
Test DB: 3.3M rows, fulltext index on message body. No attachments.
The bottleneck is disk I/O, which is pegged to 100% during upgrades. Old school HDD.

Before - 2.0.19 => 2.1.7, with #9271 only:

• MyISAM - MySQL: 60 hours (estimated, I killed it...)
• MyISAM - MariaDB: 18 hours
• InnoDB - MySQL: 60 hours
• InnoDB - MariaDB: 5 hours

Note that without the json fix #9271, Maria DB cannot complete with this dataset.

After - 2.0.19 => 2.1.7, with #9270, #9271, #9272:

• MyISAM - MySQL: 20 mins
• MyISAM - MariaDB: 15 mins
• InnoDB - MySQL: 65 mins
• InnoDB - MariaDB: 45 mins

It's pretty shocking how quick MyISAM can be if you sidestep the table restructures w/fulltext.

[jdarwood007]

I like the changes, but I am worried about these in the stable release. @Sesquipedalian, what are your thoughts on this?

[live627]

This doesn't affect the patch though.

…

On Sat, Jun 20, 2026 at 5:01 AM Jeremy D ***@***.***> wrote: *jdarwood007* left a comment (SimpleMachines/SMF#9272) <#9272 (comment)> I like the changes, but I am worried about these in the stable release. @Sesquipedalian <https://github.com/Sesquipedalian>, what are your thoughts on this? — Reply to this email directly, view it on GitHub <#9272?email_source=notifications&email_token=AADJNNYKIRV5ZVR2CJSYQLL5AZ4IZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZVG43DONBSGU4KM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4757674258>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADJNNZYWVYFTTQOOGNN2GD5AZ4IZAVCNFSNUABDKJSXA33TNF2G64TZHMZTENJXGQ3TKO2JONZXKZJ3GQ3DQNZYGMZDKNBSUF3AE> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AADJNN4DHZM2OGHAMOQ2ZLT5AZ4IZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZVG43DONBSGU4KM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSVGM33PORSXEX3JN5ZQ> and Android <https://github.com/notifications/mobile/android/AADJNN6I6VN465E72N76OS35AZ4IZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZVG43DONBSGU4KM4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jdarwood007]

i'm worried we are changing the upgrader in a way that we can't be certain we didn't introduce a breaking upgrade bug that can be caught in the limited testing that occurs during a patch. Not the actual patch file itself.