Task modification very slow when project feedback is involved

[ChargingBulle]

Taskwarrior 3.x.x is significantly slower than 2.x.x after migration: basic commands take >1 second per task creating major annoyances for me who uses taskwarrior every day. Removing rows from tasks did reduce the command time to 1/400.
I also created an index on operations as it was the largest table but that results in no performance difference. Before the prune I had 7573 rows in tasks and I think that is not an unusually high amount of tasks.
Every operation takes approx 1 second per task and is significantly slower than in the 2.x.x version:

This is that it looks like to run modify on ten tasks at the same time:

Details

task 3.4.2 built for Linux

cbulle@debian-trixie:~$ time task 3 modify wait:mon
Modifying task 3 'hidden'.
Modified 1 task.
TASKDATA override: /tmp/.task.backup
Project 'finance' is 95% complete (6 of 125 tasks remaining).

real	0m1.086s
user	0m1.069s
sys	0m0.017s
cbulle@debian-trixie:~$ time yes | task 10-20 modify wait:mon | ts
2026-06-27 06:34:35 This command will alter 11 tasks.
2026-06-27 06:34:35   - Wait will be changed from '9999-12-30' to '2026-06-29'.
2026-06-27 06:34:36 Modify task 10 'hidden'? (yes/no/all/quit) Modifying task 10 'hidden'.
2026-06-27 06:34:36 
2026-06-27 06:34:36   - Wait will be changed from '2027-08-30' to '2026-06-29'.
2026-06-27 06:34:37 Modify task 11 'hidden'? (yes/no/all/quit) Modifying task 11 'hidden'.
2026-06-27 06:34:37 
2026-06-27 06:34:37   - Wait will be changed from '9999-12-30' to '2026-06-29'.
2026-06-27 06:34:38 Modify task 12 'hidden'? (yes/no/all/quit) Modifying task 12 'hidden'.
2026-06-27 06:34:38 
2026-06-27 06:34:38   - Wait will be changed from '9999-12-30' to '2026-06-29'.
2026-06-27 06:34:39 Modify task 13 'hidden'? (yes/no/all/quit) Modifying task 13 'hidden'.
2026-06-27 06:34:39 
2026-06-27 06:34:39   - Wait will be changed from '9999-12-30' to '2026-06-29'.
2026-06-27 06:34:40 Modify task 14 'hidden'? (yes/no/all/quit) Modifying task 14 'hidden'.
2026-06-27 06:34:40 
2026-06-27 06:34:40   - Wait will be changed from '2026-05-03' to '2026-06-29'.
2026-06-27 06:34:41 Modify task 15 'hidden'? (yes/no/all/quit) Modifying task 15 'hidden'.
2026-06-27 06:34:41 
2026-06-27 06:34:41   - Wait will be changed from '2026-10-01' to '2026-06-29'.
2026-06-27 06:34:43 Modify task 16 'hidden'? (yes/no/all/quit) Modifying task 16 'hidden'.
2026-06-27 06:34:43 This is a recurring task.  Do you want to modify all pending recurrences of this same task? (yes/no) 
2026-06-27 06:34:43   - Wait will be changed from '2026-06-28' to '2026-06-29'.
2026-06-27 06:34:44 Modify task 17 'hidden'? (yes/no/all/quit) Modifying task 17 'hidden'.
2026-06-27 06:34:44 
2026-06-27 06:34:44   - Wait will be changed from '2026-12-01' to '2026-06-29'.
2026-06-27 06:34:45 Modify task 18 'hidden'? (yes/no/all/quit) Modifying task 18 'hidden'.
2026-06-27 06:34:45 
2026-06-27 06:34:45   - Wait will be changed from '2026-03-05' to '2026-06-29'.
2026-06-27 06:34:45 Modify task 19 'hidden'? (yes/no/all/quit) Modifying task 19 'hidden'.
2026-06-27 06:34:45 
2026-06-27 06:34:45   - Wait will be changed from '2026-06-28' to '2026-06-29'.
TASKDATA override: /tmp/.task.backup
Warning: You have specified that the 'wait' date is after the 'due' date.
Project '40c3' is 0% complete (1 task remaining).
Project 'finance' is 95% complete (6 of 125 tasks remaining).
Project 'freunde' is 98% complete (2 of 105 tasks remaining).
Project 'haushalt' is 99% complete (2 of 650 tasks remaining).
Project 'health' is 99% complete (1 of 403 tasks remaining).
Project 'job' is 98% complete (1 of 59 tasks remaining).
Project 'partner' is 97% complete (5 of 219 tasks remaining).
Project 'sec' is 96% complete (1 of 32 tasks remaining).
Project 'self' is 99% complete (1 of 162 tasks remaining).
2026-06-27 06:34:46 Modify task 20 'hidden'? (yes/no/all/quit) Modifying task 20 'hidden'.
2026-06-27 06:34:46 Modified 11 tasks.

real	0m10.578s
user	0m10.555s
sys	0m0.053s

cbulle@debian-trixie:~$ time task stats
TASKDATA override: /tmp/.task.backup

Category                   Data         
Pending                    2
Waiting                    32
Recurring                  2
Completed                  5555
Deleted                    1982
Total                      7573
Annotations                7953
Unique tags                11
Projects                   232
Blocked tasks              0
Blocking tasks             0
Undo transactions          850
Sync backlog transactions  84285
Tasks tagged               31.6%
Oldest task                2019-02-24
Newest task                2026-06-26
Task used for              7.3y
Task added every           8h
Task completed every       11h
Task deleted every         1d
Average time pending       12d
Average desc length        47 characters


real	0m1.230s
user	0m1.194s
sys	0m0.036s

The v2 on the other hand was very snappy and quick given the (almost) same amount of items.
Taskwarrior v3 slower than v2 after upgrade / performance regression / migration to new version 3 causes slow operations / Performance issue post-migration slow per task / bulk commands extremely slow (but was slow in 2.x.x!) / ~1s delay for task commands / 3.0.0 new data format slowdown every task command is slower than before.

[ChargingBulle]

tried and made no difference:

create unique index operations_id_uindex
    on operations (id);

and also

DELETE FROM operations WHERE true

both to no results in speed

[ChargingBulle]

What actually helped:

DELETE FROM tasks where data LIKE '%"status":"completed"%';
DELETE FROM tasks where data LIKE '%"status":"deleted"%';

Now it's snappy:

Details

time yes | task 10-20 mod wait:aug | ts
TASKDATA override: /tmp/.task.backup
Warning: You have specified that the 'wait' date is after the 'due' date.
Project '40c3' is 0% complete (1 task remaining).
Project 'finance' is 0% co</details>mplete (6 of 6 tasks remaining).
Project 'freunde' is 0% complete (2 of 2 tasks remaining).
Project 'haushalt' is 0% complete (2 of 2 tasks remaining).
Project 'health' is 0% complete (1 task remaining).
Project 'job' is 0% complete (1 task remaining).
Project 'partner' is 0% complete (5 of 5 tasks remaining).
Project 'sec' is 0% complete (1 task remaining).
Project 'self' is 0% complete (1 task remaining).
2026-06-27 07:18:19 This command will alter 11 tasks.
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 10 'hidden'? (yes/no/all/quit) Modifying task 10 'hidden'.
2026-06-27 07:18:19 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 11 'hidden'? (yes/no/all/quit) Modifying task 11 'hidden'.
2026-06-27 07:18:19 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 12 'hidden'? (yes/no/all/quit) Modifying task 12 'hidden'.
2026-06-27 07:18:19 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 13 'hidden'? (yes/no/all/quit) Modifying task 13 'hidden'.
2026-06-27 07:18:19 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 14 'hidden'? (yes/no/all/quit) Modifying task 14 'hidden'.
2026-06-27 07:18:19 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 15 'hidden'? (yes/no/all/quit) Modifying task 15 'hidden'.
2026-06-27 07:18:19 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 16 'hidden'? (yes/no/all/quit) Modifying task 16 'hidden'.
2026-06-27 07:18:19 This is a recurring task.  Do you want to modify all pending recurrences of this same task? (yes/no) 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 17 'hidden'? (yes/no/all/quit) Modifying task 17 'hidden'.
2026-06-27 07:18:19 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 18 'hidden'? (yes/no/all/quit) Modifying task 18 'hidden'.
2026-06-27 07:18:19 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 19 'hidden'? (yes/no/all/quit) Modifying task 19 'hidden'.
2026-06-27 07:18:19 
2026-06-27 07:18:19   - Wait will be changed from '2026-11-01' to '2026-08-01'.
2026-06-27 07:18:19 Modify task 20 'hidden'? (yes/no/all/quit) Modifying task 20 'hidden'.
2026-06-27 07:18:19 Modified 11 tasks.

real	0m0.026s
user	0m0.037s
sys	0m0.013s

But I do consider this a bug because 7000 items is not a lot of rows and sqlite is perfectly capable of wortking with that. TW is slow despite sqlite being able to run well with 7000 items

[ChargingBulle]

I wrote "this is a bug" because "just shrink your database bro" is resignation of what really is a small number of tasks.

[ChargingBulle]

I have verified it's not just because I put it into /tmp (tmpfs)

[ashprice]

Hello!

I empathise with you - depending on the size of my DB and what I was doing at any given time, this had resulted in me straddling my usage between V2 and V3.

The rust/sqlite side of things can maybe be improved. Have you experienced slow operations toying with sqlite directly to modify a test DB? (This is not very advisable on your main DB, though I did have some scripts that would read from sqlite and then use task import to make modifications, such a script would easily become borked if the database schema changes during a TW release.)

In my experience, although it maybe could be improved (I haven't looked through the TC code much) that probably isn't the main bottleneck to real-world usage - any read/write from the database is still pretty quick. There's a kind of hard ~200ms floor for reading and rendering a large DB in TW, though. Similarly, the rust side of things handles modifications quite quickly. I intend to take a look at this after I've finished working on thw TW side, but I don't know how much room there is for improvement rn.

For TW, on the C++ side, the main issue for modifications is that it would re-parse the DB for every task from the rust side on the C++ side. I am currently working on a series of PRs that will address this by implementing in-memory caching on the one hand on the C++ side, and improving the internal memory management architecture wrt. making copies vs. using references. There are other issues which make other specific things slow - reports, especially with rc.urgency.inherit=1, and modifications involving dependencies, are notable offenders. Those are also being worked on - I have a local build where reports are down to 240ms for a 6329 task database with many dependencies. Making 3806 wait modifications (following your example) takes about 113 seconds, ie. ~23ms per task modified (with about 2.18 modifications per task.) I did have it faster than that before, but I felt it sensible to implement some guardrails that guard against potential future functions reading the wrong thing in the future and causing a difference of state between the replica and the caches. I think that's up for debate given no function currently reads the flags I protected via those in the relevant way, as far as I can tell.

If you switch to the develop head, you should see some improved performance already. Of course, develop should be considered a bit less stable than the releases. I haven't seen any bugs with my changes, and I am pretty sure there aren't any. But equally, if there are any, it would be appreciated if they can be caught before any new releases are made.

[ashprice]

PS. if you're on arch, note that the task-git AUR package will fail tests because those tests only work with a debug build currently due to some recent changes on develop. You can just modify the PKGBUILD to strip out the lines that build the test runner and run tests. All tests would pass with a debug build. (I don't maintain it, so typical use AUR at-your-own-risk disclaimer.)

[ChargingBulle]

How come your tw is so much faster than mine?
I'm running v 3.4.2 on debian testing.

toying with sqlite directly

No today is the first day I ever touched the sqlite database.
I used the official migration. I can see the ND-JSON is still in the database and it's not easy to query directly e. g. all tasks related to a project. Did I perhaps use an outdated migration and a schema more suitable for the data is available?

sqlite is competent software. my task database is literially "zero million of rows"
I dislike V3 being a re-write of TaskWarrior and the performance being suddenly abysmal where it has been fast before does not help with the adoption

[ChargingBulle]

This just isn't a database layout very conductive to speed:

And I never had speed problems with taskwarrior 2.x.x.
In fact I quite liked the flat file database layout

Perhaps sqlite has native JSON support like some databases do
but to me this looks like noone ever bothered to turn this into a normalized database schema
or perhaps I am running a botched migration?

I once again want to emphasize that 7000 items is "zero million of rows" and are no problem to sqlite. Gotta be because of TaskWarrior

[ChargingBulle]

creating an index would already help a lot a lot a lot:

before:

after:

in my defense I'm no sqlite-guy. But I have worked with enough database systems to know putting an index on everything you ever want to query is a requirement for any serious database work. In fact since the indexes are missing I would consider this buggy software.

I used this tutorial: https://dadroit.com/blog/json-querying/#section-6-how-to-use-indexing-for-query-optimization-over-json-data-in-sqlite

[ashprice]

It is faster because I have already implemented the changes I mentioned + others. These changes aren't in develop yet, because it was a hacky process born out of a back and forth with callgrind which means it was unsuitable for a lump PR all at once - I didn't really keep any commit history, no clear log of changes, and some things are messy wrt. function and variable naming, lack of comments.

So I am making smaller changes relative to the develop head and gradually introducing PRs into develop which will bring it functionally/performance speaking in line with my own build. I am a novice coder as well, so I am leaning a lot on the tests and other contributors for review.

They'll be forthcoming as I find time to work on it. The critical changes aren't quite there yet, if you see PR #4127, I found about 1.3x speed improvement for reports given that PR. Modifications should also be faster than they were before that PR, though I didn't benchmark by how much.

That recent PR also won't be in any release built against the most recent release tag. You'll only get it if you use a release stream that builds from source against the develop head.

Overall though, I haven't found any bugs in my own build, and performance is generally speaking better than V2 as well as V3 for the majority of things.

If you follow the develop head you can grab them as they come out, otherwise, hopefully they will make it into TW before the next major release.