Searching also returns full-day events of adjacent days

[nrother]

Problem:

I'm trying to find all events that occur today or later. In some cases, events from yesterday are returned.

I'm not completely sure if this is a bug or just the way CalDAV works, but at least it's not what I expected.

Test setup: There are there are three full-day events in the calendar, on 18.,19.,20.08.25. I created them with Thunderbird, the server is Baikal. I did not specify a timezone (nor could I for full-day events).

Code:

from datetime import datetime, timedelta, date, timezone
import caldav

url = "..."
username = "..."
password = "..."
client = caldav.DAVClient(url, username=username, password=password)
principal = client.principal()
calendars = principal.calendars()
calendar = next(cal for cal in calendars if cal.name == "Test-Cal")

# there are three full-day events in the calendar, on 18.,19.,20.08.25

dates = [
    datetime(year=2025, month=8, day=17, hour=0, minute=10),
    datetime(year=2025, month=8, day=17, hour=0, minute=10, tzinfo=timezone(timedelta(hours=2))),
    datetime(year=2025, month=8, day=17, hour=0, minute=10, tzinfo=timezone(timedelta(hours=0))),
    datetime(year=2025, month=8, day=17, hour=0, minute=0),
    datetime(year=2025, month=8, day=17, hour=0, minute=0, tzinfo=timezone(timedelta(hours=2))),
    datetime(year=2025, month=8, day=17, hour=0, minute=0, tzinfo=timezone(timedelta(hours=0))),
]

for test_time in dates:
    print("Testing with", test_time)
    print("    | 18 | 19 | 20 |")
    for i in range(4):
        print(f" +{i} |", end="")
        t = test_time + timedelta(days=i)
        results = calendar.search(start=t, end=t + timedelta(days=1), event=True, expand=True, sort_keys="dtstart")
        summaries = [e.instance.vevent.summary.value for e in results]
        for j in range(1, 4):
            if f"Test{j}" in summaries:
                print(f" X  |", end="")
            else:
                print(f"    |", end="")
        print()

Output (with comments from me). Expected would be a diagonal in the last three lines.

Testing with 2025-08-17 00:10:00 //my actual code, running in UTC+2 just after midnight
    | 18 | 19 | 20 |
 +0 |    |    |    |
 +1 | X  |    |    |
 +2 | X  | X  |    | //the bug I noticed
 +3 |    | X  | X  |
Testing with 2025-08-17 00:10:00+02:00 //add the timezone explicitly
    | 18 | 19 | 20 |
 +0 |    |    |    |
 +1 | X  |    |    |
 +2 | X  | X  |    | //no change
 +3 |    | X  | X  |
Testing with 2025-08-17 00:10:00+00:00 //now in UTC
    | 18 | 19 | 20 |
 +0 | X  |    |    | //this is correct b/c 00:10+1d contains the next day!
 +1 | X  | X  |    |
 +2 |    | X  | X  |
 +3 |    |    | X  |
Testing with 2025-08-17 00:00:00 // test on exactly midnight
    | 18 | 19 | 20 |
 +0 |    |    |    |
 +1 | X  |    |    |
 +2 | X  | X  |    | //same problem
 +3 |    | X  | X  |
Testing with 2025-08-17 00:00:00+02:00
    | 18 | 19 | 20 |
 +0 |    |    |    |
 +1 | X  |    |    |
 +2 | X  | X  |    | //dito
 +3 |    | X  | X  |
Testing with 2025-08-17 00:00:00+00:00 //midnight, UTC
    | 18 | 19 | 20 |
 +0 |    |    |    |
 +1 | X  |    |    | //correct!
 +2 |    | X  |    |
 +3 |    |    | X  |

This is the pattern I could make up from this:

• without timezone infos, the datetime gets interpreted in the local timezone
• before matching against full-day events in the calendar, dates are shifted to UTC
• this causes the dates to move to the previous day

This is just my guesswork, someone with more insight to CalDAV can probably make more sense out of this.

[tobixen]

It could be a server issue, I should do some research on that in my upcoming caldav server tester project.

Time zones are difficult. Perhaps the best one can do is to just accept that one may get too much and do a client-side filtering. Perhaps the right fix is to fix the search method to accept dates as search parameters, deliberately convert them into timestamps including too much, and then have client-side filtering to filter out any full-day events that is outside the interval.

(In practice a "whole day event" rarely starts exactly at midnight and ends exactly at midnight - perhaps yesterdays party lasted past midnight, and that's the reason why it's included?) :-)

[nrother]

Time zones are difficult.

I guess that's kind of the summary... Client-side filtering sound like the best option for now, I'll implement this in my application.

For completeness, this is the result with the -2 time zone:

Testing with 2025-08-17 00:00:00-02:00
    | 18 | 19 | 20 |
 +0 | X  |    |    |
 +1 | X  | X  |    | //now includes events from the next day
 +2 |    | X  | X  |
 +3 |    |    | X  |

[tobixen]

I'm currently working on client-side filtering, but I've so far procrastinated to deal with date-searches and timezone handling.

(I started using the caldav library because I didn't want to deal with the CalDAV protocol - soon enough I got the maintainer hat. I thought collations was difficult, so I asked the AI to fix case-insensitive vs case/sensitive searches with collations-support and have spent days playing ball with the AI so far - who could have thought that a lower case ascii dotted i does not map to an upper case non-dotted I in Turkish? Timezones scares the shit out of me, I'm concerned that as soon as I fall properly into that rabbit hole it will take me months to get up again)

[tobixen]

So in v2.2.2 that was just released, it's possible to to calendar.search(..., post_filter=True) and client-side filtering will be done, eliminating stuff if the server sent too much data.

This works fine if doing a date-range search with exact timestamps and the events on the calendar has exact timestamps. I haven't considered yet how full-day events should be handled (I think the RFCs also doesn't say anything about that). It's trivial if everyhing is in UTC-time, it's the time zones complicating things:

• According to RFC5545, timestamps can also be "floating" - like, 09:00 means "this should take place at 09:00 in whatever time zone the participant should be in when the clock is 09:00".
• The RFC5545 explicitly states that a date cannot have a time zone - so full-day events are by definition "floating". That's fine for most Europeans, but there is more than 24 hours difference between the most extreme time zones - people living near the international date line is for sure aware of this.
• A time range search according to the CalDAV standard (RFC4791) MUST be with timestamps in UTC-time.

So it's impossible for the CalDAV server to know what full-day and floating events the end user actually wants returned when doing a time-range search, and it's always better to return a bit too much than a bit too little.

With client filtering we have some more options, calendar.search() may take both "naive" timestamps, dates and timestamps with time zones. I didn't think much of it and I haven't verified, but I believe the current logic in the library is like this:

• Assume all naive timestamps given to search are in the local time zone as configured on the local computer
• Assume all dates given in from/to to the search is timestamps happening at midnight in the time zone as configured on the local computer.
• Assume all "floating evnts" as well as all full-day-events on the calendar happens in the time zone as configured on the local computer.

This is probably "good enough" in most cases, but I think there are some things to consider (first in test code, then in the logic itself):

• I guess the end date (in searches, when dates are given) should be considered to be inclusive. Searches from 8th-9th of April should return all data for both those days, and not only the 8th.
• When a search with a timezone is given, the floating objects should be assumed to be in the same time zone as given in the search. The local time zone on the computer should NOT be applied in those cases. A test case either has to mock up the local time on the computer, or use a time zone 12 hours away from UTC, for most of the population including the github servers this may be a good enough test.
• A "naive" timestamp should of course match very well with the floating objects as well as full-day events.

To resolve this issue, the steps would be something like this:

• Add the three test cases above to the test suite.
• In the caldav-server-tester package, check what data the caldav server will be returning when it's having full-day events and "floating time"-events and time-range queries are done. I can think of three options, it's defensively sending all full-day events and floating events that are within 24 hours of the search interval, it's assuming the calendar user is in the same time zone as the calendar server, and it's just assuming that the floating items are defined in UTC-time.
• Hammer on things until the tests passes on all servers