Period Types as time interval sequences

[jbee]

Motivation

ATM periods and period types are pretending to work within any calendar system. The calendar used is a system setting (that can never change once used). This IMO is a confusion around the concept of time "buckets" (slicing up a continuous timeline into a sequence of intervals). A bucketing system such as "week of year" is very tightly connected to one calendar system. Other calendars might have a similar concept that is also similarly named but these would be 2 distinct sequences that have nothing in common other than the terms we use to refer to them. Specifically any ISO conform period is always a Gregorian calendar period. In contrast a weekly period in the traditional Chinese calendar would have 10 day weeks in a variable length Year. This is an entirely different sequence to the ISO weeks. It does not make sense for these 2 to share any code or for data of the two to be compared to each other. Attempting to do so has lead to the very complicated and fragile date related code in DHIS2 where the attempt to make it "multi-calendar ready" has moved it to a state that is inconsistent, incorrect and incomplete and very hard to reason about.

The complication is further stressed by the need for custom period types. These are essentially sequences that do not follow a calendar but a lookup table of decided dates and names for the intervals they hold. A classic example would be a school year with Spring term, Autumn term and alike.

Last but not least, the idea that all periods in a DHIS2 system must follow a single calendar is wrong on many levels. It is an unnecessary limitation, source of great confusion and risk to break the data. Imagine a country now using non-ISO weeks but entering a transition phase to join the ISO world. Such a system would clearly have multiple weekly (start Monday) types. The non-ISO one for old existing data and ISO weeks for new data. This cannot be solved in a system where one calendar rules them all.
In contrast, it is easy to see that data points entered for time intervals belonging to a specific calendar should never change their semantics. The semantics implied by the type and its implied calendar is what anchors the data in time.

Solution Outline

• remove the calendar setting and replace it with a selection setting where users select the period types they want in their system out of the offered range of sequences we have implemented. This changes period types from being a matrix of type and calendar to be a list where a calendar is implied in each type.
• make any period type a sequence within a specific calendar system, e.g. Gregorian week of the year (a.k.a. ISO week)
• make the date string (ATM wrongly called "ISO period") the external ID of a period and as such the canonical identifier (as opposed to the date interval and type)
• create a minimal interface for a sequence that has the essential functions
  • now (creates the period that contains "now" for the type)
  • date interval => ID string
  • ID string => date interval
  • next (period after a given one)
  • previous (period after a given one)
  • typical length (replaces getFrequencyOrder, e.g. week = 7)
  • infix (the letters following the year to identify the type)

Custom Period Types

A custom type is essentially a lookup table with the columns

• sequence number (1, 2, ...)
• string ID
• first day (year, month, day)
• last day (year, month, day)

The main challenge with custom period types is to make sure the ID string is unique and does not collide with the identification of other period types. At the same time it should be as readable and self explanatory as possible. It seems reasonable to limit custom periods to such sequences that split a year into intervals. Meaning they also can follow the general pattern YYYY$? where $ marks the infix that is unique for a period type. For example, for a school term type, the infix may be ST, a term can be written as 2026ST1, 2026ST2 and so forth. Like in this example, digits can follow the infix.

Prototype

enum PeriodType {
  WEEKS(new GregorianWeeklyPeriodSequence())
  //...
 ;
 final PeriodSequence sequence;
}

record FixPointDay(int daysSince2000) {}
record SpanOfDays( FixPointDay first, FixPointDay last) {}

interface PeriodSequence {
  String infix();
  
  SpanOfDays decode(String period);
  
  String encode(SpanOfDays period);

  SpanOfDays now();

  SpanOfDays next(SpanOfDays period);
  
  SpanOfDays previous(SpanOfDays period);
  
  int lengthInDays();
}

Timestamps

As the prototype indicates periods and period types or sequences can and should work without a link to an absolute timeline (like a timestamp would provide). They track days independent of when exactly such a day would have started on an absolute timeline somewhere. The reason is simply that the start of day on a absolute timeline is always linked to a place or timezone. This confuses and complicates the world or time intervals with the world of time-zones. Such complication is entirely unnecessary. Instead everything can be described in days since a fix-point. The idea is similar to a timestamp except that the fix-point is a local date without time zone.
This also has the advantage of being able to calculate the length of a period in days without understanding the calendar. It is simply the difference between the last and the first day fix-point offset. In addition it is trivial for most types to advance the next interval in the sequence.

Ideally DHIS2 would have used such a model for periods from the beginning but unfortunately we have to deal with the past of start and end dates being stored as timestamps in the database. While a big task, it is an option to also refactor the database. More realistically we maintain timestamps by making them absolute assuming the servers time-zone to compute the corresponding timestamps.

Time-zones

By its nature aggregate numbers for a span of days does not have to deal with time-zones. While it did matter for the counting to consider when days start and end once the number for a day or a span of days is counted and then reported to DHIS2 the system does not have to care about absolute time. Users will usually select days with a date picker specifying the day or day span independent of time-zones. There might be special cases where the API does not require the user to give a day but will implicitly assume that the current one is to pick. This is where time-zones matter when it is determined what the current day is in the calendar that is implied in a period sequence. Maybe we can do without such a feature. It is unclear if such a period creation is used beyond tests.

[jbee]

Web API

Some clarifications on dates in the web API (REST API) in the face of multi-calendar and custom periods...

Timestamp Inputs
As discussed some data is captured in terms of timestamps and thereby is agnostic to period types (time bucketing systems) in their raw data form. Some metadata related to the data would link the data to one or more period type(s) for which the raw data should be aggregated. This provides the means of computing the buckets in terms of timestamps so that a timestamp can easily be placed into them. The same linkage also provides the labeling system if the aggregated analytics data then is viewed by users.

Period Inputs
Other data is captured in the web API with period strings as date range references. Maybe this only applies to aggregate data? In such cases ATM clients send a canonical string identifier of the period which can be parsed and turned into a specific bucket in a specific period sequence. This works for periods which do have an ISO-ish notation but gets wonky once other period types should be referenced. We already have this happening with e.g. financial periods (and others) for which we invented our own notation. ATM this forces the client to also understand and encode these notations in order to speak to the web API. This approach does not really scale well as all clients would need updating when new period types and notations are invented. Also this duplicates the logic of such types on the client asking for issues because of discrepancies in the implementation. With increasing number of period types collisions in the notations become more likely and so to mitigate against it notations become more odd (less readable) to ensure uniqueness. Especially in the face of custom periods, which would be dynamically created by users, this system breaks down. To get out of this dilemma a "universal" period notation is required. In contrast to special notations invented so far which all have implicit assumptions the universal notation is just a notation to express a date range in an agreed upon format. In the simplest form imaginable it would simply state the start and end date as ISO dates, for example

2026-01-01...2026-01-08

In such a form any day precision bucket can be expressed exactly without relying on dedicated formats for different lengths, different calendars, custom periods etcetera. The part that such a range does not explicitly state is the period type. In most contexts this information is implicitly given by other metadata in the same request, such as the DE in a aggregate data value key combination. Where the period type is not given it can often be derived because only on the period types actually has a bucket with the given start and end date. But in theory a range like that may be ambiguous. Therefore it seems a good idea to also allow making the type explicit in the universal format, for example by giving each period type a unique two letter code which would be used as a prefix or suffix. For example, for Gregorian weeks the prefix could be GW making the universal period of the 1st week in 2026 (starting Thursdays)

GW2026-01-01...2026-01-07

Such a format is future proof and clients can extract the referenced time-span in other calendars without further context knowledge as they know the two date being ISO dates (in a specific timezone) which then can be converted to any other time labeling system via timestamps as the intermediate representation. Instead of having to implement custom parsing and time arithmetic for a growing and potentially even dynamic range of period types all that clients need to know is a single format and at times maybe the 2 letter code for a specific type.

The example so far used the most straight forward way of expressing a range. To keep the notation shorter one could naturally encode it differently. For example, by giving the start as ISO date, and the length in days. The above 1st week in 2026 could also be

GW2026-01-01+7

[larshelge]

Adding some brief feedback below.

First I agree that the current period type solution is not perfect and has its challenges.

That said I see challenges with the suggested design.

• Custom period types will be very complex and we have not decided it is worth it from a cost/benefit perspective.

• Custom period types have no support in analytics and will be tricky to implement from a complexity and performance perspective.

• There is no need to support multiple calendars in a single DHIS2 instance, at least I have never heard the need for it in the past 15 years.

• There are no countries "transitioning" between calendars in the foreseeable future, and hence we should not design for that.

• Introducing a new concept "sequence" which is not commonly understood has a high cost for the rest of the developer team, and it adds complexity which may not be justified.

• Most DHIS2 instances operate in a single time-zone, and aggregate time periods are selected irrespective of time zone, and so refactoring to support time zones for periods sounds like it is not worth it.

• Java 8 introduced Java https://docs.oracle.com/javase/8/docs/api/java/time/chrono/Chronology.html which states:

Most other calendar systems also operate on the shared concepts of year, month and day, linked to the cycles of the Earth around the Sun, and the Moon around the Earth. These shared concepts are defined by ChronoField and are available for use by any Chronology implementation.

• Will this become a breaking change in the time period format, e.g. "202601", "2026Q1", etc, into "GW2026-01-01...2026-01-07", etc? If so that will be an extremely invasive breaking change for implementations which exchange data values, which is unacceptable and not worth it.

[jbee]

Thanks Lars for the feedback.
Some misunderstandings and future direction aside, I wish our reality now would be as clear cut as you make it sound. To say it is challenging is a very positive way to phrase serious issues that are not well suited for public discussion.

[jbee]

POC can be found at https://github.com/jbee/dhis2-time (WIP)