8272194: java.sql.Date::toLocalDate() broken for dates before 1 A.D.#31808
8272194: java.sql.Date::toLocalDate() broken for dates before 1 A.D.#31808Sholto wants to merge 17 commits into
Conversation
|
👋 Welcome back Sholto! A progress list of the required criteria for merging this PR into |
|
@Sholto This change is no longer ready for integration - check the PR body for details. |
Webrevs
|
|
Hi, I am not sure that adding a protected method to |
…e/LocalDateTime from calendar It was decided that adding a new protected method to java.util.Date isn’t the safest thing to do as it is a widely extended class. The new method could unintentionally break the java.time conversion for these classes. The conversion still cannot by using Instant due to the difference between the milli representations as stated in 8061577. Therefore, constructing a new GregorianCalendar is the next best way to get the correct year/month/day values.
@naotoj Understood. I have gone with the |
I now think this approach would be significantly slower, although it would still handle the situation correctly, as it creates GregorianCalendar each time the method is called (time zone is not an issue here). Accessing the internal calendar via shared secrets might be possible, but I am not sure that is the right approach either. |
|
Regardless of the workaround, we could use the result of |
|
I did some benchmarking and the We could potentially expand the number of dates covered by this faster codepath by using a negative unix time representing a much earlier year. |
naotoj
left a comment
There was a problem hiding this comment.
We could potentially expand the number of dates covered by this faster codepath by using a negative unix time representing a much earlier year.
I think we can push the check close to the BC/AD switch, as we should account for local-time offset.
| // Ajdust the BC date into a negative astronomical date. | ||
| // As there is no year 0 in the proleptic Gregorian calendar | ||
| // we also have to adjust the BC year by 1. | ||
| // 1 BC becomes year 0, 2 BC becomes year -1 and so on. |
There was a problem hiding this comment.
I think you meant GregorianCalendar rather than "proleptic Gregorian calendar" here? GregorianCalendar has no year zero, whereas the ISO proleptic-year numbering used by LocalDate does. Also, there is a typo: "Ajdust" should be "Adjust".
There was a problem hiding this comment.
Thank you for spotting those. I have fixed the comments as recommended.
| year = 1 - calendar.get(Calendar.YEAR); | ||
| } else { | ||
| year = calendar.get(Calendar.YEAR); | ||
| } | ||
| return LocalDate.of(year, calendar.get(Calendar.MONTH) + 1, calendar.get(Calendar.DAY_OF_MONTH)); |
There was a problem hiding this comment.
The values of these fields should continue to be obtained by calling getYear(), getMonth(), and getDate(), rather than from the calendar instance. Otherwise, if a subclass overrides any of these methods, the override would be ignored.
There was a problem hiding this comment.
Updated as suggested. I have done this on Timestamp as well.
| // that we would ideally avoid if possible. | ||
| // Given that we can comfortably state that any dates after the unix epoch | ||
| // are AD, we can use a much faster local date time derivation for these dates. | ||
| if (getTime() >= 0) { |
There was a problem hiding this comment.
We could potentially expand the number of dates covered by this faster codepath by using a negative unix time representing a much earlier year.
That's a good idea. I agree that representing some negative time with a magic number is not as straightforward as 0, but considering we can retain almost the same performance for an extra ~1970 years, I think it is a worthwhile tradeoff. Granted we provide a big enough buffer to account for an arbitrary time zone offset, and the value is documented properly.
That would pretty much make only the BC years suffer the slowdown, which seems extremely justifiable since those are exactly the cases where the behavior was incorrect.
There was a problem hiding this comment.
I have pushed this date back to 0002-01-01. While overkill, I think that doing an entire year offset is cleaner than doing something like 0001-01-03 which would yield the same result but wouldn't give us as comfortable a distance from possible timezone problems.
I have also just gone with directly using the millis in a constant rather than deriving them from a calendar and converting it into millis in a static initialiser.
However, I am happy to change to that approach if you think it is clearer.
There was a problem hiding this comment.
I think the wide cushion is right. SimpleTimeZone can be created with offset up to Integer.MAX_VALUE offset in miliseconds, which gets up to ~25 days, so even a few day buffer would not be enough.
There was a problem hiding this comment.
Since there is good traction on this PR, I think you can go ahead and update the copyright years for any files that need it. Also, since these tests don't have individual Jtreg headers, I think it is worth including the JBS bug ID 8272194 in the test comments to better associate the change with the test.
| LocalDate ld1 = LocalDate.of(-4, 1, 1); | ||
| Date d1 = Date.valueOf(ld1); | ||
| LocalDate ld2 = d1.toLocalDate(); | ||
| assertTrue(ld1.equals(ld2), "Error ld1 != ld2"); |
There was a problem hiding this comment.
You might use assertEquals here and in the other occurrences as a more accurate assertion. Perhaps also update the error message to make it clear that the failed assertion is about negative years.
There was a problem hiding this comment.
Done. I have also improved the test adding more coverage and improving the error messages.
I have also moved away from the negative year nomenclature as that obfuscates year 0 (1BC).
I have added tests to ensure we are checking this year 0 case as well, as it is an obvious possible boundary condition.
…ocalDateTime conversion
…ateTime conversion
| year = 1 - calendar.get(Calendar.YEAR); | ||
| } else { | ||
| year = calendar.get(Calendar.YEAR); |
There was a problem hiding this comment.
getYear() is still bypassed on this path. The calendar should only be used to determine the era. The year value should continue to be obtained through getYear().
There was a problem hiding this comment.
Can I just double check, do you think that we should be using getYear() for the BC path as well?
There was a problem hiding this comment.
Yes, I believe year should always be derived from getYear().
There was a problem hiding this comment.
Cool I have made that changed. I have also moved away from the if-else block to only a BC if block. This brings the logic inline with how it is done in the GregorianCalendar class.
I realise that with always using getYear() we could also create a faster code path for BC dates by also adding a year 2 BC epoch millisecond and checking if getTime() is less than that. That way we only do the calendar check for the narrow band of dates we cannot be certain are BC or AD. However, I recognise this is an extra optimisation which few consumers would benefit from.
There was a problem hiding this comment.
Yes, getYear() should always be called, regardless of the era. Otherwise, this would introduce a behavioral change. BC dates could also use the fast path, but the added benefit is limited and likely not worth doubling the complexity.
| List<LocalDateTime> bcLocalDateTimes = List.of( | ||
| LocalDateTime.of(0, 1, 1, 0, 0, 0, 0), | ||
| LocalDateTime.of(-4, 9, 8, 23, 11, 59, 999_999_999), | ||
| LocalDateTime.of(-1000, 3, 22, 8, 30, 20, 0) | ||
| ); |
There was a problem hiding this comment.
Could you make this a JUnit parameterized test, using @ParameterizedTest, rather than iterating over a List? Please also add dates/date-times immediately around the cutoff to verify the conversion there, particularly that values at or after the cutoff are AD, even with the possible negative offset?
There was a problem hiding this comment.
I have made the tests @ParameterizedTest
There was a problem hiding this comment.
I have also added tests to check the dates around our BC check threshold.
| assertEquals(bcTimestamp.toLocalDateTime(), bcLocalDateTime, "The LocalDateTime created from the Timestamp does not match the original BC LocalDateTime."); | ||
| assertEquals(Timestamp.valueOf(bcTimestamp.toLocalDateTime()), bcTimestamp, "The BC Timestamp did not yield the expected result on a round trip Timestamp / LocalDateTime conversion."); |
There was a problem hiding this comment.
The expected and actual arguments to assertEquals appear to be reversed.
There was a problem hiding this comment.
I have fixed this while making the @ParameterizedTest change.
…e/LocalDateTime conversion
…onversion parameterized
|
@Sholto Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration. See OpenJDK Developers’ Guide for more information. |
…conversion parameterized
justin-curtis-lu
left a comment
There was a problem hiding this comment.
Thanks for your changes so far. These are the only remaining issues I spotted.
| getMinutes(), | ||
| getSeconds(), | ||
| getNanos()); | ||
| getNanos()); // We have to use nanos here as the calendar does not support nano precision |
There was a problem hiding this comment.
This comment is stale and can be removed.
There was a problem hiding this comment.
Good spotting. I've removed that now.
| * Validate that Date.valueOf and Date.toLocalDate yield the expected results for dates with BC years. | ||
| * Ensures that the fix for 8272194 has not regressed. | ||
| */ | ||
| @ParameterizedTest(autoCloseArguments = false) |
There was a problem hiding this comment.
I understand you are following the existing test convention, but I would remove the autoCloseArguments = false here and in the other occurrences since the parameter does not implement AutoCloseable, so it's just noise. (It is added to those other tests even though not needed as part of an earlier migration to conservatively match TestNG semantics.)
There was a problem hiding this comment.
That is good to know. I have removed them from the tests I have added, but I have left the existing uses alone.
justin-curtis-lu
left a comment
There was a problem hiding this comment.
The latest changes look good to me. But please also let @naotoj finish taking a look as well.
|
/reviewers 2 |
|
@justin-curtis-lu |
Excellent. I am just wondering, what is the next step of the process? |
justin-curtis-lu
left a comment
There was a problem hiding this comment.
After the PR is approved, we generally wait around 24-hours and if there are no further comments, the author can comment /integrate to proceed with integrating the pull request. After which, one of us can sponsor the change.
But before you do that, a few final suggestions =). Following some offline discussion, we think that the core logic is in good shape, but there is opportunity to reduce some duplication. Please see the other comments I left.
| * timezone offset, it gives us a comfortable distance while still | ||
| * providing a faster code path for almost 1970 years. | ||
| */ | ||
| private static final long TWO_AD_AT_UTC_EPOCH_MILLIS = -62104233600000L; |
There was a problem hiding this comment.
We can make this package-private and have Timestamp use it as well to reduce some duplication.
There was a problem hiding this comment.
I've left this private as Timestamp no longer needs this constant as it is directly using the new Date.toProlepticYear.
| // Given that we can comfortably state that any dates on or after | ||
| // 0002-01-01 are AD, we can use a much faster local date derivation | ||
| // for these dates. | ||
| if (getTime() >= TWO_AD_AT_UTC_EPOCH_MILLIS) { |
There was a problem hiding this comment.
We can pull out the shared logic here and in Timestamp and create a static package-private helper in java.sql.Date.
Something like
static int toProlepticYear(long millis, int year) {
if slow path
set calendar using setTimeInMillis
if calendar is in BC
adjust year
return year;
}
The helper can call GregorianCalendar.setTimeInMillis(...) instead of GregorianCalendar.setTime(...). That lets us pass its millisecond value via getTime() rather than having to pass this as a java.util.Date to the helper method. I also think within the helper, you can inverse the getTime() >= TWO_AD_AT_UTC_EPOCH_MILLIS conditional to simplify the logic some more.
After that, toLocalDate need only look something like,
return LocalDate.of(
toProlepticYear(getTime(), getYear() + 1900),
getMonth() + 1,
getDate());
There was a problem hiding this comment.
Yup great idea. I have updated this as suggested.
There was a problem hiding this comment.
Did we want unit tests specifically for the new toProlepticYear method? Or are we happy with the coverage provided by the existing dateTimesAroundBcCheckThreshold checks?
There was a problem hiding this comment.
I also just wanted to confirm if toProlepticYear is the correct nomenclature.
There was a problem hiding this comment.
Thanks for your update, it looks good to me. I'll run your change on our CI.
Did we want unit tests specifically for the new toProlepticYear method?
I don't think we need white-box tests for the helper method, the existing public API tests are sufficient.
I also just wanted to confirm if toProlepticYear is the correct nomenclature.
I think this name is correct. We are taking the Gregorian era and year and convert it to the ISO proleptic year. That is how the converted java.time classes refer to the year.
https://docs.oracle.com/en/java/javase/26/docs//api/java.base/java/time/LocalDate.html#getYear()
https://docs.oracle.com/en/java/javase/26/docs//api/java.base/java/time/LocalDateTime.html#getYear()
There was a problem hiding this comment.
I think this name is correct. We are taking the Gregorian era and year and convert it to the ISO proleptic year. That is how the converted java.time classes refer to the year.
Okay, scratch that.
We are just deriving the java.time value from the date's values for year, month, etc. Technically, the conversion to the ISO calendar system is not exact, since GregorianCalendar is a hybrid Julian-Gregorian calendar.
We might better name it as toGregorianProlepticYear, which only indicates that the year and era are used to derive a Gregorian proleptic year (and not an exact ISO date conversion).
There was a problem hiding this comment.
Ok I've renamed that as suggested.
…lDate/LocalDateTime conversions
Also covers: 8249280
I will first give a quick summary of the problem.
Put simply, the
LocalDateform of thejava.sql.Dateis derived using thegetYearmethod ofjava.util.Date. This in turn returns the year of the normalised internal calendar.However, the internal calendar
getYearhas an extra layer of complexity.The calendar has an additional era field, which captures BC/AD.
getYeartherefore just returns the year of that era.For example, the year 6BC and the year 6AD both return
getYearas 6.This means that for BC dates, our
LocalDateconversion loses the sign of the year.This leads to additional problems down the line, as the year 1BC is for calculations sake is considered to be year 0 (and 2BC us considered year -1 and so on). As a result, the various leap year calculations are WRONG for these years, causing year format validation failures in situations like marshalling/unmarshalling the dates with a DB.
There are two seemingly obvious fixes here, however I will attempt to explain why I did not proceed with them.
Firstly, it seems sensible is to derive the
LocalDatefrom anInstantcreated from the millisecond representation of theDate. After all, why we are having to use the deprecatedgetYear,getMonthandgetDaymethods anyway?The answer lies in 8061577.
The underlying millisecond representation between
java.time.Instantandjava.util.Dateis fundamentally different. Read that ticket for a greater explanation.Ultimately though, it means that the for older dates, the only real way to bridge between the two calendar systems is to use these year/month/day methods.
This is where the second possible solution appears.
The underlying calendar representation that
java.util.Dateuses actually does have a year method which gives you the correctly signed year, that beinggetNormalizedYear.In fact,
java.util.Dateuses the setter counterpartsetNormalizedYearis itssetYearmethod.Given this, it seems natural that
getYearshould similarly callgetNormalizedYear.I think this would be my ideal solution, however I recognise that
getYear only returning a positive year is very long standing behaviour. Given how widely spreadjava.util.Dateis, I felt it was perhaps better not to rock the boat too much.I have therefore taken the decision to add an equivalent
getNormalizedYearmethod tojava.util.Date.This is the simplest way to expose the real inner year of the date to
java.sql.Dateandjava.sql.Timestamp.This fixes the negative year issue in the LocalDate conversion in an incredibly low impact way, and without altering long standing
java.util.Datebehaviour.I chose to make the method
protectedto prevent unwanted consumption of the method. However, I did not make itDeprecatedlike the other accessor methods which perhaps I should have done.Let me know what you think.
Progress
Warning
8272194: java.sql.Date::toLocalDate() broken for dates before 1 A.D.Issue
Reviewers
Reviewing
Using
gitCheckout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/31808/head:pull/31808$ git checkout pull/31808Update a local copy of the PR:
$ git checkout pull/31808$ git pull https://git.openjdk.org/jdk.git pull/31808/headUsing Skara CLI tools
Checkout this PR locally:
$ git pr checkout 31808View PR using the GUI difftool:
$ git pr show -t 31808Using diff file
Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/31808.diff
Using Webrev
Link to Webrev Comment