fix(recordchecker): potential precision issues + better logs (google#5135)

According to docs, [GCS Custom-Time is millisecond
precision](https://docs.cloud.google.com/storage/docs/metadata#custom-time),
[Datastore is
microsecond](https://docs.cloud.google.com/datastore/docs/reference/data/rest/Shared.Types/Value#:~:text=A%20timestamp%20value.%20When%20stored%20in%20the%20Datastore%2C%20precise%20only%20to%20microseconds%3B%20any%20additional%20precision%20is%20rounded%20down.)
(though Datastore now uses Firestore which is apparantly nanosecond
precision 🤷).

I'm wondering if this is causing issues with the recordchecker thinking
things are out of date, but I would expect this to loop infinitely if
that were the case (unless something else writing is
millisecond-precise?)

Added millisecond leeway and some better logging for this.

Author: michaelkedar

go/cmd/recordchecker/recordchecker.go

@@ -293,15 +293,24 @@ func checkRecord(ctx context.Context, cl *datastore.Client, storageCl clients.Cl
 	attrs, err := storageCl.ReadObjectAttrs(ctx, fmt.Sprintf("all/pb/%s.pb", id))
 	if err != nil {
 		res.needsRetry = true
-		if !errors.Is(err, clients.ErrNotFound) {
+		if errors.Is(err, clients.ErrNotFound) {
+			logger.InfoContext(ctx, "record not found in gcs", slog.String("id", id))
+		} else {
 			// Log the error if it's not the expected "not found" error.
 			res.err = fmt.Errorf("failed to read object for %s: %w", id, err)
 		}
 
 		return res
 	}
 
-	if attrs.CustomTime.Before(vuln.Modified) {
+	// CustomTime is millisecond precision, while Modified is microsecond precision.
+	// So we add a millisecond to CustomTime to account for the difference.
+	if attrs.CustomTime.Add(time.Millisecond).Before(vuln.Modified) {
+		logger.InfoContext(ctx, "record is stale in gcs",
+			slog.String("id", id),
+			slog.String("custom_time", attrs.CustomTime.String()),
+			slog.String("modified", vuln.Modified.String()),
+		)
 		res.needsRetry = true
 	}