Domino on CentOS/RHEL compatible Linux - Timezone issues

Wednesday, May 17, 2023 at 7:54 PM UTC

Backstory

With a new customer server we stumbled upon a strange behaviour using our low-code platform Aveedo we never saw before. We created the server at Hetzner using the Rocky 9 base installation. Rocky is binary compatible with RHEL and should be used in favour of CentOS in general as it is newer and still maintained. Domino installs and runs fine on Rocky. The version of choice was 12.0.2 with Fixpack 1. We do not use additional features like Nomad or Leap at the moment.

Aveedo, Java backend

With our custom CRM app built in Aveedo we faced some issues with timestamps, i.e. date fields of all kinds. The system fields Created and Modified were ok, but we also have own fields to store these date values in. On top of that we also have documents that hold other date fields which you can freely set to any date and/or time like an order date or birthdays.

But let's stick with the system values and the corresponding custom fields: we have our own doc-info functionality and also a control that can display all fields, like the Notes-own document properties window. All is based on XPages technology with a custom backend.

Using those, we saw very different values, but have a look for yourself:

The first thing that strikes the eye is the creation date being 2 hours earlier and the modified date even 2 more hours earlier - which totally doesn't make sense at all.

Even worse

This is just a display issue you might think when you look at the field values - but this is not the only "effect". When you use these date values (or the ones on other, non-system like fields) to e.g. print them in a Word document, the values were also wrong. For example, a field holding just a date w/o any time also was shifted 2 hours back (based on an "imaginary" 12:00 AM time value) - which then brought the printed date to a whole day early. For example: date is set to 17.05.2023 (12:00 AM is added somehow without being in the field), minus 2 hours = 16.05.2023, 10:00 PM. Then the wrong date is used later.

Tests

After looking at the code behind all this, nothing suspicious was found and the only approach to narrow it down was to test the same environment on different operating systems. We tested other versions of Rocky (8), even CentOS 7 - all had the same issue. Ubuntu, though not supported, did not.

Domino on Docker, based on the Rocky Linux base image, did not show this behaviour - because the build script for the image takes care of what I will show you now - in this way or another. If you are interested in details, please ask Daniel Nashed - I actually have no clue what's going on there ;-)

Preperations

Though setting the timezone of the host using

timedatectl set-timezone Europe/Berlin

and checking if everything was set correctly

we still had this issue. By the way: this all has nothing to do with locale settings of the OS, locale is only responsible when it comes to date formatting. Aveedo in this case takes care of the date formatting when being displayed, so this is a no-brainer to us.

The cause

The missing part was the content of the file /etc/timezone which contained the value Etc/UTC even after setting the timezone using timedatectl.

Again, Daniel gave me the final hint here so then I changed this value to Europe/Berlin, too - matching the timezone we already had set.

One Domino restart later, the issue was gone (almost):

The custom modified date you see here is only fixed once you save the document again, otherwise it still displays a value 2 hours early than the stored value:

 

This is still strange but it somehow fits to the fact that this field was displayed with a value of minus 4 hours before, so the cure is to re-save the document again to get the corrected value.

Resume

This was a very huge WTF and it took us several days for all the tests we made. It's quite ridiculous that the reason was only this litte file/value that caused us some very busy days. So please make sure you put this on your checklist when you install Domino on the OS mentioned above even if you run Domino in the UTC timezone - then set it to Europe/London.







Leave a comment right here