Why your Domino servers might have been using 30% more CPU recently – and how to fix it

A success story you can be part of – with the FREE panagenda MarvelClient Essentials

Shortly before Christmas 2018, one of our customers noticed a 25-30% increase of CPU load on all of their (many) IBM Domino servers. The first suspect was their recent server upgrade to Domino 9.0.1 FP9. However, (spoiler alert!) it turns out this issue was caused by the Notes clients, and not related to the server at all!

Together with the customer, we went to investigate the issue using panagenda GreenLight. By looking at the number of sessions and users in the time period from a month before the upgrade and up to two weeks thereafter, it was evident that both the number of users and sessions per server had started to increase quickly.

Since we do not share customer data, the following shows similar data from our very own, much smaller environment with less than 100 users:

From the red trendline, you can see an increase of CPU usage – Domino statistic platform.system.PctCombinedCpuUtil – over the before mentioned time period.

It doubled from an average of 5 percent to 10 percent.

Next, we started investigating the issue on one IBM Notes client by using client clocking (via CLIENT_CLOCK=1 and additional debug parameters in notes.ini). The resulting log file revealed that the client would open many different user mail files upon client startup and regularly thereafter.

Wondering where this could have come from, we soon discovered numerous entries in notes.ini similar to the following:

We found as many as 35 such entries in notes.ini. After deleting these entries, the client stopped opening all the corresponding mail files.

After opening up a PMR with IBM, we were informed that the issues were caused by a new feature added in Notes 9.0.1 FP9 called “Delegated mail files are refreshed automatically“. Even though the documentation states “Ask your administrator to enable this feature.”, the functionality is turned on by default.

When an end-user opens another user’s calendar, the notes client will automatically add the above mentioned DELEGATED_* entries to notes.ini. This can happen for various reasons, such as:

Using the “Find Available Times” feature and then right-clicking on “open calendar”

By directly opening another user’s calendar

By using calendar federation

Naturally, a user must have access to another user’s calendar. In the case of the customer, all users can access the calendars of all other users to ensure optimal collaboration.

We then looked at the MarvelClient Analyze database to see how many entries like that exist across the entire Notes client landscape – which you can easily do on your own by using MarvelClient Essentials!

As you can see in the screenshot of the Database, even in our own small environment there are a large number of DELEGATED_MAIL_* entries.  Our customer had many more:

Download MarvelClient Essentials, check if your system is affected and ensure optimum performance and business continuity!

Find out how:

Finally, we set out to create an Action in the MarvelClient Config database to clean up the issue …

… and configured the Action as follows:

The first line sets DEBUG_DISABLE_AUTO_REFRESH_DELEGATED_INBOX=1, thereby disabling automatic refreshing of “delegated” mailboxes in general – a notes.ini setting IBM shared with us whilst working on the PMR.

The second line removes any entries that begin with “DELEGATED_” – a very helpful feature to cleanup notes.ini files, brought to you by panagenda MarvelClient, and included in the free MarvelClient Essentials (try that with Policies :-P).

We removed over 670,000 notes.ini entries and reduced CPU load by 20% and more across several hundred servers. For further reference regarding the relevant notes.ini settings, here is the original PMR in Japanese and translated using Google.

Are you sure that you’re on top of your Notes environment?