Note: This text is provided as a courtesy. It has been automatically translated using software and may not have been proofread. The English language version is considered the official version and you can find the most up-to-date information there.

Why your Domino servers might have been using 30% more CPU recently – and how to fix it

There is new behavior in Notes 9.0.1 FP9 and later that leads to unnecessary load on your Domino servers. Read on to learn what it is, how to discover if you are affected, and how to fix it easily!

by Markus Sablatnig

A success story you can be part of – with the FREE panagenda MarvelClient Essentials

There is a new behavior in Notes 9.0.1 FP9 and later that leads to unnecessary load on your Domino servers. Read on to learn what it is, download MarvelClient Essentials, discover instantly if you are affected, and fix it easily!

Shortly before Christmas 2018, one of our customers noticed a 25-30% increase of CPU load on all of their (many) IBM Domino servers. The first suspect was their recent server upgrade to Domino 9.0.1 FP9. However, (spoiler alert!) it turns out this issue was caused by the Notes clients, and not related to the server at all!

What’s going on?

Together with the customer, we went to investigate the issue using panagenda GreenLight. By looking at the number of sessions and users in the time period from a month before the upgrade and up to two weeks thereafter, it was evident that both the number of users and sessions per server had started to increase quickly.

Since we do not share customer data, the following shows similar data from our very own, much smaller environment with less than 100 users:

Chart created in Excel via CSV export from GreenLight

From the red trendline, you can see an increase of CPU usage – Domino statistic platform.system.PctCombinedCpuUtil – over the before mentioned time period.

It doubled from an average of 5 percent to 10 percent.

Finding the cause

Next, we started investigating the issue on one IBM Notes client by using client clocking (via CLIENT_CLOCK=1 and additional debug parameters in notes.ini). The resulting log file revealed that the client would open many different user mail files upon client startup and regularly thereafter.

Wondering where this could have come from, we soon discovered numerous entries in notes.ini similar to the following:

DELEGATED_MAIL_FILEn=firstname lastname_server/certifier!!mail\firstlast.nsf_DD.MM.YYYY

We found as many as 35 such entries in notes.ini. After deleting these entries, the client stopped opening all the corresponding mail files.

After opening up a PMR with IBM, we were informed that the issues were caused by a new feature added in Notes 9.0.1 FP9 called “Delegated mail files are refreshed automatically“. Even though the documentation states “Ask your administrator to enable this feature.”, the functionality is turned on by default.

When an end-user opens another user’s calendar, the notes client will automatically add the above mentioned DELEGATED_* entries to notes.ini. This can happen for various reasons, such as:

Using the “Find Available Times” feature and then right-clicking on “open calendar”

By directly opening another user’s calendar

By using calendar federation

Naturally, a user must have access to another user’s calendar. In the case of the customer, all users can access the calendars of all other users to ensure optimal collaboration.

We then looked at the MarvelClient Analyze database to see how many entries like that exist across the entire Notes client landscape – which you can easily do on your own by using MarvelClient Essentials!

As you can see in the screenshot of the Database, even in our own small environment there are a large number of DELEGATED_MAIL_* entries.  Our customer had many more:

a whopping 670,000 such notes.ini entries
across several 10,000 users.

Fixing the problem

Download MarvelClient Essentials, check if your system is affected and ensure optimum performance and business continuity!

Find out how:

Finally, we set out to create an Action in the MarvelClient Config database to clean up the issue …

… and configured the Action as follows:

The first line sets DEBUG_DISABLE_AUTO_REFRESH_DELEGATED_INBOX=1, thereby disabling automatic refreshing of “delegated” mailboxes in general – a notes.ini setting IBM shared with us whilst working on the PMR.

The second line removes any entries that begin with “DELEGATED_” – a very helpful feature to cleanup notes.ini files, brought to you by panagenda MarvelClient, and included in the free MarvelClient Essentials (try that with Policies :-P).

The result

We removed over 670,000 notes.ini entries and reduced CPU load by 20% and more across several hundred servers. For further reference regarding the relevant notes.ini settings, here is the original PMR in Japanese and translated using Google.

Are you sure that you’re on top of your Notes environment?

Comments (9 Comments)
  • Johnny Oldenburger says:

    Problem solved with the great help of Florian. DELEGATED Entries are deleted after restart of the Notes Client. Thanks for your super-fast support!

  • Marc Thomas says:

    It depends on the selected runtype in the action. If you leave the setting as is, it should right after login executed and deleted / change the mentioned INI entries.

  • Johnny Oldenburger says:

    I created the document as described above. When are the delegated entries deleted in the NOTES.INI files. I restarted the Notes Clients but until now the entries are still in the NOTES.INI files

  • Lars Berntrop-Bos says:

    Hi Florian, I have the checkquotas library. While I am not assigned delegated access to any other mail files, I thought that one of the the scenarios listed under ‘This can happen for various reasons, such as’ would also generate trigger the behaviour. In my case, I can remember using all three methods in the recent past, but I cannot find any INI lines with ‘DELEGATED’ in the setting name.

  • Vladimir says:

    Great job! Thank you a lot! I`ll check this case in my production envirement.

  • Florian Vogler says:

    Hi Lars, all mail files (=those that you open) must have the latest design, too – see CheckQuotas Scriptlibrary, CheckQuota\SetCalMgrINI in mail file design. Looking forward to your feedback!

  • Lars Berntrop-Bos says:

    Strange. I am running the FP9 client since it was published. Also, my mail file has the FP9 design. But I have no DELEGATED entries in my Notes.ini.. And I have links to other agenda’s, and have opened other agenda’s. There must be an additional trigger.

  • Lars Berntrop-Bos says:

    What is the trigger? i am running the 9.0.1 FP9 client since it was published, have opened other calendars, but do not have any DELEGATED* ini settings.

  • Fred Kuijpers says:

    Great article!
    Thanx a lot for this info.

Leave a Reply

Find more information in our Privacy Policy.