A success story you can be part of – with the FREE panagenda MarvelClient Essentials

There is a new behavior in Notes 9.0.1 FP9 and later that leads to unnecessary load on your Domino servers. Read on to learn what it is, download MarvelClient Essentials, discover instantly if you are affected, and fix it easily!

Shortly before Christmas 2018, one of our customers noticed a 25-30% increase of CPU load on all of their (many) IBM Domino servers. The first suspect was their recent server upgrade to Domino 9.0.1 FP9. However, (spoiler alert!) it turns out this issue was caused by the Notes clients, and not related to the server at all!

What’s going on?

Together with the customer, we went to investigate the issue using panagenda GreenLight. By looking at the number of sessions and users in the time period from a month before the upgrade and up to two weeks thereafter, it was evident that both the number of users and sessions per server had started to increase quickly.

Since we do not share customer data, the following shows similar data from our very own, much smaller environment with less than 100 users:

Chart created in Excel via CSV export from GreenLight

From the red trendline, you can see an increase of CPU usage – Domino statistic platform.system.PctCombinedCpuUtil – over the before mentioned time period.

It doubled from an average of 5 percent to 10 percent.

Finding the cause

Next, we started investigating the issue on one IBM Notes client by using client clocking (via CLIENT_CLOCK=1 and additional debug parameters in notes.ini). The resulting log file revealed that the client would open many different user mail files upon client startup and regularly thereafter.

Wondering where this could have come from, we soon discovered numerous entries in notes.ini similar to the following:

DELEGATED_MAIL_FILEn=firstname lastname_server/certifier!!mail\firstlast.nsf_DD.MM.YYYY

We found as many as 35 such entries in notes.ini. After deleting these entries, the client stopped opening all the corresponding mail files.

After opening up a PMR with IBM, we were informed that the issues were caused by a new feature added in Notes 9.0.1 FP9 called “Delegated mail files are refreshed automatically“. Even though the documentation states “Ask your administrator to enable this feature.”, the functionality is turned on by default.

When an end-user opens another user’s calendar, the notes client will automatically add the above mentioned DELEGATED_* entries to notes.ini. This can happen for various reasons, such as:

Using the “Find Available Times” feature and then right-clicking on “open calendar”

By directly opening another user’s calendar

By using calendar federation

Naturally, a user must have access to another user’s calendar. In the case of the customer, all users can access the calendars of all other users to ensure optimal collaboration.

We then looked at the MarvelClient Analyze database to see how many entries like that exist across the entire Notes client landscape – which you can easily do on your own by using MarvelClient Essentials!

As you can see in the screenshot of the Database, even in our own small environment there are a large number of DELEGATED_MAIL_* entries.  Our customer had many more:

a whopping 670,000 such notes.ini entries
across several 10,000 users.

Fixing the problem

Download MarvelClient Essentials, check if your system is affected and ensure optimum performance and business continuity!

Find out how:

Finally, we set out to create an Action in the MarvelClient Config database to clean up the issue …

… and configured the Action as follows:

The first line sets DEBUG_DISABLE_AUTO_REFRESH_DELEGATED_INBOX=1, thereby disabling automatic refreshing of “delegated” mailboxes in general – a notes.ini setting IBM shared with us whilst working on the PMR.

The second line removes any entries that begin with “DELEGATED_” – a very helpful feature to cleanup notes.ini files, brought to you by panagenda MarvelClient, and included in the free MarvelClient Essentials (try that with Policies :-P).

The result

We removed over 670,000 notes.ini entries and reduced CPU load by 20% and more across several hundred servers. For further reference regarding the relevant notes.ini settings, here is the original PMR in Japanese and translated using Google.

Are you sure that you’re on top of your Notes environment?