The way incremental collections work is that config manager starts by evaluating the “All Systems” collection then it evaluates all the collections limited to “All Systems” and it keeps on like that until it has evaluated all the collections which have incremental updates enabled.
This takes between 0,3 seconds and 2,5 seconds for each collection depending on:
-Complexity of membership rules
-Total number of collections
-Total number of clients
-Frequency of recources being added or changed
Since collection evaluation is single threaded, it only evaluates one collection at any given time.
We have about 2700 collections in our environment. So if we had enabled incremental update on all collections – Which we had at one point – (2700 collection x 2 seconds in average) it would take about 90 minutes for the cycle to complete. Given that the default refresh rate for incremental evaluation is 5 minutes you can see that this would not work..
This is the reason “Best Practices” states you should not have more than 200 incremental collections.
For us there was a bit of decision-making to do in regards to what collections really needed to be incrementally updated and what to schedule – and how often.
Basically all our “top level” and OSD collections need to be incremental – the rest could be on a schedule.
We created a script that scheduled most of our collections to be updated once every night at a random time between 02:00 and 05:00.
Other collections were scheduled to update every 3 hour and some every hour.
We can really “feel” the increase in performance on the site server and actually also in the console.
If you are not sure if you have too many incremental collections my tip is to open the coleval.log in cmtrace on the site server and filter on “All Systems” and check how often it is evaluated. If it is every 5 minutes (default value) you are good. If not you need to do something..
To see how many collections you have with the different kind of refresh types you can run this SQL query:
Select (Case when RefreshType = 1 then 'Manual'
when RefreshType = 2 then 'Scheduled'
when RefreshType = 4 then 'Incremental'
when RefreshType = 6 then 'Scheduled and Incremental'
else 'Unknown' end) as RefreshType, count(SiteID) as Collections
from v_Collections
group by RefreshType
To see which refresh type each collection has:
Select SiteID, CollectionName,
(Case when RefreshType = 1 then 'Manual'
when RefreshType = 2 then 'Scheduled'
when RefreshType = 4 then 'Incremental'
when RefreshType = 6 then 'Scheduled and Incremental'
else 'Unknown' end) as RefreshType
from v_Collections