How to determine the best "simultaneous index operations" configuration?
What is "simultaneous index operations"?
"Simultaneous index operations", also known as "Indexing threads", is a setting in ConfigHub which allows you to configure how many indexing tasks TrendMiner can process in parallel.
This setting is global across all data sources and all types of index tasks (historical indexing and continuous indexing).
The default is 2 which means that at any point in time a maximum of 2 index tasks can be executed in parallel. Other index tasks are queued for processing.
How to determine the best setting for your installation?
If the simultaneous index operations is configured too low it will take too much time to complete all index tasks and indexing might get congested which could result in monitoring delays and users complaining about slow performance or missing data.
If the simultaneous index operations is configured too high it might occupy too many system resources, resulting in degraded performance for other TrendMiner functionality or even services which become unstable because they do not have enough resources. And even if the TrendMiner server has sufficient resources the bottleneck might move towards the data sources and they can become congested and unresponsive because of the many data requests which are sent by TrendMiner. Putting more pressure on a data source typically makes it slower so increasing the simultaneous index operations too much can have an opposite effect and decrease the indexing performance.
To guarantee the best possible performance in TrendMiner it is important to take into account all parameters and find the happy medium.
The following factors influence the optimal setting:
Data source response time: the faster a data source can return data the more indexing threads can be configured. If you notice the response times increase when you increase the indexing threads this typically is an indication that the data source is stressed too much. In this case it is advised to either lower the indexing threads or increase the data source resources (CPU/memory).
Number of indexed tags: the more tags need to be kept up to date, the more indexing threads are required to make sure the indexing system does not congest. If you notice delayed tags it is advised to increase the indexing threads.
Number of monitors: also the indexing policy of a tag, and more specifically the number of monitored tags, have a big influence on the indexing capacity as they determine the frequency of the index updates. The more monitored tags, the more indexing threads are needed.
Available CPUs: each indexing thread needs a CPU to process the data. The CPU power is shared between all TrendMiner services so determining how many CPU power is available for indexing is a case by case exercise. In any case the number of indexing threads configured should never exceed the number of available CPUs, and preferably some CPU power is left over for other services to avoid performance degradation. In Edge Manager the CPU usage can be monitored. If the CPU usage is too high it makes sense to either increase the available CPUs or decrease the indexing threads.
Available memory: the more data is being processed in parallel, the more memory is required to store the data which is retrieved from the data source before it is written into index files. If services become unstable (start crashing) it could be an indication that too much memory is used for indexing. The required memory also depends on the index resolution (the higher the index resolution the more memory is required). Although the memory is reserved per service it is not trivial to give exact calculations on the required memory for indexing since the same memory/service also processes other tasks like fetching live data for charting, checking permissions, synchronizing tags from data sources, ...
Parallel connections: the number of parallel connections determines how many parallel connections can be set up to the data source to retrieve the index data. If many index tasks from the same data source are processed in parallel and the parallel connections for that data source is too low, the data requests will pile up in the data source queue and the indexing threads will stay idle, waiting for the requests to be processed. If CPU usage is low and the data source does not show congestion symptoms (slower response times, timeouts, ...) it might be good to increase the parallel connections to that data source. In general we would advise to keep the parallel connections aligned to the indexing threads (unless the data source is stressed too much in which case the parallel connections to that specific data source can be lowered).
Examples
Here we list some example configurations which allow you to assess how many simultaneous index operations should be configured on your instance to avoid index delays. In the example tables the left column shows the number of indexed tags. The following columns show the data sources required response times in milliseconds based on the number of indexing threads which are required to be able to index all tags without delay. E.g. example table 1: #tags = 50K and average data sources response time = 500ms => configure 8 simultaneous index operations.
Example 1: in this example we assume that all tags have the default index policy of 1 hour. If 100K tags needs to update hourly and there is only 1 index thread, each update should be completed in 36ms (3.600.000 milliseconds to complete 100.000 index tasks) to be able to finish all requests within the hour. The table below shows more combinations.
# Tags | 1 thread | 4 threads | 6 threads | 8 threads | 10 threads | 16 threads | 20 threads |
|---|---|---|---|---|---|---|---|
5K | 720 | 2880 | 4320 | 5760 | 7200 | 11520 | 14400 |
10K | 360 | 1440 | 2160 | 2880 | 3600 | 5760 | 7200 |
20K | 180 | 720 | 1080 | 1440 | 1800 | 2880 | 3600 |
50K | 72 | 288 | 432 | 576 | 720 | 1152 | 1440 |
100K | 36 | 144 | 216 | 288 | 360 | 576 | 720 |
200K | 18 | 72 | 108 | 144 | 180 | 288 | 360 |
500K | 7 | 29 | 43 | 58 | 72 | 115 | 144 |
1M | 4 | 14 | 22 | 29 | 36 | 58 | 72 |
Example 2: In the first example we assumed all tags have the default index policy of 1 hour. In an average installation however some tags will have a reduced index policy of 1 day (instead of 1 hour) which will positively impact the index performance, and some tags will be used in a monitor and will have a 2 minute index policy, which will negatively impact the index performance.
In this example we assume that 2% of the (indexed) tags are monitored (with a max of 1000) with a 2 min policy; 80% degraded tags with a 1 day policy and 18% default tags with a 1 hour policy.
# Tags | 1 thread | 4 threads | 6 threads | 8 threads | 10 threads | 16 threads | 20 threads |
|---|---|---|---|---|---|---|---|
5K | 885 | 3541 | 5311 | 7082 | 8852 | 14164 | 17705 |
10K | 443 | 1770 | 2656 | 3541 | 4426 | 7082 | 8852 |
20K | 221 | 885 | 1328 | 1770 | 2213 | 3541 | 4426 |
50K | 89 | 354 | 531 | 708 | 885 | 1416 | 1770 |
100K | 70 | 281 | 421 | 561 | 701 | 1122 | 1403 |
200K | 50 | 198 | 297 | 396 | 495 | 793 | 991 |
500K | 26 | 105 | 158 | 211 | 263 | 421 | 527 |
1M | 15 | 59 | 89 | 118 | 148 | 237 | 296 |
Conclusion
Determining the optimal setting requires some thought and should be assessed based on the above parameters. The following guidelines give a high level indication on the best possible setting for your setup:
If there are no delayed tags no need to change the simultaneous index operations setting.
If increasing the number of simultaneous index operations does not improve the delays, check if the parallel connections setting (global or for the most important data sources) can be increased.
If increasing the number of simultaneous index operations worsens the delays, lower the number of simultaneous index operations again and/or increase the resources of your data source(s).
When increasing the number of simultaneous index operations, do not overshoot it:
Preferably increase with 1 or 2 at a time to see the effect.
Wait at least 24 hours before evaluating the change. The queues need time to process.
If you notice performance degradation in other parts of TrendMiner, lower the simultaneous index operations again and/or add more system resources.
Use the above table from example 2 to get a realistic indication on a sensible setting. E.g. if you have 50K indexed tags and your data sources return data on average within 1 second, you might want to start with 10 simultaneous index operations and check if delays decrease when increasing the index threads further.
Never configure more indexing threads as there are available CPUs.
As an alternative workaround to solve index delays you can also consider deleting some tag indexes for tags which are not used. If a tag index is deleted is will be rebuilt as soon as a user uses it again so deleting one tag index too much is typically no problem.