Diagnostics
This section allows administrators to monitor the synchronization status of context-capable data sources. Both live and historical sync can be enabled under data sources, and their status can be viewed in detail here.
TrendMiner uses several sync types to keep context items aligned with their source system:
Live synchronization
Live synchronization keeps context items up to date by polling the data source every 20 seconds for event frames that have been created or modified. Each interval retrieves up to 200 event frames, with a maximum of 800 per interval. If more than 800 event frames are detected in a single interval, the processing is automatically handed off to an excessive interval sync to avoid delaying the next cycle.
Excessive interval sync
Excessive interval sync handles large batches of event frames that exceed the live sync threshold. It runs in parallel with live sync so that high-volume intervals do not block the regular polling cycle.
Historical synchronization
Historical synchronization allows administrators to resynchronize event frames from a specific period in the past. It runs in parallel with live sync and excessive interval sync and must be triggered manually from ConfigHub.
Cleanup processes
Cleanup processes run automatically in the background to maintain data quality. They check open context items, remove items that no longer exist in the source system, and resolve items that could not be fully processed during a previous sync.
All sync types are driven by the last modified date of event frames in the source system, not by their start or end date. An event frame is only picked up if its last modified date falls within the sync interval. This applies to both newly created and recently modified event frames, as creation also sets the last modified date.
Additionally, automatic cleanup processes, such as the nightly cleanup and resync of context items, run in the background.
For every context-capable data source with live synchronization enabled, TrendMiner polls the data source every 20 seconds for event frames whose last modified date falls within the current interval. Event frames are processed in the order they are received. Each data source with live sync enabled runs on a dedicated thread, so multiple data sources sync independently and in parallel.
The application keeps track of synchronization progress. If the sync is interrupted due to a connection issue, it will continue from where it left off once the issue is resolved.
If the live sync encounters a hard exception, for example when the data source cannot be reached, it receives the Failed status. The exception message can be viewed by clicking the arrow in the left-most column of the table.
Failures on individual event frames do not affect the overall sync status. These items are skipped and tracked separately in the Failed Context Items section.
When the live sync detects more than 800 event frames in a single interval, that interval is automatically handed off to the excessive interval sync. This keeps the live sync running on schedule while the large batch is processed separately in parallel.
The number of threads available for excessive interval sync scales with the number of enabled data sources:
Enabled data sources | Max threads |
|---|---|
1-2 | 2 |
3-4 | 3 |
5-6 | 4 |
7 or more | 5 |
The table keeps track of a full historical overview of all excessive interval synchronizations that have occurred. For each one, the following information is made available:
Data source: The data source for which the need for an excessive interval sync was detected.
Interval: The time interval during which the need for an excessive interval sync arose.
Progress: The progress column combines the status of the synchronization with a progress bar. The following statuses are possible:
Queued: In case too many excessive interval synchronizations are already running, additional ones will be queued until resources are available to pick it up.
Failed: The excessive interval sync encountered a hard exception, resulting in some or all event frames not being processed. The interval synchronization can be retried by clicking the retry icon in the right-most column of the table (only visible for failed rows). The exception encountered can be shown by clicking the arrow in the left-most column.
Done: The excessive interval sync has completed successfully. All event frames were either processed successfully or have been added to the list of failed context items.
A historical sync resynchronizes all event frames whose last modified date falls within a specified period in the past. It can be requested on-demand by selecting a data source in the table and configuring a start and end date.
Only one historical sync can run at a time across all data sources. If a historical sync is already running when a new one is triggered, the new sync will be queued with the status Waiting until the running sync completes.
Note
The sync interval is based on the last modified date of event frames, not on their start or end date. If event frames are missing after a historical sync, check whether their last modified date in the source system falls within the configured interval. If not, either run a new historical sync covering the correct period, or modify the items in the source system, which updates their last modified date and causes live sync to pick them up automatically.
The table keeps track of a full historical overview of all historical synchronizations that have occurred. The information available in the table is the same as for the excessive interval sync, including the possibility to retry failed synchronizations.
Cleanup processes maintain the quality of synchronized context items. They check whether open items are still present and up to date in the source system, and resolve items that could not be fully processed during a previous sync.
The cleanup processes share a thread pool with excessive interval sync. The number of available threads scales with the number of enabled data sources (see Excessive Interval Sync above).
Nightly sync
The automatic nightly resync of open items appears as an active job on the diagnostics screen once it starts. Every night, the system checks for context items that have been open for more than 24 hours, meaning they have a start date but no end time, indicating they are still open.
Items open longer than 24 hours may suggest the start event was successful, but an issue occurred during the closing event, preventing the context item from closing properly.
During the nightly cleanup, context items that should be closed will be updated. If they are not closed, they will remain open until an end event is received from the source.
Note
Nightly syncs run automatically by default. If needed, you can trigger a manual cleanup from the data source settings in ConfigHub. For more details, refer to the "Data Source" section above.
AF sync (Asset Framework)
A cleanup process for context items is triggered when either an AF sync, started from asset-capable data sources in ConfigHub, or an AF import, initiated by TrendMiner admins in ContextHub, completes successfully.
When assets are added or removed from the source AF, the AF sync identifies the differences and updates the TrendMiner AF accordingly. This triggers an automatic resync of context items without an associated component, which will appear as a scheduled job in the diagnostics screen.
Note
To add an AF sync, look to the "Asset data" section above.
In case an event frame fails to process correctly, and the corresponding context item cannot be created or updated, it will be added to the table of failed context items.
The table keeps track of a full historical overview of all such failures, and for each one the following information is made available:
External ID: The ID of the corresponding event frame, in the source system.
Data source: The data source in which the event frame resides.
Sync date: The time on which the synchronization last occurred (and failed).
Error message: The error message encountered at the time of failure.
Additionally, the table offers has two additional features:
View data source response: By clicking on the arrow in the left-most column, the administrator can view the payload that was received from the data source.
Re-process event frame: By clicking the retry icon in the right-most column, the application will try to reprocess the event frame, potentially resolving the failure.
The “asset framework sync” section of diagnostics page in the data section of ConfigHub allows administrators to effectively monitor the synchronization status of asset capable data sources.
The history table keeps track of the full history of asset framework synchronizations.
The following information is available to the administrator:
Data source: The data source from which the asset structure was synchronized.
Start date: The date and time on which the synchronization was started.
End date: The date and time on which the synchronization concluded.
Status: The final status of the synchronization
For failed synchronizations, an error message can be viewed by clicking on the arrow in the left-most column of the table.