Page History
...
Available task types and their descriptions:
| Task name | Description |
|---|---|
REPORT_BROADCAST_BROADCASTTASK | Report Broadcast |
REPORT_BROADCAST_MIREPORTTASK | Report data refresh |
ETL_PROCESS_TASK | Data Transformation |
FILTER_CACHE | Cached filter refresh |
SOURCE_FILTER_REFRESH | Access filter refresh |
SOURCE_FILTER_UPDATE_REMINDER | Access filter update reminder |
ORGREF_CODE_REFRESH | Refresh org ref codes |
THIRD_PARTY_AUTORUN | Third-party connector data cache refresh |
SIGNALS_DCR_TASK | Pre-run task for Signals |
SIGNALS_ANALYSIS_TASK | Signals Analysis |
SIGNALS_CLEANUP_TASK | Post-run task for Signals |
COMPOSITE_VIEW_REFRESH | Refresh data for composite views |
| SIGNALS_CORRELATION_TASK | Detecting correlations in Signals |
Here are some recommendations:
Determine which tasks will be run repeatedly in the system.
Dedicate nodes for high frequency, resource intensive tasks, such as Signals and Data Transformations.
A node can run as many tasks in parallel, as the number of cores. However, some tasks may have a high ratio of wait time to compute time, which means it can still run tasks while others are waiting. One would have to profile the application to fine tune the total number of threads. Yellowfin recommends not more than five threads per core. This number may then be divided amongst the tasks which are configured to run on the node. For example, if the deployment runs many Cached Filter refresh tasks, a higher number should be set as its “MaxParallelTaskCount”.
The number of threads spawned for all background tasks may be controlled using the following parameters:
TaskSchedulerThreads – maximum number of threads available for all background tasks
TaskSchedulerMaxThreadQueue – number of slots in the “wait” queue. Items in the wait queue are run as soon as a thread becomes available, without having to wait for the next cycle.
Consider the following configuration:
TaskSchedulerThreads = 5
TaskSchedulerMaxThreadQueue = 15
TaskType | MaxParallelTaskCount = FILTER_CACHE | 8
ETL_PROCESS_TASK | 3
SIGNALS_ANALYSIS_TASK | 10
The node can run 5 concurrent tasks of any of the three configured types.
When 5 tasks are running, the next task will be added to the “wait” queue. As soon as a task finishes execution, the waiting task will begin running.
When 5 tasks are running and 15 tasks are waiting for a free thread, the next task will be rejected.
Ideally, (TaskSchedulerThreads + TaskSchedulerMaxThreadQueue) = Σ MaxParallelTaskCount
...