Page History
Each Yellowfin node will be configured by default to run background tasks. The cluster should be properly configured for task execution, depending on the cluster type.
...
| Code Block | ||
|---|---|---|
| ||
<servlet>
<servlet-name>SystemTaskManager</servlet-name>
<servlet-class>com.hof.servlet.SystemTaskManager</servlet-class>
<load-on-startup>8</load-on-startup>
</servlet>
|
...
| Anchor | ||||
|---|---|---|---|---|
|
DYNAMIC/REPOSITORY Cluster
...
A node may be configured to become Master but never a Slave. To do this, do not include any TaskType in the ClusterManagement servlet block in web.xml. If multiple nodes are configured to become the Master to allow for failover, then the node that gets started first becomes the Master.
A node may be configured to become a Slave but never a Master. To do this, add the following to the MIStartup servlet block in web.xml:
Code Block language xml <init-param> <param-name>DisableTaskScheduler<name>DisableTaskSchedulerPolling</param-name> <param-value>TRUE</param-value> </init-param>
Web-facing nodes may be configured to never become a Master or Slave. This way, background tasks don’t compete for resources with online processing. To do this, add the following to the MIStartup Servlet block in web.xml:
Code Block language xml <init-param> <param-name>DisableTaskScheduler</param-name> <param-value>TRUE</param-value> </init-param>
And ensure to not include any TaskType in the ClusterManagement servlet block in web.xml.
Nodes may be configured to run specific tasks. The maximum number of concurrent tasks of each type should also be configured. To do this, add the following snippet in the ClusterManagement servlet block in web.xml.
Code Block language xml <init-param> <param-name>TaskTypes</param-name> <param-value> FILTER_CACHE, ETL_PROCESS_TASK, SOURCE_FILTER_REFRESH </param-value </init-param> <init-param> <param-name>MaxParallelTaskCounts</param-name> <param-value> 1, 5, 3 </param-value> </init-param>This configures the node to run a maximum of one Filter Cache refresh task, five Data Transformation tasks and three Source Filter Refresh tasks at the same time.
If this snippet is not configured in the ClusterManagement servlet, the node will not run any background task.
...
Available task types and their descriptions:
| Task name | Description |
|---|---|
REPORT_BROADCAST_BROADCASTTASK | Report Broadcast |
REPORT_BROADCAST_MIREPORTTASK | Report data refresh |
ETL_PROCESS_TASK | Data Transformation |
FILTER_CACHE | Cached filter refresh |
SOURCE_FILTER_REFRESH | Access filter refresh |
SOURCE_FILTER_UPDATE_REMINDER | Access filter update reminder |
ORGREF_CODE_REFRESH | Refresh org ref codes |
THIRD_PARTY_AUTORUN | Third-party connector data cache refresh |
SIGNALS_DCR_TASK | Pre-run task for Signals |
SIGNALS_ANALYSIS_TASK | Signals Analysis |
SIGNALS_CLEANUP_TASK | Post-run task for Signals |
COMPOSITE_VIEW_REFRESH | Refresh data for composite views |
| SIGNALS_CORRELATION_TASK | Detecting correlations in Signals |
Here are some recommendations:
Determine which tasks will be run repeatedly in the system.
Dedicate nodes for high frequency, resource intensive tasks, such as Signals and Data Transformations.
A node can run as many tasks in parallel, as the number of cores. However, some tasks may have a high ratio of wait time to compute time, which means it can still run tasks while others are waiting. One would have to profile the application to fine tune the total number of threads. Yellowfin recommends not more than five threads per core. This number may then be divided amongst the tasks which are configured to run on the node. For example, if the deployment runs many Cached Filter refresh tasks, a higher number should be set as its “MaxParallelTaskCount”.
The number of threads spawned for all background tasks may be controlled using the following parameters:
TaskSchedulerThreads – maximum number of threads available for all background tasks
TaskSchedulerMaxThreadQueue – number of slots in the “wait” queue. Items in the wait queue are run as soon as a thread becomes available, without having to wait for the next cycle.
Consider the following configuration:
TaskSchedulerThreads = 5
TaskSchedulerMaxThreadQueue = 15
TaskType | MaxParallelTaskCount = FILTER_CACHE | 8
ETL_PROCESS_TASK | 3
SIGNALS_ANALYSIS_TASK | 10
The node can run 5 concurrent tasks of any of the three configured types.
When 5 tasks are running, the next task will be added to the “wait” queue. As soon as a task finishes execution, the waiting task will begin running.
When 5 tasks are running and 15 tasks are waiting for a free thread, the next task will be rejected.
Ideally, (TaskSchedulerThreads + TaskSchedulerMaxThreadQueue) = Σ MaxParallelTaskCount
...
A node running only SIGNALS_DCR_TASK, SIGNALS_ANALYSIS_TASK and SIGNALS_CLEANUP_TASK will be designated as a “Signals Node”. The node may run one or more of these types, but it shouldn’t run any others. A cluster may have one or more Signals Nodes. Signals nodes are specially licensed and cannot be used for processing web requests or other background tasks. A Signals node cannot become a Master.
| Code Block | ||
|---|---|---|
| ||
<init-param>
<param-name>TaskTypes</param-name>
<param-value>
SIGNALS_DCR_TASK,
SIGNALS_ANALYSIS_TASK,
SIGNALS_CLEANUP_TASK
</param-value
</init-param>
<init-param>
<param-name>MaxParallelTaskCounts</param-name>
<param-value>2,5,2
</param-value>
</init-param>
|
...
| Section | |||||||
|---|---|---|---|---|---|---|---|
|
...