Event rules for things that don't have built-in event triggers

We saw this question on one of the online Workload Scheduler forums recently.

Is anyone aware of a way to set up a rule to send an e-mail when a workstation fence is > 0?  I know there are no event specifically for this, however if anyone can think of a way to accomplish this, I would appreciate it.

Solving the problem (alerting when specific supplied event triggers do not exist) involves a few preliminary steps.

  1. Is the information provided somewhere that can be accessed easily?
  2. Is it in a reasonable format that we can use the data?
  3. Can we transform it to fit into an existing Workload Scheduler event trigger?

Remember, event rules don't check for anything. You can't just write an event to see if a fence is greater than zero. Event rules only detect when things change (like when a fence changes). Sometimes, you need a combination of event rules to accomplish the task. Also, you cannot use variables from variable tables in event rules. However event rules have their own variables, fed by the event that triggers the rule. That means you wouldn't want to write an event rule to monitor a file that's name changes every day. That's why I use two rules.

  1. The first rule sets a symbolic link to the new TWSMERGE log when a new file is created in the stdlist/logs directory. This provides a file with consistent name that the other event rule can use to detect the fence change. Now you can monitor this file for many different kinds of text that BATCHMAN produces. The following command links the new log file to the TWSMERGE_latest.log file, so you can use the same file name every day. It's a "soft link" command. No contents are copied. It's just a link. The "-f" means "freshen"; "-s" means "symbolic link"
    1. ln -f -s path_to_new_file /opt/IBM/TWA/TWS/stdlist/logs/TWSMERGE_latest.log 
  2. Monitor the consistent MERGE log for specific text to perform the alerting. A message similar to the following text appears whenever a workstation fence changes.
    13:46:15 20.11.2015|BATCHMAN:#S5392/Operator command: FENCE CPU=FTA1;13

By using the combination of event rules, you'll see a message in the Operator Messages task (and on the Workload Dashboard in DWC 9.3) whenever the "FENCE CPU" message is issued, from the logs. You must use the Master's workstation name (and of course set the path to where the stdlist files actually live) in the rule definition. If your Workload Scheduler installation is in a different directory than the default, then of course you need to adjust the commands for the real installed location. The full definitions for the event rules are posted below.

Troubleshooting

Try running the link command outside of Workload Scheduler (from the shell). First, find the latest MERGE log in /opt/IBM/TWA/TWS/stdlist/logs. Then, create a link to it.

[twsuser]$ cd /opt/IBM/TWA/TWS/stdlist/logs
[twsuser]$ ls -ltr
-rw-r--r-- 1 twsuser twsuser 1971685 Oct 25 00:00 20151024_TWSMERGE.log
-rw-r--r-- 1 twsuser twsuser      70 Oct 25 00:02 20151025_APPSRVMN.log
-rw-r--r-- 1 twsuser twsuser 1990838 Oct 26 00:00 20151025_TWSMERGE.log
-rw-r--r-- 1 twsuser twsuser     853 Oct 26 12:30 20151026_APPSRVMN.log
-rw-r--r-- 1 twsuser twsuser    2518 Oct 26 12:32 20151026_NETMAN.log
-rw-r--r-- 1 twsuser twsuser 5021116 Oct 26 14:03 20151026_TWSMERGE.log
[twsuser]$ ln -f -s /opt/IBM/TWA/TWS/stdlist/logs/20151026_TWSMERGE.log /opt/IBM/TWA/TWS/stdlist/logs/TWSMERGE_latest.log
[twsuser]$ echo $?
0

If the event rule doesn't pick up the latest logs, for whatever reason, you could schedule a job to do the linking steps (find the latest log file, then run the ln command).

MASTER_WORKSTATION#LINKMERGE
 DOCOMMAND "cd /opt/IBM/TWA/TWS/stdlist/logs/ ;ln -s -f $(ls -tr 2*TWSMERGE.log| tail -1) TWSMERGE_latest.log"
 STREAMLOGON twsuser
 DESCRIPTION "Link the latest MERGE log to a common name"
 TASKTYPE UNIX
 RECOVERY STOP

Event rule to link TWSMERGE_latest.log

Below is the event rule that makes a symbolic link to the newest TWSMERGE log file. You need to use the Master's workstation name for the Workstation attribute, and of course set the path to where the stdlist files actually live for the FileName filter.

<?xml version="1.0"?>
<eventRuleSet     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules"
        xsi:schemaLocation="http://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules http://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules/EventRules.xsd">
    <eventRule name="LINK_MERGE" ruleType="filter" isDraft="no">
        <eventCondition name="fileCrtEvt1" eventProvider="FileMonitor" eventType="FileCreated">
            <scope>
                /OPT/IBM/TWA/TWS/STDLIST/LOGS/20*TWSMERGE.LOG ON MDM0
            </scope>
            <filteringPredicate>
                <attributeFilter name="FileName" operator="eq">
                    <value>/opt/IBM/TWA/TWS/stdlist/logs/20*TWSMERGE.log</value>
                </attributeFilter>
                <attributeFilter name="SampleInterval" operator="eq">
                    <value>60</value>
                </attributeFilter>
                <attributeFilter name="Workstation" operator="eq">
                    <value>MASTER_WORKSTATION</value>
                </attributeFilter>
            </filteringPredicate>
        </eventCondition>
        <action actionProvider="GenericActionPlugin" actionType="RunCommand" responseType="onDetection">
            <scope>
                LN -F -S %{FILECRTEVT1.FILENAME} /OPT/IBM/TWA/TWS/STDLIST/LOGS/TWSMERGE_LATEST.LOG
            </scope>
            <parameter name="Command">
                <value>ln -f -s %{fileCrtEvt1.FileName} /opt/IBM/TWA/TWS/stdlist/logs/TWSMERGE_latest.log </value>
            </parameter>
        </action>
        <action actionProvider="MessageLogger" actionType="MSGLOG" responseType="onDetection">
            <scope>
                OBJECT=PLAN MESSAGE=NEW MERGE %{FILECRTEVT1.FILENAME}
            </scope>
            <parameter name="Message">
                <value>New MERGE %{fileCrtEvt1.FileName}</value>
            </parameter>
            <parameter name="Severity">
                <value>Info</value>
            </parameter>
            <parameter name="ObjectKey">
                <value>Plan</value>
            </parameter>
        </action>
    </eventRule>
</eventRuleSet>

 

Event rule to watch the MERGE log for CPU FENCE changes

Below is the event rule that watches for CPU FENCE changes. You need to use the Master's workstation name for the Workstation attribute, and of course set the path to where the stdlist files actually live for the FileName filter.

<?xml version="1.0"?>
<eventRuleSet     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules"
        xsi:schemaLocation="http://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules http://www.ibm.com/xmlns/prod/tws/1.0/event-management/rules/EventRules.xsd">
    <eventRule name="FENCE_CHANGE" ruleType="filter" isDraft="no">
        <eventCondition name="logMessWritEvt2" eventProvider="FileMonitor" eventType="LogMessageWritten">
            <scope>
                FENCE CPU IN /OPT/IBM/TWA/TWS/STDLIST/LOGS/TWSMERGE_LATEST.LOG ON MDM0
            </scope>
            <filteringPredicate>
                <attributeFilter name="FileName" operator="eq">
                    <value>/opt/IBM/TWA/TWS/stdlist/logs/TWSMERGE_latest.log</value>
                </attributeFilter>
                <attributeFilter name="MatchExpression" operator="eq">
                    <value>FENCE CPU</value>
                </attributeFilter>
                <attributeFilter name="SampleInterval" operator="eq">
                    <value>60</value>
                </attributeFilter>
                <attributeFilter name="Workstation" operator="eq">
                    <value>MASTER_WORKSTATION</value>
                </attributeFilter>
            </filteringPredicate>
        </eventCondition>
        <action actionProvider="MessageLogger" actionType="MSGLOG" responseType="onDetection">
            <scope>
                OBJECT=WORKSTATION MESSAGE=%{LOGMESSWRITEVT2.LINETEXT}
            </scope>
            <parameter name="Message">
                <value>%{logMessWritEvt2.Linetext}</value>
            </parameter>
            <parameter name="Severity">
                <value>Info</value>
            </parameter>
            <parameter name="ObjectKey">
                <value>Workstation</value>
            </parameter>
        </action>
    </eventRule>
</eventRuleSet>
Tags: