Nagios Core - HP OML

Preface

This document will describe integration of Nagios Core in to HP Operation Manager for Linux – HP OML (as well-known with old name as HP Open View – HP OVO) environment.

The goal of this documents is not to go too deep in to HP OML, as I do not see the reason to supply official documentation of this product.

On another hand, I think that light overview of this products can help us to understand how they are working.

Way

There are several reason way someone would need to do integration like this.
- There is some business need to run the monitoring on HP OML, but it is too expensive for the company.
- You are out of licenses on HP OML, but you need to do monitoring for additional infrastructure.
- Historically you are using HP OML for integration of events, but you are looking for alternative. In this way you can start the migration without service interruption.
- You are looking for flexibility of Nagios Core but you still need to integrate it with HP OML.
- …

Before we'll start

We can say that Nagios Core as well HP OML are mainly monitoring tools (even they can do more).

Before we'll benefit from integration of mentioned tools together, we’ll need to get basic understanding how each of this tool is working.

Nagios Core host/service

The configuration of Nations Core is spited across several configuration “Objects”, where the main monitoring object is “host” object.
Host object macro is used at “service” object definition. In this way we are creating a relation between “host” and “service” objects.
Inside of Nations Core Object definitions it’s possible to use custom variables (user defined macros) that are providing us higher definition flexibility.

Example:

The simplest example is probably monitoring of standard server with web service:
- Host – in this case it will include information about the server. As well it will define up/down monitoring for the server.
- Service – it will define monitoring of:
————– HDD
————– CPU
————– RAM
————– SWAP
————– Web reachability
————– Log files
————– SNMP traps
————– ….

HP OML/OVO monitoring

HP OML is using complex configuration, as it's providing several functionalities (not only monitoring). According to this the configuration of node is probably not flexible like at Nagios Core. You will need to define the OS, HW, IP, name, if it is running HP OML/OVO agent or not, … .

HP OML monitoring definition is based on “Policy” (as well-known as HP OVO templates) definitions. We can compare it with definition of “service” objects at Nagios Core.

Policy Type Agent Type Editor
ConfigFile configfile midas:text
Event_Correlation ec midas:ec
Logfile_Entry le midas:logfile
Measurement_Threshold monitor midas:advmonitor
Node_Info configsettings midas:text
Open_Message_Interface msgi midas:opcmsg
Scheduled_Task sched midas:schedule
Service_Auto_Discovery svcdisc midas:svcdisc
Service_Process_Monitoring monitor midas:svcprocmon
SiteScope sitescope midas:text
SNMP_Interceptor trapi midas:snmp
Subagent subagent midas:text
Windows_Event_Log le midas:winevtlog
Windows_Management_Interface wbemi midas:wmi

HP OML/OVO Customization

It’s possible to say that most common way of HP OML monitoring customization, is to use one of this HP OML Policy's:

Open_Message_Interface

With help of this HP OML Policy it's possible to use monitoring scripts that will call shall command “opcmsg” (with parameters). In this way it's possible to forward alarm from script in to HP OML. On another hand sometime it's really difficult to do investigation of this monitoring, as from HP OML point of view it’s impossible to identify script that has sent the alarm.

Syntax.

# opcmsg -help

opcmsg - submit a message for HP OpenView Operations

Usage : opcmsg [-help] [-id] [severity=normal|warning|minor|major|critical]
               application=<application> object=<object> msg_text=<text>
               [msg_grp=<message group>] [node=<node>] [service_id=<svcid>]
               [-option <var>=<value>]*

We can compare HP OML “Open_Message_Interface” with Nagios Core “nagios.cmd”.

At Nagios Core we can create passive monitoring for “service”/”host” with help of messages sent to “nagios.cmd” :
“PROCESS_SERVICE_CHECK_RESULT “
“PROCESS_HOST_CHECK_RESULT”

Logfile_Entry

This type of HP OML Policy is used for monitoring of any text based files (for interesting strings). In this case we need to specify “Match Text”, it's regular expression like way to search of interesting string.

So far I know Nagios Core is not providing this kind of monitoring out of the box.

To provide same service at Nagios Core we can use for example LogSurfer and passive “service” monitoring, with help of messages sent to “nagios.cmd” interface.

Scheduled_Task

In this case it's possible to schedule monitoring based on customized scripts. That will periodically run to monitor you infrastructure. Monitoring result is provided to HP OML with exit code of the monitoring script.

When you are looking for similar way of monitoring as Nagios Core is delivering out of the box. You'll like this HP OML/OVO Policy.

SNMP_Interceptor

This HP OML Policy is able to process received SNMP trap to get some information out of this SNMP trap and create meaning full alarm.

HP OVO as well using “trapd.conf” file, to define how to process interesting SNMP traps. It was similar way like it's provided at SNMPTT tool.

On another hand, in some case, it's not the best way to manage SNMP traps, as the same Enterprise OID can deliver each time different set of conditional OID’s.

At Nagios Core we can use:

- Net-Snmp with SnmpTT forwarding interesting SNMP traps like passive monitoring to Nagios Core with help of “nagios.cmd”.
- Net-Snmp with custom script that will send message to Nagios Core with help of “nagios.cmd” (when SNMPtrap with same Enterprise OID is providing each time different conditional OID’s).

Windows_Event_Log

When you are planning to monitor MS Windows servers it's interesting to monitor as well “Event Viewer” messages for interesting alarms. According to this HP OML is providing this Policy to monitor MS Windows “Event Viewer” for interesting messages. The configuration from user point of view is similar to “Logfile_Entry” Policy at HP OML.

At Nagios Core we have the possibility:

- To use check scripts for this kind of monitoring.
- To configure MS Windows to send SNMP trap to our SNMP trap receiver (“evntwin” - as well on this web site is document describing this integration).
- To forward events from MS Windows server in to standard Syslog server.
- To use WMIC for Linux to run WQL query to search for interesting events. (“Win32_NTLogEvent”, “WMI”, “WQL” - as well on this web site is document describing this integration)
- ….

Windows_Management_Interface

This HP OML Policy is providing the possibility to work with WMI interface of your MS Windows server.

At Nagios Core it is possible:

- To use check scripts for this kind of monitoring.
- To use WMIC for Linux to run WQL query to search for interesting events. (“Win32_NTLogEvent”, “WMI”, “WQL” - as well on this web site is document describing this integration)
- To use SNMP as most COM values are accessible with standard SNMP query to your MS Windows server. The benefit of using SNMP query is that it need less resources (look for “SNMP Service” on your MS Windows server)
- ….


Nagios Core Event Correlation

Nagios Core is providing event correlation based on:
- Hostdependency = Host <–to–> Host
- Hostescalation = Host and reoccurrence of the issues
- Servicedependency = (Host+Service)<–to–>(Host+Service)
- Serviceescalation = (Host+Service) reoccurrence of the issues
- reoccurrence of the issues:
- - It's possible to configure Nagios Core to send only alarm in the case of status change
- - It's possible to configure Nagios Core to send alarm after some time period
- - …

In the case that you would like to have correlation of events in Nagios Core based on physical or logical topology of your infrastructure it's required to configure this relations manually with help of Nagios Core configuration Objects.

In some cases it's possible to use integration with another tools, to configure relation between the Nagios Core configuration Objects (“NetDisco” for network discovery, “CMDB” with documentation of relations), in most cases it's required to write scripts that will help to automatize it.

HP OML Event Correlation

In this point we can think about HP OML like about Event broker. It means that there are several ways how to send alarms in to HP OML, each message will include several attributes like:

Application

Variable that is required at HP OML. It shall be used to identify the related application.

Comparing it to Nagios Core, we can think about this like the name of the “service” object name.

Message Group

Variable that is used to define who can see this alarm at HP OML

Comparing it to Nagios Core, it's possible to say that this functionality in multi-tenant environment is providing “contact” and “contactgroup” object at Nagios Core

Node

Variable that is used to define name of the node that has sent particular alarm.

Comparing it to Nagios Core, it's possible to say that this parameter is describing “host” object.

Object

In some cases it's used to identify additional information that can help at processing of the alarm.

Severity

Definition of the event priority.

HP OML priorities:
- Unknown
- Normal
- Warning
- Minor
- Major
- Critical

Nagios Core is not using same priorities for alarms.

On another hand for similar purpose we do have status of the “host”/”servide” objects.
host:
d = DOWN state
u = UNREACHABLE state
r = recoveries (OK state)
f = flapping

Service:
w = WARNING state
u = UNKNOWN state
c = CRITICAL state
r = recoveries (OK state)
f = flapping

Message Text

Variable that is used to provide meaning full message, for support agents that will handle the alarm.

Comparing it to Nagios Core, it's possible to think about it like the message that each check command need to provide on “STDOUT” after it has finished. This will be taken as a comment to the Nagios Core “host”/”service”.

Message Key

This parameter will be created based on the HP OML Policy. Based on this it's possible to identify related alarms for correlation of events. The best practice is to take some information from alarm that was delivered to the HP OML to create meaningful string.

In the case that we'll not specify this parameter in our HP OML Policy Condition, rest of mentioned parameters will be taken for correlation. It mean that Message Text will be taken as well for correlation purposes. In this case the correlation of events will probably not work in the same way as we'd like.

Nagios Core “hostgroups” - HP OML “Node Groups”

Nagios Core as well HP OML are grouping monitored objects. Mentioned grouping is providing possibility to assign configuration to the group of monitored “hosts”/“nodes”.

- Nagios Core it's possible to assign one “service” to “hostgroup” so that all “hosts” from mentioned “hostgroup” will be monitored in same way.
- HP OML it's possible to assign “Policy” or “Policy Group” to “Node Group”. In this way all nodes from mentioned “Node Group” will be able to use monitoring defined at “Policy” or “Policy Group”

HP OML (OVO) Agent

Before we can start with the integration we'll need to install HP OML(as well-known as HP OVO) Agent on our Nagios Core Server. This will provide us possibility to forward our alarms to HP OML.

In general HP OML agent:
- Is including several components used for several different type of monitoring.
- Is providing the possibility to run commands initialized at HP OML to run at monitored node.
- It is possible to extend him with additional plugins (for monitoring of Databases, SAP, performance, …).
- Is providing interface for file transfer.
- …

Standard communication between HP OML agent and HP OML server is using TCP/383 port. Protocol itself is SOAP encrypted in HTTPS. According to this mentioned communication can pass through PROXY.

Installation of HP OML Agent

Before you'll start test this mandatory points as you'll be not able to do proper installation of HP OML Agent.

- DNS resolving need to work
- - At monitored node (Nagios Core server) you need to be able to resolve FQDN of HP OML server
- - At HP OML server you need to be able to resolve FQDN of the monitored node (Nagios Core server)

- ICMP need to work
- - At monitored node (Nagios Core server) you need to be able to ping FQDN of HP OML server
- - At HP OML server you need to be able to ping FQDN of the monitored node (Nagios Core server)

- TCP/383
- - At monitored node (Nagios Core server) you need to be able to telnet to FQDN of HP OML server on port 383
- - At HP OML server you need to be able to telnet FQDN of the monitored node (Nagios Core server) on port 383 (as it's not listening on this port, use tcpdump)

[root@NagiosCore ~] # mkdir /usr/src/HPOA                                   # I like the old school UX/Linux way to store installation scripts
[root@NagiosCore ~] # cd /usr/src/HPOA/                                     # Copy here your HP OML/OVO agent source
[root@NagiosCore HPOA] # ls
HPOA_11.13_Linux_64-v2.tar.gz                                               # My Nagios Core is on CentOS => I need to use HP OML Agent for Linux
[root@NagiosCore HPOA] # tar -vxzf HPOA_11.13_Linux_64-v2.tar.gz            # Unpack the installation file

[root@NagiosCore HPOA] # ls
HPOA_11
HPOA_11.13_Linux_64-v2.tar.gz

[root@NagiosCore HPOA] # cd HPOA_11                                         # Here we do have the installation source of HP OML Agent

[root@NagiosCore HPOA_11] # ll
total 80
drwxrwxr-x 7 root sys   4096 Jan 22  2014 agents
-rw-r--r-- 1 root sys     96 Jan 15  2013 ALL_ITO_MGR.example1
-rw-r--r-- 1 root sys     44 Jan 15  2013 ALL_ITO_MGR.example2
-rwxr-xr-x 1 root sys    932 Jan 24  2013 CustomA1installPost
-rwxr-xr-x 1 root sys    931 Jan 24  2013 CustomA1installPre
drwxr-xr-x 3 root sys   4096 Jan 22  2013 dsi
drwxr-xr-x 2 root sys   4096 Nov  9  2012 env_scripts
-rw-r--r-- 1 root sys   1589 Jun 18  2013 install-error-codes.txt
-rw-r--r-- 1 root sys    773 Nov 27  2012 NameSpace.dat
-rwxr-xr-x 1 root root  1332 Jan 22  2014 QCIM1A171156.sh
-rwxr-xr-x 1 root root 38512 Jan 22  2014 root_task                         # Installation scrip

[root@NagiosCore HPOA_11] # ./root_task <FQDN_of_your_HP_OML_Server>        # Install the HP OML Agent
.
.
.
You will see very little output to the screen
If you want to follow the install, open a new shell and
tail -f /var/tmp/HPOA_11_Inst.log                                           # Here is possible to see current status of installation
or for errors: tail -f /var/tmp/HPOA_11_script_error.log
.
.

Check after installation of HP OML Agent

On Nagios Core server with running HP OML Agent:

[root@NagiosCore HPOA_11]# /opt/OV/bin/ovc -status                               # Status check of the HP OML Agent
coda        OV Performance Core                 COREXT       (8509)   Running
opcacta     OVO Action Agent                    AGENT,EA     (8485)   Running
opcmsga     OVO Message Agent                   AGENT,EA     (8497)   Running
ovbbccb     OV Communication Broker             CORE         (7216)   Running
ovcd        OV Control                          CORE         (7097)   Running
ovconfd     OV Config and Deploy                COREXT       (8466)   Running

[root@NagiosCore HPOA_11]# /opt/OV/bin/opctemplate                               # List of Installed Policy
       * List installed policies for host 'localhost'.
   No policies are installed on host 'localhost'.

[root@NagiosCore HPOA_11]# /opt/OV/bin/ovcconfget                                # Configuration of HP OML Agent
.
.
.

On HP OML Server that is Manage our Agent:

[root@HP_OML ~]# opccsa -list_pending_cr -format rhimpto                         # List of pending certificate requests on HP OML Server
11111111-1111-1111-1111-111111111111  NagiosCore.test.com  10.0.0.2  Linux 2.6.32, CPU: x64  01/01/14 19:38:09  22222222-2222-2222-2222-222222222222

As well you can find all pending certificate requests on you HP OML server with using Web interface:

OMU→Deployment→Pending Certificate Requests

Create node at HP OML

At WEB Interface:

OMU->Edit->Add Node->
select
"Node Type"                                                # According your OS
"Host Name"                                                # FQDN of the monitored node
"IP Address"                                               # IP of the monitored node
"Parent Group"                                             # Layout Group the node belong to
.
.                                                          # In the case that you use at your HP OML some additional parameters for your managed nodes please configure them
.
->Submit

At CLI:

[root@HP_OML ~]# /opt/OV/bin/OpC/utils/opcnode -add_node node_name=nagioscore.test.com net_type="NETWORK_IP" mach_type=MACH_BBC_LX26RPM_X64 node_label=nagioscore comm_type=COMM_BBC
No Node group was specified for assignment. Node 'nagioscore.test.com' will be moved to Holding Area.
Operation successfully completed.

[root@HP_OML ~]# /opt/OV/bin/OpC/opcsw nagioscore.test.com                           # Set software status in HP OML database

This will create a node that will be pleased in to “Holding Area”, as it's missing “Node Group” definition.

At this point I need to leave it at you, to assign the “Node” to correct “Node Group” at your HP OML. As it's dependent on the way how you are manage your “Nodes” at HP OML.

You can extend the command with parameter “group_name=<Layout Group>” to assign the node to correct node group.

Check new node at HP OML

At WEB Interface:

OMU->Browse->All Nodes                                                                   # List of all Nodes

or

OMU->Browse->Node Bank->HoldingArea                                                      # In our case Node was not assigned to any Node Group

or

OMU->Browse->Node Bank-><YOUR_NODE_GROUP>                                                # In the case that you have assigned the Node to an Node Group

or

OMU->Browse->All Layout Groups-><YOUR_LAYOUT_GROUP>                                      # To see your Node at Layout Group

or

OMU->Find->Locate->
Set:
    "Locate" = "Node"
    "where name" = "contains" = "<Name_of_Your_Node>                                     # To search for your node
->Locate

At CLI:

[root@HP_OML ~]# opcnode -list_nodes                                                     # This will provide the whole list of known nodes

or
                                                                                         # This will create a CSV output of known nodes
[root@HP_OML ~]# echo "Name;Label;IP-Address;Machine Type" ; opcnode -list_nodes |
awk '{ \
if ($1 == "Name"){printf "%s;",$NF} \
if ($1 == "Label"){printf "%s;",$NF} \
if ($1 == "IP-Address"){printf "%s;",$NF} \
if (($1 == "Machine") && ($2 == "Type")) {printf "%s;\n",$NF} \
}'

Grant Node Certificate at HP OML

List of Certification Requests:

[root@HP_OML ~]# opccsa -list_pending_cr -format rhimpto                                                # Pending certificate requests at HP OML Server
11111111-1111-1111-1111-111111111111  NagiosCore.test.com  10.0.0.2  Linux 2.6.32, CPU: x64  01/01/14 19:38:09  22222222-2222-2222-2222-222222222222

Assign certificate to Node

[root@HP_OML ~]# opccsa -map_node 11111111-1111-1111-1111-111111111111=nagioscore.test.com              # Map Certification Request to Node

or                                                                                                      # OR

[root@HP_OML ~]# opcnode -chg_id node_name=nagioscore.test.com id=22222222-2222-2222-2222-222222222222  # Map Certificate ID to Node

Grant certificate to Node

[root@HP_OML ~]# opccsa -grant 11111111-1111-1111-1111-111111111111                                      # Grant Certificate based on the Request ID

or                                                                                                       # OR

[root@HP_OML ~]# ovcm -grant 11111111-1111-1111-1111-111111111111 -host nagioscore.test.com              # Grant Certificate based on the Request ID to Node

Test the connection to HP OML Agent

[root@HP_OML ~]# opcragt nagioscore.test.com                                                             # Connect to Agent from HP OML Server
Node nagioscore.test.com:                                                                                # Running HP OML Agent components
HPOM Managed Node status :
-------------------------
OV Performance Core       coda        (8509) is running
OV Communication Broker   ovbbccb     (7216) is running
OV Control                ovcd        (7097) is running
OV Config and Deploy      ovconfd     (8466) is running
Subagent EA:
Action Agent              opcacta     (8485) is running
Message Agent             opcmsga     (8497) is running
Done.

[root@HP_OML ~]# ovdeploy -cmd "hostname ; uname -a" -host nagioscore.test.com                           # Run shell command on remote host
NagiosCore
Linux NagiosCore 2.6.32-431.20.3.el6.x86_64 #1 SMP Thu Jan 1 21:14:45 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Short summary

Until now we've compared Nagios Core to HP OML (as well-known as HP OVO). To be honest, I think most of you are pretty familiar with most of described points. I assume that you have deeper understanding of described points, as it was only quick over view of most important points (as we don’t have real space to go deeper, as well I don’t what to substitute official product documentation).

Until now:

- The differences between Nagios Core and HP OML.
- How to install HP OML Agent on Nagios Core server.
- How to enable the communication between Nagios Core and HP OML
- (How to install Nagios Core is described in another document at this web page)

Now it will be interesting

As we are going to:
- Configure HP OML Policy (at HP OVO it’s called template) that will provide us the possibility to forward events from Nagios Core in to HP OML
- Configure Nagios Core to forward events in to HP OML

HP OML Policy that we’ll use?

Scheduled_Task

We can use scheduled template (“Scheduled_Task”) to check Nagios Core for particular “host” or “service” status. On another hand, there are better ways how we can do the integration.

As:

- This can be done in several way:
- - Write script that will be checking “/opt/nagios- <VERSION> /var/status.dat
- - In case that we use “NDOUtils”. Write script that will use MySql query to get “host” or “service” status.
- - ..

- To be honest this is not the right way as:
- - It's not the most effective way.
- - Time frame between detecting of issue at Nagios Core and delivering alarm in to HP OML will be longer as it can be with using another method.

Logfile_Entry

We can use monitoring of log file where Nagios Core will write each event.

To be honest, this is as well not the best way to solve our case. Even it's better as to use Scheduled HP OML Policy, as it will not require to use additional script to create alarm at HP OML.

On another hand:
- It's still not the most effective way.
- Time frame between detecting of the issue at Nagios Core site and delivering alarm in to HP OML will be longer as it can be with using another method.

SNMP_Interceptor

It's possible to send SNMP trap from Nagios Core server to HP OML server, that will include several conditional OID’s with information related to our alarm.

The idea is really nice, as we can forward our alarms from Nagos Core server to HP OML direct after the issue was detected.

As well this is probably “cheapest” way how to deliver the alarm to HP OML server as:
- We do not need to use HP OML Agent, this will safe us licenses.
- We are using UDP instead of TCP, this has less overhead.

Just we need to take in to account as well:
- It's UDP and not TCP. According to this Nagios Core server don't know if alarm has arrived (ok there are types of SNMP traps that can provide this information)
- We’ll need to get our own Enterprise OID for our service to prevent interruption with another vendors.
- …

Open_Message_Interface

I do personally prefer to use standard HP OML Policy Message for forwarding of Events from Nagios Core in to HP OML.

It's not the “cheapest” way, as we need to use HP OML Agent, as well it's using TCP instead of UDP protocol.

On another hand:
- In basic setup we'll not need to use additional script to parse log files.
- It's possible to call HP OML Agent directly after detecting of issue from Nagios Core.
- It's possible to use correlation of events at HP OML Agent site, before alarm will be sent to HP OML. This will do offload for HP OML Server.
- It's using TCP instead of UDP protocol.
- …

Any way if you feel more comfortable to use another way to do this kind of integration you can test it. (Probably you can write to me about the results, as it's any time interesting to discover something new)

HP OML Configuration.

At HP OML it’s important to create for each type of monitoring a Policy. This will enable us to use particular monitoring feature and provide us the possibility to filter only the interesting alarms. According to this in our case we will need to configure HP OML Policy that will match all our interesting alarms that we would like to transfer from Nagios Core in to HP OML Server.

Create HP OML Policy Group:

Before we'll start to create the HP OML “Policy” it's best practice to create HP OML “Policy Group”. In this way we can group all our Policy’s in one group, so that we can assign the “Policy Group” to “Node” or to the “Node Group”. In large environments it's providing us higher visibility of the relations between monitoring configuration and monitored Nodes.

Use Web GUI:

OMU->Edit->Add Policy Group                                                  # We'll create a Policy Group that will include all our Policy’s

Name: NagiosCore                                                             # Name of Policy Group
Description: This PG is used for NagiosCore->OML Integration                 # Optional, Description of the Policy Group
Information: This PG is used for NagiosCore->OML Integration                 # Optional, Additional information about Policy Group
Parent Group:                                                                # optional, Can be used for nesting of the Policy Groups in to another PG

->Save                                                                       # Submit the change

Now we can see our new Policy group at:

OMU->Browse->All Policy Group

or

OMU->Find->Locate
   Locate: Policy Group
   where name: contains: NagiosCore
->Locate

Create HP OML (message) Policy:

At this point we'll create a HP OML “Policy” that will be used for matching of our interesting alarms sent from Nagios Core to HP OML Server.

At this point it depend on your decision how you would like to filter events.

There are many possibilities, but I don't have space to describe all of them. Instead of this I've tried to describe configuration possibilities at beginning of this document. In this way I hope that I've provided you high level overview of the possibilities so that you'll know what to search for

OMU->Edit->Add Policy Group->Open_Message_Interface                        # We'll create a Policy
                                                                           # It has several configuration tabs
->Properties
    Name:           "NagiosCore_opcmsg"                                    # Name of the Policy
    Type:           "opcmsg"                                               # Information about Policy type
    Version:        "auto-increment on save"                               # Default, it will any time increment the version after update
    Latest Version: "n/a"                                                  # Default, as this is new Policy
    Description:    "NagiosCore_opcmsg"                                    # Description, should contain information about this policy regarding usage, functions, etc.
    Information:                                                           # Information, should contain information about this policy regarding usage, functions, etc.
    Parent Group:   "NagiosCore"                                           # Name of PG that the Policy belong to
------------------------------------------------------------------------------------------------------------------
->Message Defaults
    Message Group:                                                         # Specification of message group, as we can have several different of them. We’ll leave it empty.
                                                                           # default for attribute "Message Group" in set part of condition
    Service ID:                                                            # Default service id to use in generated messages unless overridden in a condition
    Service hosted on:                                                     # Name of the host the service is hosted on

    ->Message Defaults->Instructions:                                      # It's possible to include some text for support team in to each alarm
        Type:                                                              # Don't use instruction text / Instruction text / Instruction Text Interface

    ->Message Defaults->Correlation                                        # It's providing us the possibility to create correlation parameters for all condition
        Message Key:                                                       # Correlation message Key

    ->Message Defaults->Correlation->Duplicate Message Suppression:        # Specification of correlation
        Type: "Suppress Identical Input Events"                            # Suppress Message Matched Condition / Suppress Identical Input Events / Suppress Identical Output Messages
        Suppression Settings:                                              # Set time frame and reoccurrence of alarm
            Suppression Time Interval: "15 min"                            # This will suppress identical alarms for 15 min

    ->Message Defaults->Advanced                                           # This is related to Message stream interface we do not need to use it in our case
------------------------------------------------------------------------------------------------------------------
->Conditions                                                               # Rule that will match our alarm. I will describe it deeper later on
    ->Conditions->Add Condition (icon)
------------------------------------------------------------------------------------------------------------------
->Options
    Local Logging                                                          # Definition of processing of the messages
       - Log Messages matching Message Conditions
       - Log Messages matching Suppress Conditions
       - Log unmatched Messages

    Unmatched Messages                                                     # Definition of precessing unmatched messages
       - Forward to Server
       - On Server Log Only
------------------------------------------------------------------------------------------------------------------
->Save                                                                     # To save our HP OML Policy

At this point we do have configured the HP OML Policy with basic configuration related to event handling. This will be inherently taken over in to configurations Conditions of this HP OML Policy.

Create HP OML (message) Policy Condition:

Now we are going to configure the most important part of the HP OML in our case. Conditions are rules that are grouped in HP OML “Policy”. Each HP OML “Policy” can include several ordered “Conditions”. It's important to order all HP OML Policy “Conditions” in right way, as at this point we can use suppression conditions that will filter not interesting alarms.

In general it's possible to say that the logic is: “First come first served.”

At Web GUI follow this steps:

OMU->Browse->All Policy Groups->NagiosCore->NagiosCore_opcmsg                           # Open Policy to edit the Conditions

or

OMU->Find->Locate:
Locate: "Policy"
where name: Contains: "NagiosCore_opcmsg"
->Locate                                                                                # Open Policy to edit the Conditions

------------------------------------------------------------------------------------------------------------------
At top of the web page you shell see the name of the HP OML Policy and near to it is an “cogged wheel” like in the KDE logo. Please click on it with mouse.
------------------------------------------------------------------------------------------------------------------
->Edit                                                                                  # To Edit the HP OML Policy
->Conditions                                                                            # To Edit Conditions
->add Condition (icon "like in Word for new document")                                  # To add a Condition

------------------------------------------------------------------------------------------------------------------
Type: "+ Message On Matched Conditoin"                                                  # How to handle matched allarm
                                                                                        # - Suppress Matched Condition / = Suppress Unmatched Condition / + Message On Matched Condition
Description "NagiosCore_opcmsg_Critical"                                                # Name of the condition

------------------------------------------------------------------------------------------------------------------
->Condition                                                                             # Definition of the Condition
    Severity: "Critical"                                                                # Is definition of the alarm priority
                                                                                        # Normal / Warning / Minor / Major / Critical
    Node:                                                                               # Name of the related node. We will not use it
    Application: "NagiosCore_<*>"                                                       # Name of the Affected Application.
                                                                                        # In our case Service we will use prefix NagiosCore to each alarm.
    Message Group:                                                                      # Message Group it depend on your setup of HP OML how you will use it
    Object:                                                                             # We'll use it to identify the "host" at Nagios Core
    Match Text:                                                                         # It's possible to parse the message text.

->Message                                                                               # Definition of the message that will be present at HP OML as alarm
    Severity: "Unchanged"                                                               # In general it depend on your internal process haw
    Node                                                                                # you are handling alarms at HP OML.
    Application                                                                         # In our case I do not see any reason to modify this values
    Message Group                                                                       # as we will send the alarm in the format
    Object                                                                              # that we would like that the support agent will see.
    Message Text
    Service ID
    Service hosted on
    Message type

->Actions                                                                               # Definition of the Action that shall start after Alarm will arrive to HP OML
    On Server Log Only                                                                  # Acknowledge the alarm and send him in to history browser.
    Automatic                                                                           # Automatic action that can start shell script (to fix issue)
      Command                                                                           # Shell Command to start
      Node                                                                              # Where to start it
      Acknowledge if successful                                                         # Auto-acknowledge message if action is successfully executed
      Add output as annotation                                                          # Collect output of action in annotation

         Send message immediately                                                       # Handle the event immediately after received
         Send message when action is finished                                           # After Automatic action,it can decide if it's needed to create alarm
            If the action completes successfully
            If the action fails

    Operator initiated                                                                  # Commands That can start support Agent with help of ITO/OVO/OML Browser
      Command                                                                           # Shell Command to start
      Node                                                                              # Where to start it
      Acknowledge if successful                                                         # Auto-acknowledge message if action is successfully executed
      Add output as annotation                                                          # Collect output of action in annotation

    Forward To Trouble Ticket                                                           # Use TT Gate to forward alarm in to another tools
      Acknowledge                                                                       # Auto-acknowledge message if action is successfully executed

    Notification                                                                        # Use Notification interface to notify right team

->Custom Attributes                                                                     # It depend on your setup of HP OML if you need to use it

->Correlation                                                                           # Is used for Event correlation
    Message Key:"<$MGMTSV_KNOWN_MSG_NODE_NAME>:<$MSG_NODE_NAME>:<$MSG_SEV>:<$MSG_GRP>:<$MSG_OBJECT>:<$MSG_APPL>:Critical"      # Message Key that will be assigned to the Alarm handled in this Condition
    Pattern Matching                                                                    # In some cases like UP/DOWN monitoring it is possible to use acknowledging of opposite alarm based on they MSGKEY.
       Acknowledge Messages Matching This Message Key Pattern:                          # MSGKEY of alarm to acknowledge
    Pattern Matching
       Field Separators
       Case Sensitive
    Duplicate Message Suppression                                                       # Correlation of duplicate alarms
       Type: "Suppress Identical Input Events"                                          # Suppress Message Matched Condition / Suppress Identical Input Events / Suppress Identical Output Messages
       Suppression Settings                                                             # Set time frame and reoccurrence of alarm
          Suppression Time Interval: "15 min"                                           # This will suppress identical alarms for 15 min (in our case based on MSGKEY)

    Message Defaults->Instructions:                                                     # It is possible to include some text for support team in to each alarm
       Type:                                                                            # Do not use instruction text / Instruction text / Instruction Text Interface
    Advanced                                                                            # It is possible to use MSI to correlate/modify messages before HP OML will process them
                                                                                        # (We are not going to use it)
       Pattern Matching                                                                 # Filter of events based on Message text
          Case Sensitive
     Message Stream Interface                                                           # Decide if we will use MSI of HP OML Agent or HP OML Server
          Agent MSI:
          Server MSI:
->Save                                                                                  # To save the Condition

Now you can copy this condition and change the “Critical” to Normal / Warning / Minor / Major so that you will get a one condition for each priority.

Description "NagiosCore_opcmsg_Critical"

->Condition
    Severity: "Critical"

->Correlation
    Message Key:"<$MGMTSV_KNOWN_MSG_NODE_NAME>:<$MSG_NODE_NAME>:<$MSG_SEV>:<$MSG_GRP>:<$MSG_OBJECT>:<$MSG_APPL>:Critical"

Assign Policy group on NagioCore Node

We have already created a HP OML “Policy Group” including our “Policy” with all “Conditions”. At this moment it's required to push this configuration to HP OML Agent that is running on our NagiosCore server. After it will be done we can start to configure our Nagios Core to use “opcmsg” command for alarm forwarding in to HP OML.

At Web GUI

OMU->Find->Locate                                                                        # Find Nagios Core Server at HP OML
   Locate: Node
   where name: Contains: "NagiosCore.test.com"
->Locate

Near to the node you shall see “cogged wheel” like in the KDE logo click on it->         # We've assigned the Policy Group to the NagiosCore server
->Assign node to Policy/Policy Group
    Locate : Policy Group
    Name : Contains : "NagiosCore"
    ->Filter->
    ->Mark the Policy Group called "NagiosCore"->
->OK

Near to the node you shall see “cogged wheel” like in the KDE logo click on it->         # We'll push configuration to HP OML Agent
->Deploy Configuration
    ->Mark options that you need
->OK

OR

At CLI

[root@HP_OML ~]# /opt/OV/bin/OpC/utils/opcnode -assign_pol_group node_name="nagioscore.test.com" net_type="NETWORK_IP" pol_type="MONITOR_TEMPLATE" pol_group="/NagiosCore"
Operation successfully completed.

[root@HP_OML ~]# opcragt -distrib -f nagioscore.test.com                                 # Push HP OML Policy on HP OML Agent

Check HP OML Agent

We've pushed new configuration to our HP OML Agent, according to this we’ll be able to see the HP OML Policy on our NagiosCore server.

On NagiosCore server running HP OML Agent:

[root@NagiosCore ~]# opctemplate                                                                  # This is older command used mainly at HP OVO
       * List installed policies for host 'localhost'.

     Version            Status
 --------------------------------------------------------------------
OPCMSG              "NagiosCore_opcmsg"                  enabled    0001.0000

######################################################################################
OR                                                                                                # OR
######################################################################################

[root@NagiosCore ~]# ovpolicy -list                                                               # This is new command used at HP OML
       * List installed policies for host 'localhost'.

     Version            Status
 --------------------------------------------------------------------
msgi              "NagiosCore_opcmsg"                  enabled    0001.0000

As well from HP OML server we can check it with command:

[root@HP_OML ~]# ovdeploy -cmd "opctemplate" -host nagioscore.test.com                            # This is older command used mainly at HP OVO
       * List installed policies for host 'localhost'.

Version            Status
 --------------------------------------------------------------------
OPCMSG              "NagiosCore_opcmsg"                  enabled    0001.0000

######################################################################################
OR                                                                                                # OR
######################################################################################

[root@HP_OML ~]# ovdeploy -cmd "ovpolicy -list" -host nagioscore.test.com                         # This is new command used at HP OML
        * List installed policies for host 'localhost'.

Version            Status
 --------------------------------------------------------------------
msgi              "NagiosCore_opcmsg"                  enabled    0001.0000

Nagios Core Configuration

To finis our integration we'll need to configure Nagios Core to forward interesting events in to HP OML (as well-known as HP OVO).

Based to the previous configuration that we have done at HP OML site, we will use command “opcmsg” at Nagios Core to forward the event in to HP OML like an alarm.

According to this we’ll need to:
- Create additional “command” object at Nagios Core, that will call directly or indirectly “opcmsg” command.
- Configure additional user defined variables related to “host” and “service” objects at Nagios Core.
- In the case that we'll need to backtrack each event sent from Nagios Core in to HP OML, we’ll need to write script that will manage it for us.

"opcmsg"

At this point it is interesting for use to get basic understanding of the HP OML command that we’ll use for alarm forwarding.

Variables:

It’s possible to run this command from CLI, this can provide us the possibility to integrate HP OML with another tools.

This command will take all parameters that we’ll provide him and it will create an alarm. To prevent overload of HP OML, we have configured our message policy. Mentioned policy and conditions are like firewall that is filtering and correlating our alarms before they will be sent to HP OML.

According to this we’ll need to use some parameters to match our HP OML Policy.

opcmsg [-id]                                                  # Optional, Alarm ID used for backtracking of alarm at HP OML
       [severity=normal|warning|minor|major|critical]         # Optional, Severity of HP OML Alarm
       application=<application>                              # Mandatory,Application used at HP OML Alarm, in our case “NagiosCore_<*>”
       object=<object>                                        # Mandatory,In our case we’ll use “host” object name used at Nagios Core
       msg_text=<text>                                        # Mandatory,Alarm message for support agent.
       msg_grp=<message group>]                               # Mandatory,Message group. Definition of who shell see this alarm at HP OML.
       [node=<node>]                                          # Optional, Default=[FQDN of node running HP OML Agent], We’ll use it to identify NagioCore server.
       [service_id=<svcid>]                                   # Optional, We’ll not use it as, we’ll provide this information at Application already.
       [-option <var>=<value>]                                # Optional, Depend of your HP OML setup if it’s needed to provide additional information.

Exit code:

In some cases it’s interesting to know if it was possible to forward the alarm to HP OML Agent. According to this “opcmsg” is providing the exit code with help of “$?” environment variable.

Please keep in mind that this exit code will not tell us if the alarm was delivered to HP OML.

This exit code can be used for validation of calling “opcmsg” command.

Successful

[root@NagiosCore ~]# opcmsg severity="normal" application="test" object="test" msg_text="test"; echo $?
0                                                                                                          # 0 == OK

Not Successful

[root@NagiosCore ~]# opcmsg severity="normal" application="test"  msg_text="test"; echo $?
The object parameter is required.  (OpC30-904)
Usage : opcmsg [-help] [-id] [severity=normal|warning|minor|major|critical]
               application=<application> object=<object> msg_text=<text>
               [msg_grp=<message group>] [node=<node>] [service_id=<svcid>]
               [-option <var>=<value>]* (OpC30-900)
2                                                                                                          # 0 <> NOK

Alarm ID

In some cases it’s important to backtrack alarms sent to HP OML. According to this we can ask HP OML Agent to provide us Alarm ID.

For some of you it’s probably not really important information. In this case you can configure Nagios Core to call the “opcmsg” command directly, without using any script.

On another hand I think it will be nice to provide information related to HP OML Alarm ID at each “host” and “service” object at Nagios Core like a “Comment”. I’ll use a script that will do it for us.

[root@NagiosCore ~]# opcmsg severity="normal" application="test" object="test" msg_text="test" -id
c8b5d07e-2964-71e4-1a58-0a000c680000                                                                       # Alarm ID

Simple Script

I'll use Location: "/opt/nagios-<VERSION>/sbin/Send_alarm_to_OML.pl"

This is just simple example of a script that I’ll use for event forwarding from Nagios Core in HP OML.

- After detecting that “host “or “service” has some issues, Nagios Core will call “command” object that is using this script.
- This script will take all relevant information and forward them in to HP OML Agent with help of “opcmsg” command.
- After the alarm was received at HP OML, we’ll get back the alarm ID and exit code from “opcmsg”
- Based on the exit code we’ll write some information to our “host” and “service” object at Nagios Core like a comment.

Please take this script only as an example as:

- It’s depending on setup of your monitoring tools, how you need to integrate them together.

- In real life, I would say that using a “Comments” at “host” and “services” to keep HP OML Alarm ID, isn’t the best way how to keep this information. When you’ll need to keep this information, please use standard log file. (You’ll need to change “nagios.cmd” to another log file and it will work)

- In the cases that you have implemented correlation at HP OML Policy Condition, it’s important to get information from Nagios Core when the Issue is gone. In this case it’s required to extend the script with additional parameter where we can provide new status of monitored object.

- In larger environments with plenty of “hosts” and “service” it’s better to keep all information in file with CSV format, instead of using custom variables (user specificity macros). In this way you can safe some resources and we can get this information on demand from mentioned file.

[root@NagiosCore ~]# cat /opt/nagios-4.0.7/sbin/Send_alarm_to_OML.pl
#!/usr/bin/perl -w

#
# This script can be used for alarm forwarding from NagiosCore in to HP OML.
# It's using “opcmsg” command delivered at HP OML Agent.
# According to this you need to have correctly configured HP OML Policy.
# As well it is logging the event ID in to NagiosCore like comment related to Host and Service.
#
# It's distributed under GNU GLP license.
#
# 2014-08-25 - iggy.ttt@gmail.com - Created
#
# Send_alarm_to_OML.pl [--host|-H] <NagiosCore_Host_Name> <--service|-S> <NagiosCore_Service_Name>
#                     [--severity|-se] [normal|warning|minor|major|critical] <--object|-o> <HP_OML_Object>
#                     <--application|-a> <HP_OML_Application> [--text|-t] [Message_Text]
#                     [--msg_grp|-mg] [Message_Group] <--help|-he]
#

use strict;
use Getopt::Long;

my $OPCMSG="/opt/OV/bin/opcmsg" ;                                     # Path to "opcmsg", change it if it is needed
my $NAGIOS_CMD="/opt/nagios-4.0.7/var/rw/nagios.cmd" ;                # Path to "nagios.cmd", change it if it is needed

my $date= time();                                                     # Local UX ERA time, we'll need it to write in to "nagios.cmd"
my $user=`whoami`;                                                    # Who is running the script, we'll need it to write in to "nagios.cmd"
chomp($user);

my $nagios_host="" ;                                                  # NagiosCore affected "host" name
my $nagios_service="Host_UP_DOWN" ;                                   # NagiosCore affected "Service" name
my $hp_oml_severity="" ;                                              # opcmsg severity, normal|warning|minor|major|critical
my $hp_oml_object="" ;                                                # opcmsg object, we'll use NagiosCore "host" name
my $hp_oml_application="" ;                                           # opcmsg application, we'll use NagiosCore "Service" name
my $hp_oml_message_text="" ;                                          # opcmsg msg_text
my $hp_oml_message_group="" ;                                         # opcmsg msg_grp, we'll use NagiosCore user defined macro
my $help=0 ;

my $cmd="";
my $hp_oml_alarm_id="";                                               # ID of the alarm sent to HP OML Agent
my $cmd_result="";                                                    # Command exit code

#
#### Get Variables ####
#
GetOptions ( "host|H=s"        => \$nagios_host,
             "service|S=s"     => \$nagios_service,
             "severity|se=s"   => \$hp_oml_severity,
             "object|o=s"      => \$hp_oml_object,
             "application|a=s" => \$hp_oml_application,
             "text|t=s"        => \$hp_oml_message_text,
             "msg_grp|mg=s"    => \$hp_oml_message_group,
             "help|he!"        => \$help,)
or die("Error in command line arguments\n");

#
#### HELP ####
#
print "
Send_alarm_to_OML.pl [--host|-H] <NagiosCore_Host_Name> <--service|-S> <NagiosCore_Service_Name>
                     [--severity|-se] [normal|warning|minor|major|critical] <--object|-o> <HP_OML_Object>
                     <--application|-a> <HP_OML_Application> [--text|-t] [Message_Text]
                     [--msg_grp|-mg] [Message_Group] <--help|-he]

       --host        -H        Mandatory, Nagios Core Host
       --service     -S        Optional,  Nagios Core Service (Default:Host_UP_DOWN)
       --severity    -se       Mandatory, HP OML Severity (normal|warning|minor|major|critical)
       --object      -o        Optional,  HP OML Object (Default same as --host|-H)
       --application -a        Optional,  HP OML Application (Default same as --service|-S)
       --text        -t        Mandatory, HP OML Message Text
       --msg_grp     -mg       Mandatory, HP OML Message Group
       --help        -he       Help
" if ($help);

#
#### Set missing variables ###
#
$nagios_service="Host_$nagios_host" if ($nagios_service eq "");                    # In the case that Service is not defined
$hp_oml_object=$nagios_host if ($hp_oml_object eq "");                             # NagiosCore Host == HP OML Object
$hp_oml_application="NagiosCore_$nagios_service" if ($hp_oml_application eq "");   # NagiosCore Service == HP OML Application

#
#### Send Alarm to HP OML Agent (opcmsg) #####
#
$cmd="$OPCMSG severity=\"$hp_oml_severity\" application=\"$hp_oml_application\" object=\"$hp_oml_object\" msg_text=\"$hp_oml_message_text\" msg_grp=\"$hp_oml_message_group\" -id" ;

$hp_oml_alarm_id=`$cmd`;                                              # Send alarm in to HP OML Agent
chomp($hp_oml_alarm_id);                                              # ID of the HP OML Alarm
$cmd_result=$?;                                                       # Exit code of command opcmsg OK == 0 | NOK <> 0
chomp($cmd_result);

#
#### Send Comment (HP OML alarm ID) in to the Nagios Core Host and Service ####
#
open(my $fh, ">", $NAGIOS_CMD )
    or die "cannot open $NAGIOS_CMD: $!";

#### HOST ALARM ####
print $fh "[$date] ADD_HOST_COMMENT;$nagios_host;1;$user;Success: HP OML ID = $hp_oml_alarm_id  for Host $nagios_host\n" if (($cmd_result == "0")&&($nagios_service eq "Host_UP_DOWN")) ;

print $fh "[$date] ADD_HOST_COMMENT;$nagios_host;1;$user;NOT Success: HP OML ID = $hp_oml_alarm_id  for Host $nagios_host\n" if (($cmd_result != "0")&&($nagios_service eq "Host_UP_DOWN")) ;

#### SERVICE ALARM ####
print $fh "[$date] ADD_HOST_COMMENT;$nagios_host;1;$user;Success: HP OML ID = $hp_oml_alarm_id  for $nagios_host - $nagios_service\n" if (($cmd_result == "0")&&($nagios_service ne "Host_UP_DOWN")) ;

print $fh "[$date] ADD_HOST_COMMENT;$nagios_host;1;$user;NOT Success: HP OML ID = $hp_oml_alarm_id  for Host $nagios_host - $nagios_service\n" if (($cmd_result != "0")&&($nagios_service ne "Host_UP_DOWN")) ;

print $fh "[$date] ADD_SVC_COMMENT;$nagios_host;$nagios_service;1;$user;Success: HP OML ID = $hp_oml_alarm_id  for $nagios_host - $nagios_service\n" if (($cmd_result == "0")&&($nagios_service ne "Host_UP_DOWN")) ;

print $fh "[$date] ADD_SVC_COMMENT;$nagios_host;$nagios_service;1;$user;NOT Success: HP OML ID = $hp_oml_alarm_id  for Host $nagios_host - $nagios_service\n" if (($cmd_result != "0")&&($nagios_service ne "Host_UP_DOWN")) ;

close($fh);

exit(0);

Make sure that this script is owed at Nagios user:

[root@NagiosCore ~]# chown nagios:nagios /opt/nagios-4.0.7/sbin/Send_alarm_to_OML.pl

Commands Object

I'll use Location: "/opt/nagios-<VERSION>/etc/objects/test_node.cfg

To use our script it’s required to configure “command” 's object at Nagios Core.

Mentioned “command” 's object will:
- Take variables from Nagios Core to call our script.
- Be called in the case that Nagios Core will detect some issue related to monitored object.

define command{
        command_name    host-to-hp_oml
        command_line    /opt/nagios-4.0.7/sbin/Send_alarm_to_OML.pl --host $HOSTNAME$ --severity $_HOSTSEVERITY$ --object $HOSTNAME$ --text "Host $HOSTNAME$ is $HOSTSTATE$" --msg_grp $_HOSTMSGGRP$
        }

define command{
        command_name    service-to-hp_oml
        command_line    /opt/nagios-4.0.7/sbin/Send_alarm_to_OML.pl  --host $HOSTNAME$ --service $SERVICEDESC$ --severity $_SERVICESEVERITY$ --object $HOSTNAME$ --text "Service $SERVICEDESC$ on $HOSTNAME$ is $SERVICESTATE$"  --msg_grp $_SERVICEMSGGRP$
        }

Contact / Contactgroup Object

I'll use Location: "/opt/nagios-<VERSION>/etc/objects/test_node.cfg

Nagios Core is providing several possibilities where we can define the command that will be used for alarm handling.

I’ll use definition of “contactgroup” object to call “command” object, handling detected event. It’s possible to create several “contact” and “contactgroup” objects, which can be used to handle detected event in different way.

define contact{
        contact_name                    test_nagiosadmin
        alias                           Nagios Admin
        email                           nagios@localhost
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,f                                 # Call our Command only when there is failure
        host_notification_options       d,u,f                                   # Call our Command only when there is failure
        service_notification_commands   service-to-hp_oml                       # Our Command object
        host_notification_commands      host-to-hp_oml                          # Our Command object
        }

define contactgroup{
        contactgroup_name               test_admins                             # We will assign this contact group to our host's and service's
        alias                           Nagios Administrators
        members                         test_nagiosadmin
        }

Host / Service

I'll use Location: "/opt/nagios-<VERSION>/etc/objects/test_node.cfg

In most cases it’s possible to use template that will be including all common parameters of “host” or “service” object at Nagios Core. In this way you can safe lot of space as you can prevent duplicity of same information.

In my example I’m going to use the whole configuration at “host” and “service” as it can help us understand the configuration.

As well we’ll need to use some custom variables (user defined macros) that will provide us additional information that are required at HP OML Agent (opcmsg command).

define host{
        host_name                       test_node.test.com                        # Name of node
        alias                           test_node
        address                         test_node.test.com                        # FQDN or IP
        check_period                    24x7
        check_interval                  5
        retry_interval                  3
        max_check_attempts              1
        check_command                   check-host-alive
        notification_period             24x7
        notification_interval           0                                         # Call our script only 1 time, when the issue was detected
        notification_options            d,u,f                                     # Only when it has an issue
        contact_groups                  test_admins                               # Contact Group that will call our script
        notifications_enabled           1
        event_handler_enabled           1
        flap_detection_enabled          1
        process_perf_data               1
        retain_status_information       1
        retain_nonstatus_information    1
        notification_period             24x7
        _severity                       critical                                  # Custom Variable required at HP OML Agent (opcmsg)
        _msggrp                         test                                      # Custom Variable required at HP OML Agent (opcmsg)
        }

define service{
        host_name                       test_node.test.com                        # Related host
        service_description             PING
        check_command                   check_ping!100.0,20%!500.0,60%            # Monitoring "command" object
        max_check_attempts              1
        normal_check_interval           5
        retry_check_interval            1
        active_checks_enabled           1
        passive_checks_enabled          1
        parallelize_check               1
        obsess_over_service             1
        check_freshness                 1
        notifications_enabled           1
        event_handler_enabled           1
        flap_detection_enabled          1
        process_perf_data               1
        retain_status_information       1
        retain_nonstatus_information    1
        is_volatile                     1
        check_period                    24x7
        normal_check_interval           5
        retry_check_interval            3
        contact_groups                  test_admins                               # Contact Group that will call our script
        notification_options            w,u,c,f                                   # Call our script only when there is an issue
        notification_interval           0                                         # Call our script only 1 time, when the issue was detected
        notification_period             24x7
        _severity                       critical                                  # Custom Variable required at HP OML Agent (opcmsg)
        _msggrp                         test                                      # Custom Variable required at HP OML Agent (opcmsg)
      }

nagios.cfg

Location: "/opt/nagios-<VERSION>/etc/nagios.cfg

At this point it depend on your setup of Nagios Core.

Sometime it’s better to use definition of directory where Nagios Core will look for configuration files.

In our case I’ve used one file for our configuration of “test_node”. According to this it’s enough to enable this line at “nagios.cfg” file to include our configuration in to Nagios Core:

[root@NagiosCore ~]# grep test /opt/nagios-4.0.7/etc/nagios.cfg
cfg_file=/opt/nagios-4.0.7/etc/objects/test_node.cfg

Reload Nagios Core

At this moment we have done the whole configuration and all that we need to do is to restart our Nagios Core to reload the new configuration.

Validate new config of Nagios Core:

[root@NagiosCore ~]# /opt/nagios-4.0.7/bin/nagios -v /opt/nagios-4.0.7/etc/nagios.cfg

Nagios Core 4.0.7
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 06-03-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 9 services.
        Checked 2 hosts.
        Checked 1 host groups.
        Checked 0 service groups.
        Checked 2 contacts.
        Checked 2 contact groups.
        Checked 26 commands.
        Checked 5 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 2 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 5 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0                                                                           # IT's OK
Total Errors:   0                                                                           # IT's OK

Things look okay - No serious problems were detected during the pre-flight check

Restart Nagios Core:

[root@NagiosCore ~]# /etc/rc.d/init.d/nagios restart
Running configuration check...
Stopping nagios: .done.
Starting nagios: done.

Test

According to used checks used at Nagios Core, you’ll need to decide how to force Nagios Core to trigger an alarm.

In my example, I have used standard ping check. According to this it’s possible to use “iptables” to disable traffic between Nagios Core and monitored host. This shall trigger an alarm that will be forwarded in to HP OML Agent.

[root@NagiosCore ~]# iptables -I INPUT -s test_node.test.com -j DROP                        # Disable incoming traffic from our test host

After status polling of monitored object at Nagios Core it shall be possible to see:
- Alarm at HP OML Alarm browser (as well-known as ITO or OVO alarm browser)
- Comment with HP OML Alarm ID at “host” or “service” at Nagios Core Web GUI
- New “hostcomment” and “servicecomment” at Nagios Core in file “/opt/nagios-4.0.7/var/status.dat”

hostcomment {
        host_name=test_node.test.com
        entry_type=1
        comment_id=47
        source=1
        persistent=1
        entry_time=1408966049
        expires=0
        expire_time=0
        author=nagios
        comment_data=Success: HP OML ID = cc2bee3a-2c4a-71e4-14dd-103ea5650000  for test_node.test.com - PING
        }

servicecomment {
        host_name=test_node.test.com
        service_description=PING
        entry_type=1
        comment_id=48
        source=1
        persistent=1
        entry_time=1408966049
        expires=0
        expire_time=0
        author=nagios
        comment_data=Success: HP OML ID = cc2bee3a-2c4a-71e4-14dd-103ea5650000  for test_node.test.com - PING
        }

hostcomment {
        host_name=test_node.test.com
        entry_type=1
        comment_id=49
        source=1
        persistent=1
        entry_time=1408966128
        expires=0
        expire_time=0
        author=nagios
        comment_data=Success: HP OML ID = fb285aac-2c4a-71e4-0cc6-103ea5650000  for Host test_node.test.com
        }

- New logged information at Nagios Core in file “/opt/nagios-4.0.7/var/nagios.log”

[1408966049] SERVICE ALERT: test_node.test.com;PING;CRITICAL;HARD;1;PING CRITICAL - Packet loss = 100%
[1408966049] SERVICE NOTIFICATION: test_nagiosadmin;test_node.test.com;PING;CRITICAL;service-to-hp_oml;PING CRITICAL - Packet loss = 100%                                  # Service Down Alarm
[1408966049] EXTERNAL COMMAND: ADD_HOST_COMMENT;test_node.test.com;1;nagios;Success: HP OML ID = cc2bee3a-2c4a-71e4-14dd-103ea5650000  for test_node.test.com - PING       # Write HP OML ID to Host
[1408966049] EXTERNAL COMMAND: ADD_SVC_COMMENT;test_node.test.com;PING;1;nagios;Success: HP OML ID = cc2bee3a-2c4a-71e4-14dd-103ea5650000  for test_node.test.com - PING   # Write HO OML ID to Service
[1408966079] HOST ALERT: test_node.test.com;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
[1408966128] HOST ALERT: test_node.test.com;DOWN;HARD;1;PING CRITICAL - Packet loss = 100%
[1408966128] HOST NOTIFICATION: test_nagiosadmin;test_node.test.com;DOWN;host-to-hp_oml;PING CRITICAL - Packet loss = 100%                                                 # Host Down Alarm
[1408966128] EXTERNAL COMMAND: ADD_HOST_COMMENT;test_node.test.com;1;nagios;Success: HP OML ID = fb285aac-2c4a-71e4-0cc6-103ea5650000  for Host test_node.test.com         # Write HP OML ID to Host
Navigation
Print/export
QR Code
QR Code wiki:infrastructure_tools:nagios:nagios_core_-_hp_oml (generated for current page)