{"id":307,"date":"2014-07-14T14:35:13","date_gmt":"2014-07-14T10:35:13","guid":{"rendered":"http:\/\/www.bandidor.info\/wp\/?p=307"},"modified":"2017-03-02T17:15:58","modified_gmt":"2017-03-02T14:15:58","slug":"md-raid-and-zabbix","status":"publish","type":"post","link":"https:\/\/www.bandidor.info\/wp\/?p=307","title":{"rendered":"md raid and zabbix"},"content":{"rendered":"

Subject<\/h1>\n

In this article I explain how to setup Software RAID monitoring with zabbix<\/em>. The solution will use two ways – active monitoring (polling) of an array status and passive traps for array events, sent out by the mdadm<\/strong> daemon.<\/p>\n

Symptoms<\/h1>\n
There were two challenges to deal with. First, even configured properly the e-mail notification produced by mdadm monitoring was not working effectively. E-mails were often overlooked of even landed in SPAM folder. One happy day one of the array disks failed, this went unnoticed and when the next disk crashed I was really in trouble.\u00a0Secondly, there are quite a few events, generated by the mdadm<\/strong> daemon in monitoring mode and I didn’t want to configure zabbix<\/em> items and trigger manually every time<\/div>\n

Plattform\/Tools<\/h1>\n
Nothing special here, software RAID on a debian machine with Debian GNU\/Linux 7.5 (wheezy)<\/code>\u00a0and\u00a0zabbix\u00a0<\/em>sever on ubuntu Ubuntu 12.04.4 LTS<\/code>.\u00a0Zabbix\u00a0<\/em>agent\u00a0v2.2.3,\u00a0zabbix\u00a0<\/em>server –\u00a0v2.2.2.<\/div>\n

Solution<\/h1>\n

Discovery of md arrays<\/h2>\n

Create Perl script to support array discovery. I started with shell first, but then switched to my favourite Perl in particular to make sure the script sends properly formatted JSON back to zabbix. So you will need Perl JSON module installed. This looks to be a bit of heavy-weighted, \u00a0but you wouldn’t want to run discovery every second anyway.<\/p>\n

# cat \/usr\/lib\/zabbix\/externalscripts\/zabbix_mdraid.pl\r\n#!\/usr\/bin\/perl\r\nuse strict;\r\nuse warnings;\r\nuse Getopt::Std;\r\nuse JSON;\r\n\r\n# declare the perl command line flags\/options we want to allow\r\nmy %options=();\r\ngetopts(\"D\", \\%options);\r\nmy $arrays_found = undef;\r\nif ($options{D}) {\r\n        open(my $fh, '-|', 'cat \/proc\/mdstat') or die $!;\r\n        while (my $line = <$fh>) {\r\n                if ($line =~ \/^(md\\S*)\/) {\r\n                        if ($arrays_found) {\r\n                                push(@{$arrays_found->{'data'}},{'{#MD_DEVICE}' => ('\/dev\/' . $1)});\r\n                        } else {\r\n                                $arrays_found->{'data'}->[0] =  {'{#MD_DEVICE}' => ('\/dev\/' . $1)};\r\n                        }\r\n                }\r\n        }\r\n}\r\n\r\nprint encode_json($arrays_found) if ($arrays_found);<\/pre>\n
Add discovery script to zabbix agent configuration file as a UserParameter:<\/p>\n
# cat \/etc\/zabbix\/zabbix_agentd.d\/mdraid.conf\r\n...\r\nUserParameter=mdraid.discovery, sudo \/usr\/lib\/zabbix\/externalscripts\/zabbix_mdraid.pl -D\r\n...<\/pre>\n
Make sure you gave sudo permissions for the script:<\/p>\n
# cat \/etc\/sudoers.d\/zabbix\r\n...\r\nzabbix  ALL=NOPASSWD:   \/usr\/lib\/zabbix\/externalscripts\/zabbix_mdraid.pl\r\n...<\/pre>\n
And of course configure the discovery on zabbix server:<\/p>\n
\"ScreenShot066\"<\/a><\/p>\n
<\/div>\n
Reporting array status<\/h2>\n
\n
We will extend the above script\u00a0\/usr\/lib\/zabbix\/externalscripts\/zabbix_mdraid.pl<\/span>\u00a0to also support the check for the array status. It will be run with the command line options -s \/dev\/mdX<\/code> to report the status. Now it will look as follows:<\/p>\n<\/div>\n
\u00a0<\/span><\/span><\/div>\n
\n
#!\/usr\/bin\/perl\r\nuse strict;\r\nuse warnings;\r\nuse Getopt::Std;\r\nuse JSON;\r\n\r\n# declare the perl command line flags\/options we want to allow\r\nmy %options=();\r\ngetopts(\"Ds:\", \\%options);\r\n\r\nif ($options{D}) {\r\n\tmy $arrays_found = undef;\r\n\topen(my $fh, '-|', 'cat \/proc\/mdstat') or die $!;\r\n\twhile (my $line = <$fh>) {\r\n\t\tif ($line =~ \/^(md\\S*)\/) {\r\n\t\t\tif ($arrays_found) {\r\n\t\t\t\tpush(@{$arrays_found->{'data'}},{'{#MD_DEVICE}' => ('\/dev\/' . $1)});\r\n\t\t\t} else {\r\n\t\t\t\t$arrays_found->{'data'}->[0] =  {'{#MD_DEVICE}' => ('\/dev\/' . $1)};\r\n\t\t\t}\r\n\t\t}\r\n\t}\r\n\tprint encode_json($arrays_found) if ($arrays_found);\r\n} elsif ($options{s}) {\r\n\t#Will be checking status of a given array\r\n\topen(my $fh, '-|', 'mdadm --detail ' . $options{s}) or die $!;\r\n\twhile (my $line = <$fh> ) {\r\n\t\tif ($line =~ \/State\\s*:\\s*(\\S.*)$\/) {\r\n\t\t\tprint $1;\r\n\t\t\tlast;\r\n\t\t}\r\n\t}\r\n}<\/pre>\n<\/div>\n
We will add it to zabbix agent configuration file as a User Parameter:<\/p>\n
\n
...\r\nUserParameter=mdraid.status[*], sudo \/usr\/lib\/zabbix\/externalscripts\/zabbix_mdraid.pl -s '$1'\r\n...<\/pre>\n<\/div>\n
<\/div>\n
And configure a zabbix item:<\/p>\n
\"ScreenShot068\"<\/a><\/div>\n
<\/div>\n
Receiving alerts when array status changes<\/h2>\n
On my Debian\u00a0wheezy\u00a0there is a background process, which monitors the status of md arrays:<\/p>\n
<\/div>\n
\n
\r\n~# ps -ef | grep md\r\nroot 23 2 0 Jun04 ? 00:00:00 [ksmd]\r\nroot 253 2 0 Jun04 ? 00:00:00 [md]\r\nroot 260 2 0 Jun04 ? 00:19:53 [md0_raid1]\r\nroot 2998 1 0 Jun04 ? 00:00:00 \/sbin\/mdadm --monitor --pid-file \/run\/mdadm\/monitor.pid --daemonise --scan --syslog\r\nroot 10814 12784 0 19:27 pts\/1 00:00:00 grep md\r\n<\/pre>\n<\/div>\n
This process will use\u00a0PROGRAM <\/b>directive from the\u00a0mdadm.conf <\/b>file:<\/p>\n
<\/div>\n
~# cat \/etc\/mdadm\/mdadm.conf\r\n# mdadm.conf\r\n#\r\n# Please refer to mdadm.conf(5) for information about this file.\r\n#...\r\n\r\nPROGRAM \/usr\/lib\/zabbix\/externalscripts\/mdadm.pl\r\n...\r\n<\/pre>\n
<\/div>\n
The program specified by the above directive will be run when\u00a0<\/span>mdadm –monitor<\/b>\u00a0detects potentially interesting events on any of the arrays that it is monitoring. This program gets run with two or three arguments, they being the Event, the md device, and possibly the related component device.<\/span><\/p>\n
The\u00a0mdadm.pl script will send these events to zabbix server in the format similar to mdraid.status.TestMessage[\/dev\/md0]<\/code>.<\/p>\n
# cat \/usr\/lib\/zabbix\/externalscripts\/mdadm.pl\r\n#!\/usr\/bin\/perl\r\n$ZABBIX_SERVER='192.168.100.8';\r\n$MYHOSTNAME='openfiler';\r\n$SENDER=`which zabbix_sender`;\r\nchomp($SENDER);\r\n$KEY='mdraid.status';\r\n\r\nif (!$MYHOSTNAME || !$ZABBIX_SERVER || !$KEY || ! -x $SENDER) {\r\nexit;\r\n}\r\n\r\nif ($ARGV[0] =~ \/(Rebuild)(\\d+)\/) {\r\n$ARGV[0] = $1;\r\n$ARGV[2] = $2;\r\n} elsif (!$ARGV[2]) {\r\n$ARGV[2] = 'x';\r\n}\r\n\r\nmy @result = `$SENDER -z '$ZABBIX_SERVER' -k $KEY."$ARGV[0]"["$ARGV[1]"] -o "$ARGV[2]" -s '$MYHOSTNAME'`;\r\n\r\n#print @result;\r\n\r\nexit 0;\r\n\r\n<\/pre>\n
<\/div>\n
These events will be received by zabbix items of the type Zabbix trapper<\/em>. This is described in the next section.<\/p>\n
Configuring trappers in zabbix<\/h2>\n
To process mdadm events in a\u00a0form similar to \u00a0mdraid.status.TestMessage[\/dev\/md0]<\/code>\u00a0we will create a number of zabbix items of the type\u00a0Zabbix trapper<\/em>. An item has to be created for all possible mdadm events.<\/p>\n
Here is an example:<\/p>\n
\"ScreenShot074\"<\/a><\/p>\n
When all needed items are created, we will see the following in the list of host items:<\/p>\n
\"\"<\/a><\/p>\n
Event names, like TestMessage<\/em>, DegradedArray<\/em>, etc. are directly taken from the mdadm <\/strong>man page.<\/p>\n
Each trapper item will also have a corresponding trigger with the severity, which again follows the mdadm<\/strong> man page. Here is a list of triggers:<\/p>\n
\"\"<\/a><\/p>\n
 <\/p>\n
Creating performance counters for RAID devices<\/h2>\n
This is more a part of discovery and is not specific to RAID devices, so we will create some item prototypes for every discovered RAID device. For simplicity we will use new item types available in zabbix<\/em> since 2.0. These are vfs.dev.read[<device>,<type>,<mode>] <\/strong>and\u00a0vfs.dev.write[<device>,<type>,<mode>]<\/strong><\/strong>\u00a0.<\/p>\n
For example:<\/p>\n
\"ScreenShot077\"<\/a><\/p>\n
 <\/p>\n
Trigger for RAID state change<\/h2>\n
Just as a last piece in the puzzle, here is one additional trigger to control the array state. If will be using an item created previously, which in turn polls the \/proc\/mdadm<\/em> file for the array state. On the assumption that normal array operation will have array state in clean<\/em> or active<\/em>, we will be using a regular expression in the trigger condition. By the way, if we put one of the disks to a failed state via mdadm --fail \/dev\/md0 \/dev\/sdb1<\/code>, the array state will show as\u00a0clean, degraded<\/em>.<\/p>\n
\"ScreenShot078\"<\/a><\/p>\n
Trigger expressions is as follows:\u00a0mdraid.status[{#MD_DEVICE}].regexp(^clean$|^active$,#1)}=0<\/code>.<\/p>\n
Transfer to production<\/h2>\n
    \n
  1. Copy\u00a0\/usr\/lib\/zabbix\/externalscripts\/zabbix_mdraid.pl<\/code>\u00a0to the RAID host. Make sure\u00a0 JSON.pm<\/strong> is installed.<\/span><\/li>\n
  2. Copy \/etc\/zabbix\/zabbix_agentd.conf<\/code>\u00a0to the RAID host. Restart zabbix agent.<\/li>\n
  3. Add lines to\u00a0\/etc\/sudoers.d\/zabbix<\/code><\/li>\n
  4. Update\u00a0<\/span>cat \/etc\/mdadm\/mdadm.conf<\/code>\u00a0to set PROGRAM<\/strong><\/span><\/span><\/li>\n
  5. Copy\u00a0\/usr\/lib\/zabbix\/externalscripts\/mdadm.pl<\/code>\u00a0to the RAID host, update zabbix server IP and the RAID server hostname.<\/span><\/li>\n
  6. Import the template to zabbix server.<\/li>\n<\/ol>\n
    Discussion<\/h1>\n
    Not much of it actually. Probably I’ll have to wait for another big crash to see if this monitoring solution works better than the original e-mail notification, provided by\u00a0mdamd<\/strong> out of the box. Couple of things I tried in my test environment and that worked: more than one array on the same host, different combinations of disk failures, etc.<\/div>\n
    Another interesting area would be to add more items to gather statistic for discovered arrays, but I decided against that. One thing could be useful, namely\u00a0watching for\u00a0time spent doing I\/Os (ms)\u00a0<\/strong>from\u00a0\/proc\/diskstats, but this has to be done for physical disks IMHO and as zabbix is not yet providing this item type, I had to configure user defined items in\u00a0zabbix <\/em>agent configuration file. But this is a subject for another article.<\/div>\n
    Caveats<\/h1>\n
    None as of now. Just a humble suggestion to watch out for the right spelling of the macros (like {#MD_ARRAY}) in\u00a0zabbix\u00a0<\/em>discovery rules and item prototypes, looking for mistakes will take long time.<\/div>\n
    <\/div>\n","protected":false},"excerpt":{"rendered":"
    Subject In this article I explain how to setup Software RAID monitoring with zabbix. The solution will use two ways – active monitoring (polling) of an array status and passive traps for array events, sent out by the mdadm daemon. Symptoms There were two challenges to deal with. First, even configured properly the e-mail notification…<\/p>\n","protected":false},"author":1,"featured_media":394,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[4,26],"tags":[19,20,21,23,17],"class_list":["post-307","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ubuntu","category-zabbix","tag-mdraid","tag-raid","tag-software-raid","tag-ubuntu","tag-zabbix"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/www.bandidor.info\/wp\/wp-content\/uploads\/2014\/07\/DSC00727.jpg","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2EszU-4X","_links":{"self":[{"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/posts\/307","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=307"}],"version-history":[{"count":56,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/posts\/307\/revisions"}],"predecessor-version":[{"id":666,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/posts\/307\/revisions\/666"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/media\/394"}],"wp:attachment":[{"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=307"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=307"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=307"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}