{"id":571,"date":"2016-04-26T18:57:36","date_gmt":"2016-04-26T15:57:36","guid":{"rendered":"http:\/\/www.bandidor.info\/wp\/?p=571"},"modified":"2016-04-26T18:57:36","modified_gmt":"2016-04-26T15:57:36","slug":"monitor-bacula-backup-jobs-with-zabbix-part-ii","status":"publish","type":"post","link":"https:\/\/www.bandidor.info\/wp\/?p=571","title":{"rendered":"Monitor bacula backup jobs with zabbix — part II"},"content":{"rendered":"

Subject<\/span><\/strong><\/h1>\n
In my previous article<\/a> I described only a very simple part of my monitoring solution for bacula<\/em>, the focus was on discovering backup jobs and only job exit status was monitored. In this article, I’ll add more monitoring parameters and create some graphs.<\/div>\n

Symptoms<\/span><\/strong><\/h1>\n
The challenge here is that no further job parameters are provided by bacula<\/em> director when it calls the mail command and thus my wrapper script. To make more job parameters available for monitoring I’ll extend my wrapper script to also look into the bacula underlying database and fetch job statistics from there.<\/div>\n

Platform<\/b><\/span>\/Tools<\/span><\/strong><\/h1>\n
No news here, still my server is Ubuntu 14.04.3 LTS, bacula 5.2.6, zabbix 2.2.6. Also, MySQL version is\u00a05.5.44.<\/div>\n

Solution<\/span><\/strong><\/h2>\n

Monitoring individual jobs<\/h2>\n

Zabbix items<\/h3>\n
\u00a0My plan is to monitor the number of files processed by the particular job, the number of bytes read and written. So for each job our zabbix<\/em> discovery rule will have 3 additional item prototypes. Below I provide only one screenshot, other two are very similar:<\/div>\n
\"ScreenShot114\"<\/a><\/div>\n

Add new items to the wrapper script<\/h3>\n
The idea is simple, after we send the item reporting job exit status to zabbix<\/em>, we use job name passed to our script with the option -j to go to the bacula database and fetch some job statistics.\u00a0This looks a bit clumsy at the first glance, our script will need to know where the database is located and also the user name and the password making our script less portable, but that’s the price.<\/div>\n
Here we go :<\/div>\n
\r\n...\r\nmy $MYSQL_SERVER = '127.0.0.1';\r\nmy $MYSQL_PORT = '3306';\r\nmy $MYSQL_USER = 'bacula';\r\nmy $MYSQL_PASSWORD = 'yRacJaj4Eujw';\r\nmy $DATABASE = 'bacula';\r\n...\r\nmy $JOB_FILES_KEY = 'bacula.job.sumFiles[%s]';\r\nmy $JOB_BYTES_KEY = 'bacula.job.sumBytes[%s]';\r\nmy $JOB_READ_BYTES_KEY = 'bacula.job.sumReadBytes[%s]';\r\n...\r\nmy $dbh = DBI->connect("DBI:mysql:database=$DATABASE;host=$MYSQL_SERVER;port=$MYSQL_PORT",$MYSQL_USER, $MYSQL_PASSWORD,{'RaiseError' => 1});\r\n\r\n### Send statistics for individual job\r\nmy $sth = $dbh->prepare("SELECT JobFiles,JobBytes,ReadBytes, P.Name AS Pool, if(isnull(sum(`M`.`VolBytes`)),0,sum(`M`.`VolBytes`)) AS `PoolBytes`,P.NumVols FROM Job J, (`Pool` `P` left join `Media` `M` on((`P`.`PoolId` = `M`.`PoolId`))) WHERE J.PoolId = P.PoolId AND J.Job = ?;");\r\n$sth->execute($options{'j'}); #Job name in the form BackupCatalog.2015-02-01_23.10.00_03\r\nmy ($jobFiles,$jobBytes,$readBytes,$poolName,$poolBytes,$poolVols) = $sth->fetchrow_array(); #exactly one row expected\r\n$sth->finish();\r\n\r\n#Send for the job\r\nsystem(sprintf($zabbix_sender_cmd_line,sprintf($JOB_FILES_KEY,$options{'n'}),$jobFiles) . " >\/dev\/null");\r\nsystem(sprintf($zabbix_sender_cmd_line,sprintf($JOB_BYTES_KEY,$options{'n'}),$jobBytes) . " >\/dev\/null");\r\nsystem(sprintf($zabbix_sender_cmd_line,sprintf($JOB_READ_BYTES_KEY,$options{'n'}),$readBytes) . " >\/dev\/null");\r\n...\r\n$dbh->disconnect();\r\n...\r\n<\/pre>\n
Nothing special here, first we set all variables to access MySQL database, then we specify the item keys that follow the monitoring items we configured in zabbix<\/em>. Please note that the SELECT statement we use is a bit more complex than expected, this we will need for further extensions.<\/div>\n
The last step, we send 3 monitoring items to zabbix<\/em>.<\/div>\n
Adding a trigger<\/h3>\n
We start with a simple trigger, which will check for the job return status. If it’s not OK – something is wrong:<\/div>\n
\"ScreenShot117\"<\/a><\/div>\n
And we add another trigger, this one will check for the number of bytes written, if it is zero – most probably something is wrong. Still there is a small possibility that for example an incremental job has not found any changed files, therefore, if it will be only s Warning:<\/div>\n
\"ScreenShot118\"<\/a><\/div>\n
\n
Adding a graph<\/h3>\n<\/div>\n
The last step in adding prototypes for a job would be to add a graph with the information about bytes read, bytes written and files processed. Graph configuration is absolutely straightforward, so I would better prove a nice picture after some backup cycles. Below is a 3-month graph for one of the backup jobs.<\/div>\n
<\/div>\n
\"ScreenShot111\"<\/a><\/div>\n
<\/div>\n
What about 24-hour statistics?<\/h2>\n
There is a very nice feature in the webacula<\/em> tool, I installed very long ago. It shows some aggregated statistics for the last 24 hours. Although we can do some aggregation inside zabbix<\/em>, I decided to create additional monitoring items and make my wrapper script to also send aggregated statistics.<\/p>\n
More zabbix items<\/h3>\n
Here I made my first mistake, I added aggregated item prototypes to the zabbix<\/em> Discovery rule. This didn’t work because aggregated items are not bound to individual jobs, while zabbix discovery will try to create new items for every job discovered. So these items shall be created as template items:<\/p>\n
\"ScreenShot115\"<\/a><\/p>\n
Extending script<\/h3>\n
Same approach as before, we make sure that we use proper keys for our new items and a\u00a0different\u00a0SQL command:<\/p>\n
\r\n\r\n...\r\nmy $SUM_FILES_KEY = 'bacula.24hr.sumFiles';\r\nmy $SUM_BYTES_KEY = 'bacula.24hr.sumBytes';\r\nmy $SUM_READ_BYTES_KEY = 'bacula.24hr.sumReadBytes';\r\n...\r\n### Send statstics for the entire installation for the last 24 hr\r\n$sth = $dbh->prepare("SELECT sum(`JobFiles`),sum(`JobBytes`),sum(`ReadBytes`) from `Job` where (`StartTime` > (now() - interval 1 day));");\r\nmy ($jobFiles24,$jobBytes24,$readBytes24) = $sth->fetchrow_array(); #exactly one row expected\r\n$sth->finish();\r\n\r\nsystem(sprintf($zabbix_sender_cmd_line,sprintf($SUM_FILES_KEY,$options{'n'}),$jobFiles24) . " >\/dev\/null");\r\nsystem(sprintf($zabbix_sender_cmd_line,sprintf($SUM_BYTES_KEY,$options{'n'}),$jobBytes24) . " >\/dev\/null");\r\nsystem(sprintf($zabbix_sender_cmd_line,sprintf($SUM_READ_BYTES_KEY,$options{'n'}),$readBytes24) . " >\/dev\/null");\r\n### Done with the 24-hr\r\n...\r\n\r\n<\/pre>\n
Adding a trigger for 24 hr<\/h3>\n
While I have my jobs running on different schedules, I expect to backup something every night. So I decided to have a trigger, which will raise an alarm if no backups have run in last 24 hours. In fact, I’m checking for the number of bytes written in last 24 hours and my trigger will fire up if this is zero. Remember, this will be Template trigger and not a Discovery rule trigger prototype:<\/div>\n
\"ScreenShot116\"<\/a><\/div>\n
<\/div>\n
\n
24 hr graph<\/h3>\n<\/div>\n
Having collected information for the last 24 hours we have a very simple job to create a graph to show the number of bytes read, bytes written and files processed. Don’t forget though that this will be not a graph prototype, it will be a template graph. This is because we will have only one graph per host and not per discovered job.<\/div>\n
<\/div>\n
\"ScreenShot112\"<\/a><\/div>\n
<\/div>\n
Discussion<\/span><\/strong><\/div>\n
Well, graphs could have been sexier, but I leave it to the future articles.<\/p>\n
Caveats<\/span><\/strong><\/div>\n
None detected so far, but I must admit I’m publishing this article quite some time after I have implemented this solution and I don’t look at these graphs frequently.<\/div>\n
<\/div>\n
<\/div>\n","protected":false},"excerpt":{"rendered":"
Subject In my previous article I described only a very simple part of my monitoring solution for bacula, the focus was on discovering backup jobs and only job exit status was monitored. In this article, I’ll add more monitoring parameters and create some graphs. Symptoms The challenge here is that no further job parameters are…<\/p>\n","protected":false},"author":1,"featured_media":650,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[22,26],"tags":[31,27,32,17],"class_list":["post-571","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bacula","category-zabbix","tag-backup","tag-bacula","tag-monitoring","tag-zabbix"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/bandidor.info\/wp\/wp-content\/uploads\/2016\/04\/2014-11-02-20.02.56.jpg","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2EszU-9d","_links":{"self":[{"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/posts\/571","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=571"}],"version-history":[{"count":17,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/posts\/571\/revisions"}],"predecessor-version":[{"id":652,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/posts\/571\/revisions\/652"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=\/wp\/v2\/media\/650"}],"wp:attachment":[{"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=571"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=571"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bandidor.info\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=571"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}