Icinga

Install Icinga master (server)
http://www.panticz.de/Install-icinga

Configure by ssh (minimalistic setup on clients)
http://www.panticz.de/Install-icinga-SSH-on-client

Templates

/etc/icinga/objects/generic-service_icinga.cfg 

Check syntax

sudo /etc/init.d/icinga check

Check return codes

0 - OK
1 - warning
2 - critical
3 - unknown

Icons

wget http://cdn1.iconfinder.com/data/icons/fatcow/16x16/mail_yellow.png -O /usr/share/nagios/htdocs/images/logos/base/mail.png
wget http://cdn1.iconfinder.com/data/icons/free-business-desktop-icons/16/Telephone.png -O /usr/share/nagios/htdocs/images/logos/base/phone.png
wget http://cdn1.iconfinder.com/data/icons/nuvola2/16x16/devices/printer1.png -O /usr/share/nagios/htdocs/images/logos/base/printer.png

Docs
http://docs.icinga.org/latest/en/objectdefinitions.html#objectdefinitions-hostgroup - Object Definitions
http://books.google.de/books?id=Pessksf55WQC&printsec=frontcover - Learning Nagios 3.0

Change icingaadmin password

htpasswd /etc/icinga/htpasswd.users icingaadmin

Change Icinga password

htpasswd /etc/icinga/htpasswd.users foo
sed -i 's|icingaadmin|foo|g' /etc/icinga/cgi.cfg
/etc/init.d/apache2 restart
/etc/init.d/icinga restart

Accecp (new) host key

su -s /bin/bash nagios
ssh-keygen -f "/var/lib/nagios/.ssh/known_hosts" -R backup.example.com
ssh backup.example.com

CGI
http://docs.icinga.org/latest/de/cgiparams.html
http://icinga.example.com/cgi-bin/icinga/status.cgi?servicestatustypes=29&noheader=1

Enable external commands

sed -i 's|check_external_commands=0|check_external_commands=1|g' /etc/icinga/icinga.cfg
chmod 2710 /var/lib/icinga/rw
 
# fix permissions for external commands
dpkg-statoverride --update --add nagios www-data 2710 /var/lib/icinga/rw
chown nagios:nagios /var/lib/nagios /var/lib/icinga/ -R
chown nagios:www-data /var/lib/icinga/rw /var/cache/icinga/ -R

Firefox plugin
https://addons.mozilla.org/de/firefox/addon/imoin/?src=api
http://icinga.example.com/cgi-bin/icinga/status.cgi

API
https://www.hagen-bauer.de/2016/08/Icinga2-first-step-api.html
processing Icinga warning Json list

# get hosts with issues
curl -s -u "foo:bar" -k "https://monitoring.example.com/icinga/cgi-bin/status.cgi?servicestatustypes=20&noheader=1&servicestatustypes=29&sorttype=2&sortoption=3&scroll=963&jsonoutput"  | \
  jq --raw-output ' .status.service_status[] | select(.host_name | contains("dev")) | select(.status_information | contains("Apache", "Java")) | .host_name'

Debug

sed -i 's|debug_level=0|debug_level=16|g' /etc/icinga/icinga.cfg
tail -f /var/log/icinga/icinga.debug

Custom object / variables
https://docs.icinga.com/latest/de/customobjectvars.html

List by
http://monitoring.example.com/cgi-bin/icinga/status.cgi?servicegroup=foo
http://monitoring.example.com/cgi-bin/icinga/status.cgi?search_string=bar*
http://monitoring.example.com/cgi-bin/icinga/status.cgi?search_string=www.example.com&servicestatustypes=20

Fix Icinga1 Classic UI toggle menu bug
https://gist.github.com/panticz/638b8077e57147212a159b0fd4453d1e

Gearman

https://mod-gearman.org/download.html
apt install -y mod-gearman-worker
vi /etc/mod-gearman/worker.conf
/etc/init.d/gearman-job-server restart
/data/mod_gearman/bin/gearman_top

Configure icinga / gearmen / nrpe monitoring

apt install -y nagios-nrpe-server
cat /etc/nagios/nrpe.d/nrpe.cfg 
allowed_hosts=10.0.10.192
command[check_apt]=/usr/lib/nagios/plugins/check_apt

Links
http://www.icinga.org/
http://docs.icinga.org/latest/en/quickstart-icinga.html
http://exchange.nagios.org/directory/Plugins
http://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=nagios&searchon=names
https://addons.mozilla.org/de/firefox/addon/nagios-checker/
http://exchange.nagios.org/directory/Plugins/Hardware/Server-Hardware/IPMI-Sensor-Monitoring-Plugin/details
https://wiki.icinga.org/display/howtos/check_logfiles