Zabbix and CloudWatch: mix them together the Grafana way.

Many times, when I am debugging an environment, I find myself opening a lot of web browser tabs and windows to get metrics and see graphs from a number of data sources. It is a tiresome and prone to error way to go as each metric can be presented in a different timezone, and there is no way to stack them together.

Grafana is an excellent solution to this problem as it can aggregate and mix many data sources and present them in a rich way. In this post I will show how to aggregate and mix metrics from Zabbix and AWS CloudWatch.

Installing Grafana

We will install Grafana 2.5.0 on an Ubuntu 14.04 server. Full installation documentation can be found here.

First, download and install the .deb package and its dependencies:

1 2 3

$ wget https://grafanarel.s3.amazonaws.com/builds/grafana_2.5.0_amd64.deb $ sudo apt-get install -y adduser libfontconfig $ sudo dpkg -i grafana_2.5.0_amd64.deb

Now, start the service:

1	$ sudo service grafana-server start

The Grafana UI is accessible at FQDN/IP Address:3000:

The default username is admin and the default password is also admin (and yes, you should change it!)

Adding Zabbix as a Data Source

Now that we have Grafana up and running we will need to add our Zabbix server as a data source. To do this, we will use grafana-zabbix. The original instruction can be found here (go to the Grafana 2.1.x and 2.5.x section).

First, lets download the grafana-zabbix tarball to some directory on the Grafana server and extract it:

1 2	$ wget https://github.com/alexanderzobnin/grafana-zabbix/archive/v2.5.0.tar.gz $ sudo tar xvzf v2.5.0.tar.gz

Second, lets copy the Zabbix directory from the extracted tarball to the Grafana data source plugin directory and restart the Grafana server:

1 2	$ sudo cp -R grafana-zabbix-2.5.0/zabbix/ /usr/share/grafana/public/app/plugins/datasource/ $ sudo service grafana-server restart

Now, lets add the Zabbix server as a data source. on the main UI page press on Data Sources (1) and then on the Add new tab (2):

At the Add data source section give a name to the data source (1) and select Zabbix at the Type drop-down list (2). At the Http settings set the url to the API end point of the Zabbix server, which is usually http://<zabbix-server>/zabbix/api_jsonrpc.php (3). At the Zabbix API details fill in the Zabbix User and its Password (this user must have permissions that will allow us access to the required information in the Zabbix server, 4). Enable the Trends option (5) and press on Add (6):

After you have pressed on the Add button, the Test Connection button will appear (1). Press on this button to test the connection. Finally, if the connection test was successful, press on Save (2):

Adding CloudWatch as a Data Source

on the main UI page press on Data Sources (1) and then on the Add new tab (2):

At the Add data source section give a name to the data source (1) and check the Default check box (2). Select CloudWatch at the Type drop-down list (3). At the Default Region, fill in the AWS default region (4) and press on ADD (5):

To access the CloudWatch data we need permissions to the relevant AWS APIs. There are two ways to accomplish that. The first way, which you can use both if you are running your Grafana server on an EC2 instance or on a machine outside AWS, is to use AWS access keys. How to generate and download the keys is beyond the scope of this blog, but here is a good place to start. After you have generated the keys, create  a directory named .aws in the home directory of the user running the Grafana server, and there create a file named credentials:

1 2	# mkdir ~/.aws # vim ~/.aws/credentials

To this file, paste the following (replacing the AWS access and secret access keys and region with your access keys and AWS region respectively):

1 2 3 4

[default] aws_access_key_id = <aws_access_key_id > aws_secret_access_key = <aws_secret_access_key> region = <region>

A second way to access the CloudWatch APIs, which will work only for EC2 instances, is to attache an IAM role to the instance (this is the best and most secure way).

Mixing the Data Sources

Now that we have both Zabbix and CloudWatch as data sources on our Grafana server we can start visualizing the data collected. Lets start by creating a new dashboard. On the main UI page press on the dashboard drop-down list (1) and in the popup window that appeared press on +New (2):

Lest give a title to the new dashboard. Press on the Manage Dashboard button (1) and than on Settings. In the General (2) tab enter a Title for the dashboard (3) and set the Timezone (4) and finally save the changes (5) :

Now, lets create our first graph. Press on the green row  button (1) and select Add Panel (2) and Graph (3):

Press on no title (click here) on the new graph that appeared (1) and select Edit (2):

In the Graph section select the General tab (1) and give a Title to the new graph (2):

Now lets get some metrics from Zabbix. Press on the Metrics tab (1) and select Zabbix-Server as the Datasource (2). In the Group select the group (3), in the Host select the host (4), In the Application select the Zabbix application you want (5) and finally select the Zabbix item you want to plot (6). I am plotting memory metrics:

To added another metric on the same graph press on “+ Query” and follow the same steps above.

To set the correct Y-axes units for the memory metric press on the Axes & Grid tab (1), and set the Unit of the Left Y (2) to data | bytes (3):

The final result should look similar to this:

As a last step, lets add another graph, but this time we will use CloudWatch as the data source. Add another graph by Pressing on the “+ ADD ROW” button and than on the green row button and select Add Panel and Graph. Follow the same steps presented above but this time select CloudWatch as the data source (1). Select the region you are using (2), the namespace (3), the metric (4) and the Stats (5). In the Dimensions press on “+” and select the CloudWatch dimension you want and its value (6). I have chosen the sum of the 5xx error code of some elastic load balancer (ELB):

The final result with both the Zabbix and CloudWatch metrics should look similar to this: