Posts Tagged ‘UltraESB’

Zabbix Monitoring via API

A few weeks ago I was asked to implement a feature for the Uconsole, to monitor “UltraESB” artifacts exposed to JMX connection. In today’s post I’ll be explaining the approach that I followed to achieve this task.

You can get an idea how the API works here.

The “item” is the basic element that should be configured to start Zabbix Monitoring. A graph is created with relevance to an item. Each item is uniquely identified by the name given and most importantly by its key. It describes the object to be monitored by the item.

This post will mainly focus on how to work with the zabbix api and how to interact with it. To get a further understanding, refer this link. It also describes how to specify the Zabbix item key for a JMX artifact.

Creating an item via the GUI is pretty easy, but the GUI is not very helpful always. Specially when several number of items and graphs need to be created at one shot. My requirement was the latter one.

Other third party JARs used

Apache Http Client
Jackson Java JSON-processor

The Design

If you went through the zabbix api it is made clear that first a user should get an authentication token to  register  items, graphs and triggers. So first we’ll get authorized.

public String authenticate(String username, String password, String url) {
    this.password = password;
    this.userName = username;
    zabbixApiUrl = url;

    StringBuilder uiConnectMessage = new StringBuilder();
    StringBuilder sb = new StringBuilder();
    sb.append("{\"jsonrpc\":\"2.0\"").
    append(",\"params\":{").
    append("\"user\":\"").append(username).
    append("\",\"password\":\"").append(password).
    append("\"},").
    append("\"method\":\"user.authenticate\",").
    append("\"id\":\"2\"}");

    try {
        HttpResponse response = postAndGet(sb.toString());
        HttpEntity entity = response.getEntity();

        HashMap untyped = mapper.readValue(EntityUtils.toString(entity), HashMap.class);
        auth = untyped.get("result");

        if (auth == null) {
            throw new IllegalArgumentException("Authorization failed to : " + url + ", using username : "               + username);
        }
        uiConnectMessage.append("Successfully connected to the server\n");

    } catch (IOException e) {
        uiConnectMessage.append("Could not connect to the Zabbix Server at : ").
        append(url).append(" Exception : ").append(e.getMessage()).append("\n");
    }
    return uiConnectMessage.toString();
}
    private HttpResponse postAndGet(String request) throws IOException {
        HttpClient client = new DefaultHttpClient();
        HttpPost httpPost = new HttpPost(zabbixApiUrl);
        httpPost.setEntity(new StringEntity(request));
        httpPost.addHeader("Content-Type", "application/json-rpc");
        return client.execute(httpPost);
    }

The above method will authenticate you to the zabbix server when the zabbix api url, user name and the password is given.
The received authentication token is stored in the “auth” varible which is declared globally for later uses.  Bellow is a screenshot of the web management console of the UltraESB which invokes the above methods


If you are successfully authenticated to the server, a list of hosts and a list of applications defined for each hosts are  presented. It is important to choose a host and a relevant application to register items. Defining an application is optional, but if you don’t latest data for an item will not be shown on the Zabbix GUI.

These two methods are used to retrieve hosts and applications defined to each hosts. When a host is selected from the drop down menu, the application drop down will show only the applications defined for the selected host.

    private Map<String, String> hostMap = new HashMap<String, String>();
    private Map<String, Map<String, String>> appMap = new HashMap<String, Map<String, String>>();

    private void getHosts() throws IOException {
        StringBuilder sb = new StringBuilder();
        sb.append("{\"jsonrpc\":\"2.0\",");
        sb.append("\"method\":\"host.get\",");
        sb.append("\"params\":{");
        sb.append("\"output\":\"extend\"},");
        sb.append(" \"auth\":\"").append(auth).append("\",");
        sb.append("  \"id\":2}");

        HttpResponse response = postAndGet(sb.toString());
        HttpEntity entity = response.getEntity();

        JsonNode rootNode = mapper.readValue(EntityUtils.toString(entity), JsonNode.class);
        JsonNode resultNode = rootNode.path("result");
        Iterator hostList = resultNode.getElements();
        while (hostList.hasNext()) {
            JsonNode element = hostList.next();
            hostMap.put(element.findValue("hostid").toString().replaceAll("(\\[|\\]|\")", ""), element.findValue("host")
                .toString().replaceAll("(\\[|\\]|\")", ""));
        }
    }

    private void getApps() throws IOException {
        StringBuilder sb = new StringBuilder();
        sb.append("{\"jsonrpc\":\"2.0\",");
        sb.append("\"method\":\"application.get\",");
        sb.append("\"params\":{");
        sb.append("\"output\":\"extend\"},");
        sb.append(" \"auth\":\"").append(auth).append("\",");
        sb.append("  \"id\":2}");

        HttpResponse response = postAndGet(sb.toString());
        HttpEntity entity = response.getEntity();

        JsonNode rootNode = mapper.readValue(EntityUtils.toString(entity), JsonNode.class);
        JsonNode resultNode = rootNode.path("result");
        Iterator appList = resultNode.getElements();
        while (appList.hasNext()) {
            JsonNode element = appList.next();
            JsonNode hosts = element.path("hosts");
            Iterator hostList = hosts.getElements();
            while (hostList.hasNext()) {
                JsonNode hostElement = hostList.next();
                String hostID = hostElement.findValue("hostid").toString().replaceAll("(\\[|\\]|\")", "");
                if (hostMap.containsKey(hostID)) {
                    if (appMap.containsKey(hostID)) {
                        appMap.get(hostElement.findValue("hostid").toString().replaceAll("(\\[|\\]|\")", "")).put(element.
                            findValue("name").toString().replaceAll("(\\[|\\]|\")", ""), element.findValue("applicationid").
                            toString().replaceAll("(\\[|\\]|\")", ""));

                    } else {
                        Map apps = new HashMap();
                        apps.put(element.findValue("name").toString().replaceAll("(\\[|\\]|\")", ""), element.
                            findValue("applicationid").toString().replaceAll("(\\[|\\]|\")", ""));
                        appMap.put(hostID, apps);
                    }
                }
            }
        }
    }

Finally when the user chooses the host and the application, the available artifacts are displayed. The user have the will of choosing which artifact is to be monitored.

Several items, graphs and triggers can be created for one type of artifact. For an example, some of the attributes exposed by the JMX connection of an Endpoint will be active address count, failed sending messages, ready address count etc. This program facilitates registration of many attributes for particular endpoint (not only endpoints other artifacts as well).

So how is this done?
I have defined a directory structure as shown in the image bellow

If a user needs to monitor a particular attribute, all what he have to do is to drop in json template file to the relevant folder, and the program will create items or graphs or triggers by reading the template files.
Following is a sample json file stored in the items directory of endpoints

NOTE : If a user checks an artifact and clicks the register button, the program will create items, graphs and triggers for all the json templates saved in the relevant artifact directory. (the Uconsole also provides with the facility of disabling a particular template).

{
"jsonrpc":"2.0",
"method":"item.create",
"params":{
      "description": "$filename$-$id$",
      "key_": "jmx[org.adroitlogic.ultraesb.detail:Type=Endpoints,Name=$id$][Details.activeAddressCount]",
      "hostid": "$hostid$",
      "value_type": 3 ,
      "data_type" : 0,
      "status": 0,
      "applications": ["$app$"]
},
"auth":"$auth$",
"id":2
}

The following method will list all the availble json files in specified artifact directory and loops through each file for manipulation.

     /**
     * Registers configuration elements at the zabbix server.
     *
     * @param category    - the type of the items
     * @param items       - name of the item to be registered
     * @param delExisting - the parameter to determine if the user need to delete the existing items
     * @return String - A String containing information about all unsuccessful operations
     */
public void registerConfigurationElements(String category, String[] items, boolean delExisting) {

    deleteExisting = delExisting;
    final String basePath = "conf" + File.separator + "zabbix" + File.separator;

    // Finding the path to Item templates
    File dirItemPath = new File(basePath + category + File.separator + "items");
    if (!dirItemPath.exists()) {
        throw new IllegalArgumentException("Item template directory for type : " + category + " does not exists");
    }

    File dirItems = new File(dirItemPath.getAbsolutePath());
    for (File file : dirItems.listFiles()) {
        int index = file.getName().lastIndexOf(".");
        String extension = file.getName().substring(index + 1);
        if ("json".equals(extension)) {
            processFile(file.getAbsolutePath(), items,
                file.getName().replaceAll(".json", ""), true);
        }
    }

    //find the path to Trigger templates
    File dirTriggerPath = new File(basePath + category + File.separator + "triggers");
    if (!dirTriggerPath.exists()) {
        throw new IllegalArgumentException("Trigger templates for type: " + category + " does not exists");
    }

    File dirTriggers = new File(dirTriggerPath.getAbsolutePath());
    for (File file : dirTriggers.listFiles()) {
        int index = file.getName().lastIndexOf(".");
        String extension = file.getName().substring(index + 1);
        if ("json".equals(extension)) {
            processFile(file.getAbsolutePath(), items,
                file.getName().replaceAll(".json", ""), false);
        }
     }

    // Finding the path to Graph templates
    File dirGraphPath = new File(basePath + category + File.separator + "graphs");
    if (!dirGraphPath.exists()) {
        throw new IllegalArgumentException("Graph templates directory for type : " + category + " does not exists");
    }

    File dirGraphs = new File(dirGraphPath.getAbsolutePath());
    for (File file : dirGraphs.listFiles()) {
        int index = file.getName().lastIndexOf(".");
        String extension = file.getName().substring(index + 1);
        if ("json".equals(extension)) {
            createGraph(file.getAbsolutePath(), items,
                file.getName().replaceAll(".json", ""));
        }
    }
}

If you carefully went through the json template above json template you would have probably noticed some variables wrapped in dollar ($) sign. These variable are replaced by the program.

Bellow is the createGraph method which is used for creating graphs at the zabbix server, but the method for creating items and triggers is not given as functionality is very much the same. Also I have omitted code lines which re-authenticate a if the zabbix session times out, solely for the purpose of maintaining the simplicity of the post.

    private void createGraph(String path, String items[], String fileName) {

        for (String item : items) {
            String request = readFile(path);
            request = request.replace("$filename$", fileName);
            request = request.replace("$id$", item);
            request = request.replace("$auth$", auth);

            try {
                HashMap requestNode = mapper.readValue(request, HashMap.class);
                ArrayList params = (ArrayList) requestNode.get("params");
                HashMap graphItems = (HashMap) params.get(0);
                ArrayList graphItemsList = (ArrayList) graphItems.get("gitems");
                HashMap itemId = (HashMap) graphItemsList.get(0);
                String itemIdList = (String) itemId.get("itemid");
                itemId.remove("itemid");
                itemId.put("itemid", map.get(itemIdList));
                String reqMssg = mapper.writeValueAsString(requestNode);
                postAndGet(reqMssg);

            } catch (EOFException e) {
                logger.warn("Empty json file at : {} - Exception : {}", path, e);
                uiRegisterElemMessage.append("Empty json file at ").append(path).append(" ").append(e).append("\n");

            } catch (IOException e) {
                logger.warn("Could not create graph for item : {} at path : {}", item, path);
                uiRegisterElemMessage.append("Could not create graph for item ").append(item).append(" at path ").
                    append(path).append("\n");
            }
        }
    }

As you can see working with Zabbix api is not difficult. The tedious part is sending requests to the server and processing received responses.

Running the Zabbix Server

The Zabbix server should be invoked by executing this command “/etc/init.d/zabbix-server start” before you start monitoring . You can check whether the Zabbix server is running from the front end GUI.

How ever if the GUI says “NO” have a look at the zabbix server log file at /var/log/zabbix-server/zabbix_server.log. If the log message some what similar to “Access denied for user ‘zabbix’@’localhost’ (using password: YES)”, there is a conflict between the zabbix database password and the passwords saved in following files.

/etc/zabbix/dbconfig.php

/etc/zabbix/zabbix_server.conf

Make sure that all three passwords are the same. The password for the zabbix database can reset by typing “set password for ‘zabbix’@’localhost’ = PASSWORD(‘yourPassword’); ” on the mysql prompt.

Advertisements

Log4J Custom Memory Appender

Hi all,

This post will describe how to write a custom appenders for Log4J, and about the Memory Appender, which I implemented.

Log4j is a project of Apache Software Foundation, which provides Java based logging Utility. It is quite efficient do logging with log4j, than having common System.out.println s all around your code.

Ok, lets get back to our custom appender.

Writing a custom appender for log4j is pretty easy. All what you have to do is to extend the “AppenderSkeleton” class in Log4j.


package org.adroitlogic.ultraesb.core.helper.memoryappender;

import org.adroitlogic.ultraesb.Util;
import org.adroitlogic.ultraesb.jmx.JMXConstants;
import org.adroitlogic.ultraesb.jmx.core.LogManagementMXBean;
import org.adroitlogic.ultraesb.jmx.view.LogEntry;
import org.apache.log4j.AppenderSkeleton;
import org.apache.log4j.helpers.LogLog;
import org.apache.log4j.spi.LoggingEvent;

import java.util.List;

/**
* InMemoryAppender appends log messages to a LogList, and implements methods in the MXBean interface
*/

public class InMemoryAppender extends AppenderSkeleton implements LogManagementMXBean {

private volatile long id = 0;
private final LogList logList = new LogList();

public synchronized void close() {
    if (this.closed) {
    return;
}
    this.closed = true;
}

public boolean requiresLayout() {
    return true;
}

protected boolean checkEntryConditions() {
    if (this.closed) {
        LogLog.warn("Not allowed to write to a closed appender.");
        return false;
    }

    if (this.layout == null) {
        errorHandler.error("No layout set for the appender named [" + name + "].");
        return false;
    }
    return true;
}

public void setSize(int size) {
    logList.setSize(size);
}

public void append(LoggingEvent event) {
    if (!checkEntryConditions()) {
       return;
    }
    subAppend(event);
}

protected void subAppend(LoggingEvent event) {
    int index = event.getLoggerName().lastIndexOf('.');
    String loggerName;

    if (index > -1) {
        loggerName = event.getLoggerName().substring(index + 1);
    } else {
        loggerName = event.getLoggerName();
    }

    LogEntry log = new LogEntry();
    log.setId(id++);
    log.setHost(event.getProperty("host"));
    log.setIp(event.getProperty("ip"));
    log.setLoggerName(loggerName);
    log.setMessage((String) event.getMessage());
    log.setThreadName(event.getThreadName());
    log.setTimeStamp(event.getTimeStamp());
    log.setLogLevel(event.getLevel().toString());
    logList.insert(log);
}

}

The method void close(), boolean requiresLayout() and void append(LoggingEvent event) are extended from  AppendercSkeleton class. Out of these, most important method is void append(LoggingEvent event) method. Each time a log message occurs, it is passed to this method as a LoggingEvent. Once  the LoggingEvent  is captured, you are almost done with writing the custom appender.

Then the LogginEvent is passed to  “subAppend(LogginEvent event)”. In there,  the “LoggingEvent” can be processed according to one’s requirements.

My requirement was to retrieve the tail end of UltraESB log messages and to display them on the web management console.

The following class defines the data structures I used and its insertion and retrieval methods.


package org.adroitlogic.ultraesb.core.helper.memoryappender;

import org.adroitlogic.ultraesb.jmx.view.LogEntry;

import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*;

public class LogList {

/**
* Specifies the number of log messages appended. Is incremented each time, a log is added to the linked list
*/
private int usedSize = 0;
/**
* Size of the hash map
*/
private int capacity;
private Map map;
private LinkedList linkedList = new LinkedList();
private static final DateFormat dfm = new SimpleDateFormat("yyyy-MM-dd' 'HH:mm:ss");

public LogList() {
}

public void setSize(int hashMapSize) {
    this.capacity = hashMapSize;
    map = new HashMap(hashMapSize);
}

/**
* Inserts log messages to the linked list and the hash map.
* If the number of log messages exceed the hashMapSize, new logs will be added after removing, the oldest log from the linked list and the hash map.
* @param log the log entries
*/
public synchronized void insert(LogEntry log) {
    if (usedSize > capacity) {
        map.remove(linkedList.removeFirst().getId());
        map.put(log.getId(), log);
        linkedList.add(log);
    } else {
        linkedList.add(log);
        map.put(log.getId(), log);
        usedSize++;
    }
}

public List formatFields(List list) {
int i = 0;
LogEntry entry;
String formattedString;

while (i < list.size()) {
    entry = list.get(i);
    if (entry.getFormattedTime() == null) {
        synchronized (dfm) {
        formattedString = dfm.format(entry.getTimeStamp());
        entry.setFormattedTime(formattedString);
        }
    }

    formattedString = entry.getMessage().replaceAll("( ){2}?", "  ");
    entry.setMessage(formattedString);
    i++;
}
return list;
}

/**
* When called returns the entire list of logs
* @return linkedList the list containing the logs
*/
public List getAll() {
    if (linkedList != null) {
        return formatFields(linkedList);
    } else {
    return Collections.emptyList();
    }
}

/**
* When called returns a sub list of the logs starting from the specified log id
* @param start starting id
* @return list the list containing the sub list
*/
public List getListFrom(long start) {
    if (map.get(start) != null) {
        List list = new ArrayList();
        long i = start;
        while (map.get(i) != null) {
            list.add(map.get(i));
            i++;
        }
        return formatFields(list);
    } else {
        return Collections.emptyList();
    }
}
}

I suppose that you guys noticed the “void setSize(int size)” method in “InMemoryappender” class. This is where I set the size of the LogList HashMap.
This method is directly called from the Log4j properties file and anyone can change the size of the hash map by editing this file. This is how it looks like.

log4j.rootCategory=ERROR, MEMORY_APPENDER

log4j.appender.MEMORY_APPENDER=org.adroitlogic.ultraesb.core.helper.memoryappender.InMemoryAppender
log4j.appender.MEMORY_APPENDER.Size=500
log4j.appender.MEMORY_APPENDER.layout=org.apache.log4j.PatternLayout

Now I’ll try to provide a more detailed view of the LogViewer. The “void insert(LogEntry log)” method is called by the subAppend method. Each log message is inserted to a LinkedList and a HashMap. The HashMap is used to retrieve a log message by its log id. An ESB can generate thousands of log messages, but users may be interested on reading the latest logs. So only the latest 500 (this number can be configured by the user as mentioned above) messages are kept in the memory and presented to the user.

JMX capabilities are used for the interactions between the console and the back end. Communication is done via JSON messages and “Apache Wink” is used to map Json to Java and vice versa.

When a user navigates to the LogViwer tab, the “List getAll()” method is invoked, displaying all existing log messages in the linked list. The console app checks for any latest log messages with an interval of 5 seconds. “List getListFrom(long start)” method is invoked for this purpose.

So that’s all about writing a custom appender for log4j. Hope it was useful…:)