Monday, April 21, 2014

WebSphere JVM Monitor with MBean and Custom Service

“There are times when fear is good. It must keep its watchful place at the heart's controls. There is advantage in the wisdom won from pain.” - Aeschylus quotes 


There is no single tool that WebSphere administrators can rely on for all their tasks. When I wrote about Java based JMX tool for deployment, some of you expressed your concerns on introducing yet another tool or language for day to day tasks.  Though Jacl and Jython covers 99% of our day to day operations, I believe there are certain benefits to embrace Java as an utility language. Sometimes, it can take you to places, where Jython or Jacl can't go :) One such use I found was, combining WebSphere Custom Service and JMX MBean for monitoring JVM startups and shutdown events.

IBM introduced something called Custom Services in WebSphere 4.0, then there came the JMX specification which IBM began supporting it from WAS V5. IBM WAS team did a great job of integrating both, which has been useful to this day.


MBeans

There are many ways to create JMX MBeans and so are many more useful ways that you could take advantage of them. Here are some of the links I found useful to get to know more about MBeans. If not, make use of your friend "google".

Links:
Creating and registering standard, dynamic, and open custom MBeans
Extending the WebSphere Application Server administrative system with custom MBeans


Custom Service

You can create one or more custom services for an application server. Each custom services defines a class that is loaded and initialized whenever the server starts and shuts down. Each of these classes must implement the com.ibm.websphere.runtime.CustomService interface. After you create a custom service, you can use the administrative console to configure that custom service for your application servers


JVM Monitor

If you are writing your own MBean, You would have to register your MBean with the MBean Server
that runs on each JVM and the MBean needs to be activated. I picked an easy route, implemented my own JMXAgent MBean with the custom service option, so that my MBean can be activated at the server startup every time.

Here is the code template for this MBean

public class MyJMXService implements CustomService, JMXManageable {
    public class Collab extends RuntimeCollaborator  {
         public String getIdentification() {return "MyJMXAgent"; }
         // add your own method that needs to be called from wsadmin or JMX

        };  // end collab class
    public void initialize(Properties argProps) throws Exception {
    }
     public void shutdown() throws Exception {
   }
}

Methods initialize() and shutdown() are implemented from CustomService interface and will be called by MBean server upon JVM startup and shutdown events. My goal is send out some email alerts on either events, also to activate my MBean in initialize() method.

If you wonder about the innerclass Collab, it's a runtime Collaborator class and it's methods are callback methods by the MBean Server. So that's the ideal place for adding methods that could be useful for administrative tasks, which could be invoked from either wsadmin or any JMX interface.

One such task that was essential with my development community was clearing JSP cache. Sometimes, some of my developers request us to clear JSP cache even when their apps don't use JSPs! They must be superstitious to think their problem will go away with a JSP cache clearing :) So I ended up implementing a method to clear the JSP temp folder for a given application, it is being called from our JMX Deployment tool by change management team from a remote server.

Word of caution, great power comes with great responsibilities, make sure you are not doing anything that sabotage your environment, implement a good validation before firing any tasks.

public class MyJMXService implements CustomService, JMXManageable {
    public class Collab extends RuntimeCollaborator  {
         public String getIdentification() {return "MyJMXAgent"; }
             public void clearFolder(String folder) throws Exception
             {
            String result="good";
              try {
                       if (folder.indexOf("/WebSphere/AppServer/") > 0 ) && (folder.indexOf("/temp/") > 0 )  {
                              System.out.println("Clearing.."+folder);   
                              clearFolder( (new File(folder)) );
                              System.out.println("Folder cleared"+folder);
                       }
                       else  {
                              System.out.println("MyJMXAgent:"+folder+" is not a JSP cache folder error");
                              result="MyJMXAgent:"+folder+" is not a JSP cache folder error";
                        }
                  }   
                 catch (Exception e) {
                     System.out.println("MyJMXAgent:Error in clearing JSP Cache");
                     result = "MyJMXAgent:Error in clearing JSP Cache";
                 }   
                if (!(result.equals("good") )) throw (new Exception(result));
        }

        public void clearFolder(File path)   throws Exception
        {
              try {
                File[] files = path.listFiles();
                   for(int i=0; i<files.length; ++i)
                   {
                          if(files[i].isDirectory())  clearFolder(files[i]);
                         files[i].delete();
                     }
               }
              catch (Exception e)  {  throw e;  }
        }
   };  // end collab class

    public void initialize(Properties argProps) throws Exception {
            String propURL = argProps.getProperty(PROP_KEY);
            try {
                JVMName = com.ibm.websphere.runtime.ServerName.getFullName();
            }
            catch (Exception e)  {
                System.out.println(label + "JMX init error: " + e.toString());
            }
            loadProperties(propURL);
         // Notify the admins on startup      
            sendMailHelper("Started");
        // Let's activiate the bean
            MBeanFactory  mbfactory = AdminServiceFactory.getMBeanFactory();
            RuntimeCollaborator collab = new Collab();
            mbfactory.activateMBean("MyJMXBean",collab,"MyJMXAgent1",getProperty("MyJMXAgent.descriptor"));
    }
    public void shutdown() throws Exception {
        String label = mainLabel + "shutdown: ";
            sendMailHelper("Shutdown");
    }
}

// Implement your own sendMailHelper and loadProperties

Descriptor

One last piece you would need is a MBean descriptor, a simple XML file that would be used by MBean Server to understand the MBean and to expose the appropriate methods.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE MBean SYSTEM "MbeanDescriptor.dtd">
<MBean type="MyJMXService">
    <attribute name="identification" type="java.lang.String" getMethod="getIdentification"/>
    <operation name="clearFolder" role="operation" type="void" targetObjectType="objectReference" impact="ACTION" description="Clears JSP Cache">
    <signature>
        <parameter name="folder" description ="folder name" type="java.lang.String" />
    </signature>
         </operation>
</MBean>

We are exposing only method clearFolder() to users with "operation" and above access roles.

Register

Now that we have our MBean, packed in a Jar, what do we do with it? How can it be registered within WebSphere? There is an easier way and there is a better way to do it.

Easier way would be to register it thru WebSphere console. If you go under AppServers --> Administration Services, you will see a link for adding Custom Services. Lot of my friends here forget to click on StartUp check box when adding a new service.


The important configuration values for Custom Services
  • Startup - very important; unless this box is checked, the Custom Service will not be initialized during server startup.
  • External Configuration URL - optional URL to a file that contains configuration data for this Custom Service. This data can be in any format you want, depending on the implementation of your Custom Service logic.
  • Classname (required) - Provides the fully-qualified package name and class name for the class that implements the CustomService interface. The server will load this name during startup and invoke this name to initialize.
  • Display Name (required) - Identifies the configuration of this service.
  • Description - Any text that is useful to distinguish this Custom Service from others.
  • Classpath (required) - The location of the JAR file containing the implementation code for this Custom Service.

The better way: Jython
   There is even a better way to do it than Admin Console, I know how much you all love Jython, here is a snippet that adds the custom service into all WebSphere processes, including dmgr and nodeagent.


    attrs = []
    attrs.append(["externalConfigURL","/opt/apps/webdata/config/WAS/MyJMX.properties"])
    attrs.append(["classname","MyJMXService"])
    attrs.append(["displayName", "MyJMXAgent"])
    attrs.append(["classpath", "/opt/apps/webdata/config/WAS/MyJMXAgent.jar"])
    attrs.append(["enable","true"])
    

    nodes = AdminConfig.list('Node').split(lineSeparator)
    servers = AdminTask.listServers('-serverType APPLICATION_SERVER ')
    servers = AdminUtilities.convertToList(servers)

    for aServer in servers:
            AdminConfig.create("CustomService", aServer, attrs)
    AdminConfig.save( )

This JMX Mbean could be something which will complement your existing monitoring solution, in case if you don't have any sophisticated solution in place, it could simply be configured in all your WebSphere builds and used as a reliable alert solution.

Sunday, April 6, 2014

JMX for WAS Admins

Give me six hours to chop down a tree and I will spend the first four sharpening the axe.-Abraham Lincoln

Part I: JMX for Change Management


It is my personal belief that JMX had been a under dog since from it's inception, for some reason it was never adopted by development and administration communities, with exception to some commercial products.Once you see how easy it is to get it working, with it’s limitless capabilities, you will be convinced to make it a default tooling solution for your application management and admin tasks.

 

 What is JMX and Mbeans?

Java Management Extensions is a Java Technology that provides tools for managing interface and performance of Java applications. If you are interested, read up on JMX and MBeans.

Without going into boring details, here are some of the benefits to use JMX
  • Platform neutral for any admin tasks
  • Works great remotely, using SOAP or RMI. Makes it possible to provide a centralized solution
  • All appserver admin tools, WAS consol and wsadmin use JMX internally, so all admin functions are exposed as JMX Mbeans (I like Mbeans, next to my coffee bean :)
  • Rich language Java strengthens the JMX capabiities
  • Relatively easy to use same JMX tool on different appserver products, such as JBOSS, WebLogic and Oracle Appserver (never tried it myself )
  • More so on why you need JMX  from Oracle site: JMX
  • You must have really been bored with Jacl and Jython already

What can you possibly to do with JMX?

        Deployment Utility: I developed a deployment utility a while back, which has been in use since from WebSphere 5,6,7 & V8 now, pretty much with no change except for updating jars. It enabled our QM department to be able to deploy web applications from a centralized server to number of different WebSphere appservers, which are on different platforms, ranging from Windows to ZOS with varying WebSphere versions.

        JMX Agent: Have your insider, agent running in each JVM. You can write a JMX bean that could be part of JVM life cycle, alerting you on all life cycle events. It could even take commands from you and execute on the appserver. One of the major Java exploit was using this JMX feature on client side Java 7, just watch out your implementation.

You can develop a JMX monitor which could poll different appserver for specific events or triggering points for alert.

If you are developing a web application, you can write Mbeans to assist you as an insider, perhaps health checks on your application's vital components. If you ever need to refresh you global static data, you can achieve it by calling your Mbean without having to restart the application.

It is relative easy to build JMX based tool, let us see what it takes to build one. As a first task, let’s connect to an appserver.


        Properties adminProps = new Properties();
        adminProps.setProperty(AdminClient.CONNECTOR_TYPE,  AdminClient.CONNECTOR_TYPE_SOAP);
        adminProps.setProperty(AdminClient.CONNECTOR_HOST,host);
        adminProps.setProperty(AdminClient.CONNECTOR_PORT, port);
    adminProps.setProperty(AdminClient.CONNECTOR_SECURITY_ENABLED, "true");
    adminProps.setProperty(AdminClient.USERNAME, SOAPUser);
     adminProps.setProperty(AdminClient.PASSWORD, SOAPPass);
     adminProps.setProperty("com.ibm.ssl.trustStore", keyLocation+"DummyServerTrustFile.jks");
        adminProps.setProperty("com.ibm.ssl.keyStore", keyLocation+"DummyServerKeyFile.jks");
        adminProps.setProperty("com.ibm.ssl.trustStorePassword", "WebAS");
        adminProps.setProperty("com.ibm.ssl.keyStorePassword", "WebAS");
      adminClient = AdminClientFactory.createAdminClient(adminProps);
        session=adminClient.isAlive();
       System.out.println("isAlive:"+session.toString());


As you noticed, we need few things in order to connect to an appserver, valid credentials and the public certificate from the target appserver (cell cert if it is a clustered environment), since most of the installations use self signed certificate with WAS Admin security enabled. If Admin security is not enabled, you don't need SSL connection. You would have to import the server certificate into DummyServerTrustFile.jks repository.

Now that you got a valid adminClient object, you can invoke command that can perform every task that you do on WAS console. Here is a snippet for stopping an application.

           String[] signature = new String[3];
           String[] params = new String[3];
           params[0]=applicationName;
           params[1]=null;
           params[2]=session.toString();
          signature[0] = "java.lang.String";
          signature[1] = "java.util.Hashtable";
          signature[2] = "java.lang.String";

          Set oSet = adminClient.queryNames(new ObjectName(oNameQuery.toString()), null);
          Iterator i = oSet.iterator ();
          ObjectName srv = null;
          while (i.hasNext()) {
           srv= (ObjectName) i.next();
           adminClient.invoke(srv,"stopApplication",params,signature);

** Keep in mind these commands are asynchronous, in order to know the actual status, you would have to implement a callback listener which would be invoked when Dmgr emits "completion" event.

Instead of "stopApplication", you can replace it with startApplication or exportApplication (needs different parameter). Installing a new application needs few more extra steps, as you would have to map modules to containers. Here is the extract doing exactly that.

   Hashtable props = new Hashtable();
   props.put (AppConstants.APPDEPL_LOCALE, Locale.getDefault());
   Hashtable opts = new Hashtable();
   Hashtable module2server = new Hashtable();
   AppDeploymentTask task =null;
   AppDeploymentController flowController = null;
    flowController =AppDeploymentController.readArchive  (localEAR, props); 
    task =flowController.getFirstTask();
   while (task != null) 
   {
         String[][] data = task.getTaskData();
          if (task.getName().equals("MapModulesToServers" )) 
          {
              for (int i=1; i < data.length; i++)
                   module2server.put ((data[i][1]).replace(',','+'), ("Websphere:cluster="+cluster));
          }
         task =flowController.getNextTask();
    }
   AppManagement appMgmt = AppManagementProxy. getJMXProxyForClient(adminClient);
   opts.put (AppConstants.APPDEPL_MODULE_TO_SERVER, module2server);
   opts.put (AppConstants.APPDEPL_PRECOMPILE_JSP,true);
   opts.put("cell.name",cellName);
   appMgmt.installApplication(installEAR,  opts, null);

Wrapping WAR into EAR

I personally don't like applications coming in War format, as it leaves room for admins to set wrong context root, but as many of you have seen, most vendor products come in not so handy War file format.This is where JMX can help you, you can create the Ear file by wrapping on vendor War module like below.

   opts.put (AppConstants.APPDEPL_LOCALE, Locale.getDefault());
   opts.put(AppConstants.APPDEPL_WEBMODULE_CONTEXTROOT, appContextRoot);
   opts.put(AppConstants.APPDEPL_APPNAME, actualName);
   opts.put(AppConstants.APPDEPL_MODULETYPE_WEB,actualName);
   opts.put(AppConstants.APPDEPL_MODULE, warModuleName);
   opts.put(AppConstants.APPDEPL_WEB_MODULE, warModuleName);
 
   AppManagementHelper.wrapModule(warLocation, earOutputLocation, warModuleName, opts);


If you are planning to use JMX targeting to manage many different applications, it would be a good idea to keep them in a property file so that the utility can pick up appropriate values for each application. In my next blog, let's explore on how JMX could be used as an insider agent for your JVMs or applications.