The JBoss EAP / Wildfly provides a powerful concept for management, configuration and monitoring of the JBoss Application Server itself and its Java EE Applications. The concept is based on the detyped management API. All management clients of the application server use this detyped management API to interact with the server.

In this post we focus on some useful runtime metrics which are of interest when monitoring your application server and application with the Command Line Interface (CLI).

Monitoring the state of Application Server and Application

One basic and useful metric is the state of the application server. Via the state one can monitor whether the application server is available.

Connect the CLI to the application server with the following operations:

$ ./bin/jboss-cli.sh
[disconnected /] connect
[standalone@localhost:9990 /]

The next operation on the root resource of the server reads the attribute server-state:

[standalone@localhost:9990 /] :read-attribute(name=server-state)                
{
    "outcome" => "success",
    "result" => "running"
}

The server is in state running. That means the server is up and running without any outstanding configuration changes. Other states are starting, reload-required, restart-required, and stopping.

The state reload-required indicates that the server is running but a reload of all services is required in order to apply configuration changes.

The CLI operation :reload forces the server to shutdown and start again all services. The JVM itself will be not restarted.

The state restart-required means that a JVM restart is required to apply the configuration changes. A restart of the whole JVM process of the server can also executed with the CLI by the following command:

[standalone@localhost:9990 /] :shutdown(restart=true)

Runtime metrics of deployments are also available via the CLI. The current runtime status of a deployment can be retrieved like this:

[standalone@localhost:9990 /] /deployment=example.ear :read-attribute(name=status)
{
    "outcome" => "success",  
    "result" => "OK"
}

Possible status modes are OK, FAILED, and STOPPED. FAILED indicates a dependency is missing or a service could not start. STOPPED indicates that the deployment was not enabled or was manually stopped.

JVM Metrics

By default every JVM process expose some MBeans with information about the JVM and the environment. The MBeans with the ObjectName java.lang:type=* are also accessible via the CLI. The following CLI command is an example to get information about the memory usage.

[standalone@localhost:9990 /]/core-service=platform-mbean/type=memory :read-attribute(name=heap-memory-usage)                
{
    "outcome" => "success",
    "result" => {
        "init" => 67108864L,
        "used" => 58717032L,
        "committed" => 91910144L,
        "max" => 518979584L  
    }
}

Active HTTP Sessions of a Web-Application

An important metric to monitoring the current load of a web application is the number of active web sessions. This metric is available through the deployment and subsystem undertow or in case of JBoss EAP 6 through the subsystem web.

[standalone@localhost:9990 /] /deployment=example.ear/subdeployment=example-web.war/subsystem=undertow :read-attribute(name=active-sessions)
{
    "outcome" => "success",  
    "result" => 3
}

Metrics of the persistence layer

Additional useful metrics are metrics of the persistence layer of an application.

An important metric is the usage of the connection pool of a datasource. This metric and more are available through the subsystem datasource. The following CLI command shows an example of the pool usage of the ExampleDS:

[standalone@localhost:9990 /] /subsystem=datasources/data-source=ExampleDS/statistics=pool :read-resource(recursive=true, include-runtime=true)
{
    "outcome" => "success",
    "result" => {
        "ActiveCount" => "1",
        "AvailableCount" => "20",
        "AverageBlockingTime" => "1",
        "AverageCreationTime" => "60",
        "AverageGetTime" => "7",
        "BlockingFailureCount" => "0",
        "CreatedCount" => "2",
        "DestroyedCount" => "1",
        "IdleCount" => "1",
        "InUseCount" => "0",
        "MaxCreationTime" => "120",
        "MaxGetTime" => "124",
        "MaxUsedCount" => "1",
        "MaxWaitCount" => "0",
        "MaxWaitTime" => "1",
        "TimedOut" => "1",
        "TotalBlockingTime" => "2",
        "TotalCreationTime" => "121",
        "TotalGetTime" => "143",
        "WaitCount" => "0"
    }
}

In the case, that there are not enough connections available in the pool, the clients will have to wait for a free connection and will be rejected if a connection is not available within the timeout interval.

Metrics provided by the JPA Subsystem and Hibernate

Some further metrics are available through the deployment and the JPA Subsystem. These are metrics about the cache usage, such as the second level cache or the query cache, and some metrics about the entities and query execution.

However, these metrics must be activated through the following Hibernate properties within the persistence.xml, because it has an impact on the performance of the application.

<property name="hibernate.cache.infinispan.statistics" value="true"/>
<property name="hibernate.generate_statistics" value="true"/>

The example below shows the available metrics of the JPA layer. These metrics maybe useful for performance optimizations and for adjustment of the caches.

[standalone@localhost:9990 /] /deployment=web-application-example.ear/subsystem=jpa/hibernate-persistence-unit=web-application-example.ear#primary :read-resource(include-runtime=true, recursive=true)
{
    "outcome" => "success",
    "result" => {
        "close-statement-count" => 0L,
        "collection-fetch-count" => 1L,
        "collection-load-count" => 1L,
        "collection-recreated-count" => 17L,
        "collection-remove-count" => 1L,
        "collection-update-count" => 0L,
        "completed-transaction-count" => 254L,
        "connect-count" => 312L,
        "enabled" => true,
        "entity-delete-count" => 4L,
        "entity-fetch-count" => 2L,
        "entity-insert-count" => 55L,
        "entity-load-count" => 205L,
        "entity-update-count" => 4L,
        "flush-count" => 78L,
        "hibernate-persistence-unit" => "web-application-example.ear#primary",
        "optimistic-failure-count" => 0L,
        "prepared-statement-count" => 364L,
        "query-cache-hit-count" => 0L,
        "query-cache-miss-count" => 0L,
        "query-cache-put-count" => 0L,
        "query-execution-count" => 238L,
        "query-execution-max-time" => 5L,
        "query-execution-max-time-query-string" => "select generatedAlias0 from BlogEntry as generatedAlias0 order by generatedAlias0.created desc",
        "second-level-cache-hit-count" => 102L,
        "second-level-cache-miss-count" => 2L,
        "second-level-cache-put-count" => 72L,
        "session-close-count" => 255L,
        "session-open-count" => 255L,
        "statistics-enabled" => true,
        "successful-transaction-count" => 255L,
        ...

The CLI Command below shows the metrics of the second level cache from an entity:

[standalone@localhost:9990 /] /deployment=example.ear/subsystem=jpa/hibernate-persistence-unit=example.ear#primary/entity-cache=de.akquinet.jbosscc.BlogEntry :read-resource(include-runtime=true, recursive=true)
{
    "outcome" => "success",
    "result" => {
        "entity-cache-region-name" => "entity-cache-region-name",
        "second-level-cache-count-in-memory" => 5L,
        "second-level-cache-hit-count" => 31L,
        "second-level-cache-miss-count" => 0L,
        "second-level-cache-put-count" => 26L,
        "second-level-cache-size-in-memory" => -1L
    }
}

Runtime metrics of EJB Components

The EJB3 Subsystem allows the measurement of metrics, such as invocations and execution time. Because these metrics have some impact on the performance of the application, it must be activated in the EJB3 subsystem as shown below:

[standalone@localhost:9990 /] /subsystem=ejb3 :write-attribute(name=enable-statistics, value=true)

After the statistic measurement is enabled the runtime metrics of EJB components are available as shown in the example below:

[standalone@localhost:9990 /] /deployment=example.ear/subdeployment=example-ejb.jar/subsystem=ejb3/stateless-session-bean=BlogEntryDaoBean :read-resource(include-runtime=true, recursive=true)
{
    "outcome" => "success",
    "result" => {
        "component-class-name" => "BlogEntryDaoBean",
        "declared-roles" => [],
        "execution-time" => 364L,
        "invocations" => 16L,
        "methods" => {
            "persist" => {
                "execution-time" => 16L,
                "invocations" => 15L,
                "wait-time" => 0L
            },
            "findAll" => {
                "execution-time" => 348L,
                "invocations" => 1L,
                "wait-time" => 0L
            }
        },
        "peak-concurrent-invocations" => 1L,
        "pool-available-count" => 20,
        "pool-create-count" => 1,
        "pool-current-size" => 1,
        "pool-max-size" => 20,
        "pool-name" => "slsb-strict-max-pool",
        "pool-remove-count" => 0,
        "run-as-role" => undefined,
        "security-domain" => "other",
        "timers" => [],
        "wait-time" => 0L
    }
}

The most useful metrics are the usage of the bean instance pool. By default the pool configuration of the EJB3 Subsystem will be used.

The EJB3 Subsystem provides in addition some metrics of the thread pool usage for EJB timers, asynchronous and remote EJB invocation.

[standalone@localhost:9990 /] /subsystem=ejb3/thread-pool=default :read-resource(include-runtime=true, recursive=true)
{
    "outcome" => "success",
    "result" => {
        "active-count" => 0,
        "completed-task-count" => 0L,
        "current-thread-count" => 0,
        "keepalive-time" => {
            "time" => 100L,
            "unit" => "MILLISECONDS"
        },
        "largest-thread-count" => 0,
        "max-threads" => 10,
        "name" => "default",
        "queue-size" => 0,
        "rejected-count" => 0,
        "task-count" => 0L,
        "thread-factory" => undefined
    }
}

If you are not so familiar with the CLI syntax, the CLI provides a GUI. The GUI of the CLI can be started with the following command:

$ ./jboss-cli.sh --gui

In Addition the following website provides an overview of all available addresses and operations: http://wildscribe.github.io

Any questions or feedback? If so, feel free to comment on this post or contact me via email: heinz.wilming@akquinet.de