JarCastingStream-JMX
A Lightweight framework to stream and monitor JMX metrics. (Formerly known as PingJMX)
Stream JMX metrics to:
- Central monitoring server
- File, socket, log4j
- User defined destination
These metrics can be used to monitor health, performance and availability of your JVMs and applications. Use Stream-JMX to embed a monitoring agent within your application and monitor memory, GC activity, CPU as well as user defined MBeans.
Here is what you can do with Stream-JMX:
- Periodic JVM heartbeat
- Monitor memory utilization, GC activity, memory leaks
- High/Low, normal vs. abnormal CPU usage
- Monitor threading, runtime and other JVM performance metrics
- Monitor standard and custom MBean attributes
- Conditional actions based on MBean attribute values
- Conditional streaming based on custom filters
- Application state dumps on VM shut-down for diagnostics
Why Stream-JMX
Stream-JMX provides and easy, lightweight and secure way to stream and monitor JMX metrics from within java runtime containers.
- Stream JMX metrics out of the JVM container (vs. polling from outside/remote)
- Makes it easy to monitor farms of JVMs, application servers
- Reduce cyber security risk: No need to enable remote JMX, SSL, security, ports, firewalls
- Integration with monitoring tools for alerting, pro-active monitoring (AutoPilot M6)
- Integration with cloud analytics tools (https://www.jkoolcloud.com via JESL)
- Integration with log4j, slf4j, jKoolCloud (via TNT4J event sinks)
- Embedded application state dump framework for diagnostics
- Easily build your own extensions, monitors
NOTE: JESL provides a way to stream events generated by Stream-JMX to jKoolCloud. For more information on JESL visit: JESL project on Github.
Using Stream-JMX
It is simple, do one of the following:
- run Stream-JMX as a
-javaagent - attach Stream-JMX as agent to running JVM
- connect Stream-JMX over JMXConnector to locally running JVM or remote JMX service
- embed Stream-JMX code into your application
NOTE: Running Stream-JMX as -javaagent, attaching agent to running JVM or connecting over JMXConnector to locally running JVM or remote JMX service over RMI connection can be invoked without changing your application code.
NOTE: When you want to sample JMX inter-process communication way (providing -connect or -attach mode -vm parameter value as PID/process name), then Stream-JMX runner and sampled process VMs MUST be:
- built by same vendor (e.g.
IBM,Oracle,OpenJDK) - of same architecture (
x86,x64) Stream-JMXrunner VM version higher or equal compared to sampled VMs version - e.g. trying to sample JVM 10+ using JVM 8 may produce exceptions like this:
java.io.IOException: Non-numeric value found - int expected
at sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:299)
at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:63)
at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:79)
at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:103)
at com.sun.tools.attach.VirtualMachine.loadAgent(VirtualMachine.java:540)
- check which JDK
tools.jaris referred overLIBPATHvariable, it also must match runner JVM
If JVM vendors, architecture or versions does not match conditions above - then only way to collect JMX samples is using -connect mode defining JMXConnector (over RMI) URL.
NOTE: when running stream-jmx sometimes it may appear some Unsupported type exceptions in console/log, e.g.:
Failed to sample:
ojbName=java.lang:type=MemoryPool,name=PS Eden Space,
info=javax.management.MBeanAttributeInfo[description=UsageThresholdExceeded, name=UsageThresholdExceeded, type=boolean, read-only, isIs, descriptor={openType=javax.management.openmbean.SimpleType(name=java.lang.Boolean), originalType=boolean}],
exclude=false,
ex=javax.management.RuntimeMBeanException: java.lang.UnsupportedOperationException: Usage threshold is not supported
javax.management.RuntimeMBeanException: java.lang.UnsupportedOperationException: Usage threshold is not supported
... 25 more
Caused by: java.lang.UnsupportedOperationException: Usage threshold is not supported
... 27 more
It is OK, since some MBeans do not provide some attributes depending on Operating System or other run environment setup. If you would run JConsole you'll get values Unavailable for such attributes painted in red (e.g. see java.lang -> MemoryPool -> PS Eden Space attribute UsageThresholdExceeded). These Unsupported exceptions are shown on first iteration of sampling for user to know that all MBean attributes where accessed.
Common command line arguments
Arguments definition:
-vm:- virtual machine descriptor. It can bePIDorJVM process name fragment.*value is wildcard to pick all found VMs, running on local machine.-ao:- agent options string using!symbol as delimiter. Options format:mbean-filter!exclude-filter!sample-ms!init-delay-msmbean-filter- MBean include name filter defined using object name pattern:domainName:keysSet.exclude-filter- MBean exclude name filter defined using object name pattern:domainName:keysSet.sample-ms- MBeans sampling rate in milliseconds.init-delay-ms- MBeans sampling initial delay in milliseconds. Optional, by default it is equal tosample-msvalue.
-cp:- JMX connection parameter string using=symbol as delimiter. Defines only one parameter, to define more than one use this argument multiple times. Argument format:-cp:key=value. See Java SE monitoring and management documentation for more details.-slp:- sampler parameter string using=symbol as delimiter. Defines only one parameter, to define more than one use this argument multiple times. Argument format:-slp:key=valuetrace- flag indicating whether the sample listener should print trace entries to print stream. Default value -false.forceObjectName- flag indicating to forcibly addobjectNameattribute if such is not present for a MBean. Default value -false.compositeDelimiter- delimiter used to tokenize composite/tabular type MBean properties keys. Default value -\.useObjectNameProperties- flag indicating to copy MBean ObjectName contained properties into sample snapshot properties. Default value -true.excludeOnError- flag indicating to auto-exclude failed to sample attributes. Default value -false.excludedAttributes- list of user chosen attribute names (may have wildcards*and?) to exclude, pattern:attr1,attr2,...,attrN@MBean1_ObjectName;...;attr1,attr2,...,attrN@MBeanN_ObjectName.
-sp:- sampler system property string using=symbol as delimiter. Defines only one system property, to define more than one use this argument multiple times. Argument format:-sp:key=value.-ssl- flag indicating to disable SSL verification in case SSL is misconfigured: expired, self-signed, or domain mismatching certificate.
Running Stream-JMX as -javaagent
Command line to run
Executable OS shell run script files bin/stream-jmx.bat or bin/stream-jmx.sh are dedicated to do the job.
Command line to run stream-jmx as JVM agent looks like this:
java -javaagent:tnt4j-stream-jmx-core-all.jar="*:*!30000" -Dtnt4j.config=tnt4j.properties -Dlog4j.configuration=file:log4j.properties -classpath "tnt4j-stream-jmx.jar;lib/*" your.class.name your-args
The options are -javaagent:tnt4j-stream-jmx-core-all.jar="mbean-filter!sample-time-ms", classpath must include tnt4j-stream-jmx jar files as well as locations of log4j and tnt4j configuration files. See JMX Sampling Agent sampler options for details.
Attaching Stream-JMX to running JVM
Command line to run
Executable OS shell run script files bin/stream-jmx-attach.bat or bin/stream-jmx-attach.sh are dedicated to do the job.
Command line to attach local JVM process JMX looks like this:
java -Dtnt4j.config=.\config\tnt4j.properties -Dlog4j.configuration=file:.\config\log4j.properties -classpath "tnt4j-stream-jmx-core-all.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -attach -vm:activemq -ap:tnt4j-stream-jmx-core-all.jar -ao:*:*!10000
System properties -Dxxxxx defines Stream-JMX configuration. For details see Stream-JMX configuration .
SamplingAgent arguments -attach -vm:activemq -ap:tnt4j-stream-jmx-core-all.jar -ao:*:*!10000 states:
-attach- defines thatSamplingAgentshall be attached to running JVM process-vm:activemq- is JVM descriptor. In this case it is running JVM name fragmentactivemq. But it also may be JVM process identifier - PID. Mandatory argument.-ap:tnt4j-stream-jmx-core-all.jar- is agent library name. If it is class path - then only name should be sufficient. In any other case define full or relative path, e.g.,..\build\tnt4j-stream-jmx\tnt4j-stream-jmx-0.13-SNAPSHOT\tnt4j-stream-jmx-core-all.jar. Mandatory argument.-ao:*:*!10000- is JMX sampler options stating to include all MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is*:*!30000. Initial sampler delay can be configured by adding numeric parameter*:*!30000!1000defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.-slp:- any JMX sampler configuration property. See Program arguments used for details.-sp:- any system property used by sampler. See System properties used for details.
NOTE: arguments and properties defined running SamplingAgent.main is forwarded to SamplingAgent agent attached to JVM process.
NOTE: if you get exception com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize while running stream-jmx in -attach mode it is most likely that some stream-jmx used classes are not resolved in attach JVM class path. Pay attention to use tnt4j-stream-jmx-core-[VERSION]-all.jar as agent lib to have all dependent classes available within one jar. If you are trying to attach application implementing J2EE API, use tnt4j-stream-jmx-j2ee-[VERSION]-all.jar. Same approach should be followed when attaching WAS and Liberty instances - use -all.jar for matching product API.
Coding into API
You can attach SamplingAgent to JVM from your custom API by calling SamplingAgent.attach(String,String,String) method.
Sample attach call:
try {
SamplingAgent.attach("activemq", "tnt4j-stream-jmx-core-all.jar", "*:*!10000");
} catch (Exception exc) {
exc.printStackTrace();
}
Connecting Stream-JMX to local or remote JMX service
Command line to run
Executable OS shell run script files bin/stream-jmx-conenct.bat or bin/stream-jmx-conenct.sh are dedicated to do the job:
stream-jmx-connect script has 4 parameters:
- process id or service URI (required)
- agent options for MBeans include/exclude filter and sampling interval (optional,
.value sets default) - service identifier for the process/service being monitored (optional,
.value sets default) - sampling agent arguments (optional,
.value sets default)
- MS Windows
rem using URL
/bin/stream-jmx-connect.bat service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi
rem using URL with connection parameters
/bin/stream-jmx-connect.bat service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi . . -ul:admin -up:admin -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory
rem using process name part
/bin/stream-jmx-connect.bat activemq
rem using pid
/bin/stream-jmx-connect.bat 1553
- *nix
# using URL
./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi
# using URL with connection parameters
./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi . . -ul:admin -up:admin -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory
# using process name part
./bin/stream-jmx-connect.sh activemq
# using pid
./bin/stream-jmx-connect.sh 1553
To connect to local JVM process
Command line to connect local JVM process JMX looks like this:
java -Dtnt4j.config=.\config\tnt4j.properties -Dlog4j.configuration=file:.\config\log4j.properties -classpath "tnt4j-stream-jmx-core.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -connect -vm:activemq -ao:*:*!*:dummy!10000
System properties -Dxxxxx defines Stream-JMX configuration. For details see Stream-JMX configuration .
SamplingAgent arguments -connect -vm:activemq -ao:*:*!*:dummy!10000 states:
-connect- defines thatSamplingAgentshall connect to running JVM process over JMXConnector (RMI) connection.-vm:activemq- is JVM descriptor. In this case it is running JVM name fragmentactivemq. But it also may be JVM process identifier - PID. Mandatory argument.-ao:*:*!*:dummy!10000- is JMX sampler options stating to include all MBeans, exclude alldummyMBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is*:*!30000. Initial sampler delay can be configured by adding numeric parameter*:*!30000!1000defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.-slp:- any JMX sampler configuration property. See Program arguments used for details.-sp:- any system property used by sampler. See System properties used for details.
To connect to JMX service over URL
Command line to connect remote JMX service looks like this:
java -Dtnt4j.config=.\config\tnt4j.properties -Dlog4j.configuration=file:.\config\log4j.properties -classpath "tnt4j-stream-jmx-core.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -connect -vm:service:jmx:<JMX_URL> -ul:admin -up:admin -ao:*:*!!10000 -cri:30 -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory
System properties -Dxxxxx defines Stream-JMX configuration. For details see Stream-JMX configuration .
SamplingAgent arguments -connect -vm:service:jmx:<JMX_URL> -ul:admin -up:admin -ao:*:*!!10000 -cri:30 -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory states:
-connect- defines thatSamplingAgentshall connect to running JMX service over JMXConnector (RMI) connection.-vm:service:jmx:<JMX_URL>- is JMX service URL to use for connection. Mandatory argument. Full URL may be likeservice:jmx:rmi://127.0.0.1/stub/rO0ABXN9AAAAAQAlamF2YXgubWFuYWdlbWVudC5yZW1vdGUucm1pLlJNSVNlcnZlcnhyABdqYXZhLmxhbmcucmVmbGVjdC5Qcm94eeEn2iDMEEPLAgABTAABaHQAJUxqYXZhL2xhbmcvcmVmbGVjdC9JbnZvY2F0aW9uSGFuZGxlcjt4cHNyAC1qYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN0SW52b2NhdGlvbkhhbmRsZXIAAAAAAAAAAgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc2AAtVbmljYXN0UmVmMgAACzE3Mi4xNi42Ljg2AADPWKO5DJD/bZIhG9aBuwAAAVo8DdAkgAEAeA==.-ul:admin- is user login. In this case it isadmin. User login argument is optional.-up:admin- is user password. In this case it isadmin. User password argument is optional.-ao:*:*!!10000- is JMX sampler options stating to include all MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is*:*!30000. Initial sampler delay can be configured by adding numeric parameter*:*!30000!1000defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.-cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory- is JMX connector parameters definitions in properties formatkey=value. JMX connector parameters are optional and can be defined multiple times - as many as there are required JMX connector parameters. See Java API Context class documentation for available properties naming. NOTE: If you are using some API extending JNDI, check documentation if it provides some additional connection configuration properties.-cri:30- is connection retry interval in seconds. In this case it is30secbetween connect retry attempts. Connection retry interval is optional - default value is10sec. Special values are:0indicates no delay between repeating connect attempts.-1indicates no repeating connect attempts shall be made at all and application has to stop on first failed attempt to connect.
-slp:- any JMX sampler configuration property. See Program arguments used for details.-sp:- any system property used by sampler. See System properties used for details.
NOTE:
- URI of remote RMI service (e.g., to connect remote Kafka service) may require additional
/chars:
-vm:service:jmx:rmi:///jndi/rmi:///[HOST]:[PORT]/jmxrmi
e.g.:
-vm:service:jmx:rmi:///jndi/rmi:///172.16.6.35:2181/jmxrmi
- Remote end may require to enable JMX listening port in service configuration to accept connections (e.g., to connect remote Kafka service):
export JMX_PORT=${JMX_PORT:-9999}
- Tomcat and Kafka does not provide J2EE implementation, thus you need only stream-jmx
corejar inclasspathwhen sampling Tomcat/Kafka metrics over JMX. Executable OS shell run script files uses onlycoreasMODULE_SETvariable value:bin/stream-jmx-conenct.bat
set MODULE_SET=core
bin/stream-jmx-conenct.sh
MODULE_SET=("core") - See Enabling Tomcat JMX Remote to enable remote JMX access of Tomcat server instance. If your Tomcat server is behind firewall - also refer to JMX Remote Lifecycle Listener on how to setup such Tomcat server instance to be accessible from outside.
- See ActiveMQ JMX how to enable remote ActiveMQ JMX access.
Multiple VMs monitoring over single stream run
Multiple local JVM processes monitoring
If sampling agent finds multiple VM's matching provided JVM descriptor, it will bind and sample JMX beans from all these VM's, e.g.:
./bin/stream-jmx-connect.sh kafka
sampler will collect JMX beans for all local machine running VM's containing kafka token within VM process name.
Multiple VM's monitoring configuration using external file
Dedicated SamplingAgent arguments:
-f:- defines path for external configuration file, containing set of VM connection definitions.
You can configure multiple VM's connections using external configuration file. Sample configuration file is ./config/connections.cfg. The syntax for that file is:
##############################################################################################################################################################
# VM connection string # Agent options # User name # Password # Source addition #
##############################################################################################################################################################
service:jmx:rmi:///jndi/rmi://localhost:9995/jmxrmi *:*!!60000 admin admin SERVICE=@bean:org.apache.ZooKeeperService:name0=*/?ClientPort#SERVER=@bean:java.lang:type=Runtime/?Name
service:jmx:rmi:///jndi/rmi://localhost:9996/jmxrmi *:*!!60000 admin admin SERVICE=@bean:java.lang:type=Runtime/?Name#SERVER=@bean:kafka.server:id=?,type=app-info#DATACENTER=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value
service:jmx:rmi:///jndi/rmi://localhost:9997/jmxrmi *:*!!60000 admin admin SERVICE=@bean:java.lang:type=Runtime/?Name#SERVER=@bean:kafka.server:type=app-info,id=?#DATACENTER=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value
##############################################################################################################################################################
To run JMX samples streaming for multiple VM's define in external configuration file use stream-jmx-connect-file-config.bat/.sh files, e.g.:
*nix
./bin/stream-jmx-connect-file-config.sh ./config/connections.cfg
MS Windows
\bin\stream-jmx-connect-file-config.bat \config\connections.cfg
Connections configuration using stanza divided properties file
When connections configuration gets too complex to be defined using tokenized lines, there is possibility to define connections using properties divided into stanzas. One stanza defines one VM descriptor (connection) properties, like this:
##############################################################################################################################################################
{
zk.vm: service:jmx:rmi:///jndi/rmi://localhost:9995/jmxrmi
zk.vm.user: admin
zk.vm.pass: admin
zk.vm.reconnect.sec: 10
zk.agent.options: java.lang:*!!60000
zk.source.fqn: SERVICE=@bean:org.apache.ZooKeeperService:name0=*/?ClientPort#SERVER=@bean:java.lang:type=Runtime/?Name
}
##############################################################################################################################################################
{
kafka.vm: kafka:zk://127.0.0.1:2181
kafka.vm.user: admin
kafka.vm.pass: admin
kafka.vm.reconnect.sec: 10
kafka.agent.options: java.lang:*!!60000
kafka.source.fqn: SERVICE=@bean:java.lang:type=Runtime/?Name#SERVER=@bean:kafka.server:id=?,type=app-info#DATACENTER=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value
}
##############################################################################################################################################################
{
solr.vm: solr:zk://172.16.6.208:2181
solr.vm.user: admin
solr.vm.pass: admin
solr.vm.reconnect.sec: 10
solr.agent.options: java.lang:*!!5000
solr.source.fqn: SERVER=@bean:solr:dom1=core,dom2=?,dom3=*,reporter=*,category=*,scope=core,name=*
solr.other.urlPattern: service:jmx:rmi:///jndi/rmi://{0}:{1,number,######}/jmxrmi
solr.other.port: 18983
}
##############################################################################################################################################################
There are three VM descriptors defined - zk, kafka and solr. As you see first property key token (delimited using .) means VM descriptor name. You can name it freely, like solrGroup1, myZooKeeeper or kafkaCluster1. Rest is used to define particular VM descriptor properties:
vmorvm.url- defines VM or (VM orchestrator instance) URL.vm.user- defines VM connection used user name.vm.passorvm.password- defines VM connection used user password.vm.reconnect.sec- defines period in seconds to reconnect if VM connection gets interrupted. Negative value means do not try to reconnect.agent.options- defines agent sampling options.source.fqn- defines TNT4J Source FQN values mapping used by VM descriptor.other- defines group of additional custom VM descriptor properties:urlPattern- defines JMX service URL pattern to be filled in by ZooKeeper orchestrated VMs resolver. Default value -service:jmx:rmi:///jndi/rmi://{0}:{1,number,######}/jmxrmi, where{0}is replaced with JMX service runner machine host name/IP and{1}is replaced with JMX service runner machine port number.port- defines JMX service port number to be used by ZooKeeper orchestrated VMs resolver. Some services (like Apache Solr) does not provide JMX service port to ZooKeeper, so it is needed to define it manually. Apache Kafka for instance - does provide JMX connection port, so there is no need to define it additionally - it is picked automatically from ZooKeeper node provided data.
Multiple VM's monitoring notes for Source
See TNT4J Source fields configuration as base Source fields configuration reference.
Multiple VM's monitoring requires additional data Source configuration when you want to distinguish where JMX data came from.
When all monitored VM's can uniquely identify itself using same SourceFQN by using dynamic MBean property value (common when monitoring same service on different machines), e.g.
source.factory.SERVICE: @bean:org.apache.activemq:type=Broker,brokerName=localhost/?BrokerId
source.factory.SERVER: @bean:java.lang:type=Runtime/?Name
source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?
then no additional source configuration is required.
But when your services ecosystem is widely distributed running different services on different machines (or multiple different services on same machine), every service identifies itself over different MBean property. In this case you can configure source RootFQN in tnt4j.properties and define additional SourceFQN path fragment for every monitored VM. This way final VM MBeans sample will have SourceFQN made by Stream-JMX by concatenating tnt4j.properties defined RootFQN with that additional VM SourceFQN path fragment.
For example, when tnt4j.properties defines RootFQN as:
source.factory.DATACENTER: HQDC
source.factory.SERVER: @bean:java.lang:type=Runtime/?Name
source.factory.RootFQN: SERVER=?#DATACENTER=?
and having VMs configured this way:
##############################################################################################################################################################
# VM connection string # Agent options # User name # Password # Source addition #
##############################################################################################################################################################
service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi *:*!!60000 admin admin SERVICE=@bean:org.apache.activemq:type=Broker,brokerName=localhost/?BrokerId
service:jmx:rmi:///jndi/rmi://192.168.1.1:9998/jmxrmi *:*!!30000 admin admin SERVICE=@bean:kafka.server:id=?,type=app-info
##############################################################################################################################################################
may build such SourceFQNs for VMs:
localhost- `SERVICE=ID:PC-NAME-52295-1538060220413-0:1#SERVER=2453@PC-NAME#DATACENTER=HQDC192.168.1.1- `SERVICE=3SERVER=2457@boxName#DATACENTER=HQDC
NOTE: VM additional SourceFQN path must be defined fallowing same rules as in TNT4J config - SourceType1=value1#SourceType2=value2#...#SourceTypeN=valueN.
ZooKeeper orchestrated VMs access
Stream-JMX has ability to resolve JMX service connections for VMs orchestrated by ZooKeeper. In this case you'll need:
- Use
tnt4j-stream-jmx-zkmodule dependency (for development) or use built jar of this module in your JVM classpath (for deployment). Also seebin/stream-jmx-connect.bat(.sh) files for commented out variableMODULE_SETdefinition combinations. - Use predefined VM descriptor prefixes to initiate VMs JMX connection resolution over running ZooKeeper service instance:
kafka:zk://[ip]:[port]- forApache KafkaVMs access. Stream-JMX will monitor/broker/idsnode for registered brokers and will connect to all JMX enabled (exposing JMX port not equal-1) brokers.solr:zk://[ip]:[port]- forApache SolrVMs access. Stream-JMX will monitor/live_nodesnode for registered Solr node instances. NOTE: Solr does not expose JMX port, so you need to specify JMX port number asOther optionscolumn valueport=[port number].
NOTE: ZooKeeper service instance JMX sampling does not start automatically defining VM descriptors for Apache Kafka or Apache Solr. In case you want to sample JMX of referred ZooKeeper instance, you have to define ZooKeeper runner VM dedicated JMX service URL.
NOTE: When Stream-JMX binds to ZooKeeper node, it monitors for changes under that node. So if new brokers gets registered/unregistered, Stream-JMX connects/disconnects JMX services of these brokers on runtime.
Sample configuration:
############################################################################################################################################################################################################################################
# VM connection string # Agent options # User name # Password # Source addition # Other options #
############################################################################################################################################################################################################################################
service:jmx:rmi:///jndi/rmi://localhost:9995/jmxrmi *:*!!60000 admin admin SERVICE=@bean:org.apache.ZooKeeperService:name0=*/?ClientPort#SERVER=@bean:java.lang:type=Runtime/?Name
kafka:zk://127.0.0.1:2181 *:*!!60000 . . SERVICE=@bean:java.lang:type=Runtime/?Name#SERVER=@bean:kafka.server:id=?,type=app-info#DATACENTER=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value
solr:zk://172.16.6.208:2181 java.lang:*!!5000 . . SERVER=@bean:solr:dom1=core,dom2=?,dom3=*,reporter=*,category=*,scope=core,name=* port=18983
############################################################################################################################################################################################################################################
Having first line we define that we want to sample JMX of ZooKeeper VM where Kafka brokers registers.
Second line we initiate Stream-JMX use ZooKeeper instance to collect JMX service connections for all registered Kafka brokers.
In third line we initiate Stream-JMX use ZooKeeper instance to collect JMX service connections for all registered Solar node instances. Note, we also define Solr JMX connection port there - port=18983.
NOTE: if some configuration token needs to be omitted (like some user names or passwords in our configuration), you can use . symbol as configuration token placeholder.
How to migrate Source definitions from single to multi VM sampling
When you where sampling JMX running as many instances of Stream-JMX as there where monitored JVMs, you where defining complete SourceFQN within source.factory.RootFQN property of Stream-JMX TNT4J configuration.
When monitoring multiple JVMs running single instance of Stream-JMX, source.factory.RootFQN property should define only part of SourceFQN common for all monitored JVMs. Remaining (JVM unique) part of SourceFQN shall be defined in external JVM connections configuration file (e.g. config/connections.cfg). But this also implies that if SourceFQN tokens refers unique values resolved from JMX data (e.g, over @bean: reference), it is enough to have only source.factory.RootFQN property of Stream-JMX TNT4J configuration to uniquely distinguish JVMs.
Consider this Stream-JMX run scenario to monitor 3 instances of service runner VMs:
- One Stream-JMX instance per monitored VM
- you've been having such TNT4J source configuration:
source.factory.SERVICE: $sjmx.serviceId source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?- running 3 instances of Stream-JMX
./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi -ul:admin -up:admin -sp:sjmx.serviceId=Broker-0 ./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9998/jmxrmi -ul:admin -up:admin -sp:sjmx.serviceId=Broker-1 ./bin/stream-jmx-connect.sh service:jmx:rmi:///jndi/rmi://localhost:9997/jmxrmi -ul:admin -up:admin -sp:sjmx.serviceId=Broker-2
- One Stream-JMX instance per 3 monitored VMs
- you've been having such TNT4J source configuration:
source.factory.RootFQN: SERVER=?#DATACENTER=?
- configuring VM connections in
connections.cfgfile
############################################################################################################################################################## # VM connection string # Agent options # User name # Password # Source addition # ############################################################################################################################################################## service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi *:*!!60000 admin admin SERVICE=Broker-0 service:jmx:rmi:///jndi/rmi://localhost:9998/jmxrmi *:*!!60000 admin admin SERVICE=Broker-1 service:jmx:rmi:///jndi/rmi://localhost:9997/jmxrmi *:*!!60000 admin admin SERVICE=Broker-2 ##############################################################################################################################################################- running Stream-JMX
./bin/stream-jmx-connect-file-config.sh connections.cfg
Connecting remote WebSphere Application Server (WAS)
Additions needed to run SamplingAgent connected to remote WAS machine can be found in executable OS shell run script files bin/stream-jmx-conenct-was.bat or bin/stream-jmx-conenct-was.sh. It contains these major configuration additions:
- WAS environment setup configuration
- appending
LIBPATHvariable with WAS libs - adding WAS specific JMX sampler options
- defining WAS
JMXConnectorparameters - and adding them as additional
javacommand parameters
NOTE: When you have IBM JVM running on client side and getting authentication or naming related exceptions, you have to:
- Alter
sas.client.propsby settingcom.ibm.CORBA.validateBasicAuth=false - Add additional
SamplingAgentarguments:
-ul:[your WAS user]
-up:[your WAS user pass]
-cp:java.naming.factory.initial=com.ibm.websphere.naming.WsnInitialContextFactory
-cp:java.naming.factory.url.pkgs=com.ibm.ws.naming
-cp:java.naming.provider.url=corbaloc:iiop:localhost:2809/WsnAdminNameService
Connecting remote WebLogic (version 12c) server instance
Additions needed to run SamplingAgent connected to remote WebLogic machine can be found in executable OS shell run script files bin/stream-jmx-conenct.bat or bin/stream-jmx-conenct.sh. It contains these major configuration additions:
-
WebLogic environment setup configuration in
setDomainEnv.sh/setDomainEnv.cmdfile, exposing JMX access port and security options like:- .cmd
set JAVA_OPTIONS=%JAVA_OPTIONS% -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8788 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
- .sh
JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8788 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" export JAVA_OPTIONS
-
appending
LIBPATHvariable with WebLogic client libs from${WL_INSTALL_DIR}/wlserver/server/lib/:wlclient.jarwljmxclient.jarjavax.javaee-api.jar
For example:
- .bat
set WL_HOME=C:\Oracle\Middleware\Oracle_Home set WL_CLINET_LIBS=%WL_HOME%\wlserver\server\lib\wlclient.jar;%WL_HOME%\wlserver\server\lib\wljmxclient.jar;%WL_HOME%\wlserver\server\lib\javax.javaee-api.jar set LIBPATH=%LIBPATH%;%RUNDIR%..\lib\*;%WL_CLINET_LIBS%
- .sh
WL_HOME="/opt/Oracle/Middleware/Oracle_Home" WL_CLINET_LIBS="$WL_HOME/wlserver/server/lib/wlclient.jar:$WL_HOME/wlserver/server/lib/wljmxclient.jar:$WL_HOME/wlserver/server/lib/javax.javaee-api.jar" LIBPATH="$LIBPATH:$SCRIPTPATH/../lib/*:$WL_CLINET_LIBS"
-
enable
j2eemodule use:- .bat
set MODULE_SET=core j2ee
- .sh
MODULE_SET=("core" "j2ee")
-
Add additional
SamplingAgentarguments:
-cp:jmx.remote.protocol.provider.pkgs=weblogic.management.remote
# Add user credentials if needed
-cp:java.naming.security.principal=user
-cp:java.naming.security.credentials=password
# Optional naming parameters
#-cp:java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory
#-cp:java.naming.provider.url=t3://localhost:7001
WebLogic 12c provides these JNDI Names for WebLogic MBean Servers:
weblogic.management.mbeanservers.domainruntime- Domain Runtime MBean Serverweblogic.management.mbeanservers.runtime- Runtime MBean Serverweblogic.management.mbeanservers.edit- Edit MBean Server
You can find your WebLogic server exposed JMX Connectors in console output:
<May 10, 2019, 10:57:11,147 AM EEST> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.runtime.>
<May 10, 2019, 10:57:11,823 AM EEST> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.domainruntime.>
<May 10, 2019, 10:57:11,853 AM EEST> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.edit.>
JMX Service connection URL for WebLogic is like this:
service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.runtime
And complete set of stream-jmx program arguments:
-connect
-vm:service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.edit
-ao:*:*!!10000
-cp:jmx.remote.protocol.provider.pkgs=weblogic.management.remote
-cp:java.naming.security.principal=weblogic
-cp:java.naming.security.credentials=password
For more information, see Programming WebLogic JNDI, WebLogic API Environment class documentation and Accessing WebLogic Server MBeans with JMX.
Coding into API
You can connect SamplingAgent to JVM from your custom API by calling SamplingAgent.connect(String,String) method.
Sample connect call for local JVM:
try {
SamplingAgent.newSamplingAgent().connect("activemq", "*:*!!10000");
} catch (Exception exc) {
exc.printStackTrace();
}
Sample connect call for remote JVM:
try {
SamplingAgent.newSamplingAgent().connect("service:jmx:iiop://172.16.6.40:2809/jndi/JMXConnector", "*:*!!10000");
} catch (Exception exc) {
exc.printStackTrace();
}
Sample connect call for remote JVM defining user name and password:
try {
SamplingAgent.newSamplingAgent().connect("activemq", "admin", "admin", "*:*!!10000");
} catch (Exception exc) {
exc.printStackTrace();
}
or defining map of connection configuration parameters:
try {
Map<String, Object> connParams = new HashMap<String, Object>();
connParams.put("javax.net.ssl.trustStore", "/your/path/to/truststore.jks");
connParams.put("javax.net.ssl.trustStorePassword", "truststore_pwd");
SamplingAgent.newSamplingAgent().connect("activemq", "*:*!!10000", connParams);
} catch (Exception exc) {
exc.printStackTrace();
}
Sampling local process runner JVM
Command line to run
java -Dtnt4j.config=.\config\tnt4j.properties -Dlog4j.configuration=file:.\config\log4j.properties -classpath "tnt4j-stream-jmx-core.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -local -ao:*:*!*:dummy!10000
System properties -Dxxxxx defines Stream-JMX configuration. For details see Stream-JMX configuration .
SamplingAgent arguments -connect -vm:activemq -ao:*:*!*:dummy!10000 states:
-local- defines thatSamplingAgentshall run sampling on local process runner JVM.-ao:*:*!*:dummy!10000- is JMX sampler options stating to include all MBeans, exclude alldummyMBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is*:*!30000. Initial sampler delay can be configured by adding numeric parameter*:*!30000!1000defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.-slp:- any JMX sampler configuration property. See Program arguments used for details.-sp:- any system property used by sampler. See System properties used for details.
Coding into API
You can run SamplingAgent for local process runner JVM from your custom API by calling SamplingAgent.sampleLocalVM(String,boolean) method.
Sample sampleLocalVM call for a local process runner JVM:
try {
SamplingAgent.sampleLocalVM("*:*!!10000", true);
} catch (Exception exc) {
exc.printStackTrace();
}
Running Stream-JMX as servlet
Generic servlet page features
Top of the page is dedicated to list Stream-JMX used properties.
Change properties as you like in input field and press Sumbit button to apply changes. JMX sampler service will restart and changes will apply instantly without restating servlet or web container.
tnt4j.properties file is displayed bellow. You can change that online. Press Submit TNT4.config to update configuration. JMX sampler service will restart.
Bellow you can see Stream-JMX console output. Refresh page or click Update button to update console contents.
WebSphere Application Server (WAS)
Build Stream-JMX and find tnt4j-stream-jmx-was-*.ear in distribution build directory or archives produced.
Login to WebSphere Integrated Solutions Console and install TNT4J-Stream-JMX ear you've built:
- Navigate Applications->New application and choose New Enterprise Application. Choose file you have built and install Stream-JMX application you usually do, or fallowing IBM's instruction. Keep in mind context path.
- Start the application.
By default Stream-JMX connects to local WAS runner JVM using com.ibm.websphere.management.AdminServiceFactory.getMBeanFactory().
Navigate to http://localhost:9080/YOUR_CHOOSEN_CONTEXT_PATH/. TNT4J-Stream-JMX configuration/monitoring page should be displayed. See Generic servlet page features on what is available there.
Stream-JMX logging on WAS can be performed by WAS Admin console:
Troubleshooting->[your server instance]->Change log detail levels-> selectConfiguration/Runtimetab ->Components and Groups->Components-> expandAll components-> expandcom.jkoolcloud.tnt4j.*-> selectcom.jkoolcloud.tnt4j.stream.jmx*node -> scroll to the tree bottom and select level you would like to use.Troubleshooting->[your server instance]->Change log detail levels-> selectConfiguration/Runtimetab -> into text area paste: com.jkoolcloud.tnt4j.stream.jmx.*=finer(changefinerto the level you would like to use).
See Generic servlet page features.
WebSphere Liberty Server
Build Stream-JMX and find tnt4j-stream-jmx-liberty-war-*.war in distribution build directory or archives produced.
There is a simple Liberty server.xml configuration required to run Stream-JMX servlet:
<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">
<!-- Enable features -->
<featureManager>
<feature>webProfile-7.0</feature>
<feature>restConnector-1.0</feature>
<feature>localConnector-1.0</feature>
<feature>monitor-1.0</feature>
<feature>j2eeManagement-1.1</feature>
</featureManager>
<!-- To access this server from a remote client add a host attribute to the following element, e.g. host="*" -->
<httpEndpoint id="defaultHttpEndpoint"
httpPort="9080"
httpsPort="9443"
host="*"/>
<!-- Automatically expand WAR files and EAR files -->
<applicationManager autoExpand="true"/>
<webContainer deferServletLoad="false"/>
<auth-method>BASIC</auth-method>
<basicRegistry id="basic" realm="default">
<user name="Admin" password="admin" />
<user name="JMXManager" password="jmxAdmin" />
</basicRegistry>
<administrator-role>
<user>Admin</user>
</administrator-role>
<!-- Configure Stream-JMX logging level, e.g. to get DEBUG entries -->
<logging traceSpecification="*=info:com.jkoolcloud.tnt4j.stream.jmx.*=finer" />
<application contextRoot="tnt-jmx" location="tnt4j-stream-jmx-liberty-war-0.13-SNAPSHOT.war" type="war" id="tnt-jmx" name="tnt-jmx">
<application-bnd>
<security-role name="StreamJmxManager">
<user name="JMXManager" />
<run-as userid="JMXManager" />
</security-role>
</application-bnd>
</application>
</server>
By default Stream-JMX connects to local Liberty runner JVM using java.lang.management.ManagementFactory.getPlatformMBeanServer().
Navigate to http://localhost:9080/YOUR_CHOOSEN_CONTEXT_PATH/. TNT4J-Stream-JMX configuration/monitoring page should be displayed. See Generic servlet page features on what is available there.
Embed Stream-JMX code into your application
Coding
// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).run();
NOTE: setSchedule(..).run() sequence must be called to run the schedule. setSchedule(..) just sets the scheduling parameters, run() executes the schedule.
To schedule metric collection for a specific MBean server:
// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance(ManagementFactory.getPlatformMBeanServer());
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).run();
Stream-JMX supports inclusion and exclusion filters. To schedule metric collection for a specific MBean server and exclude certain MBeans: (Exclusion filters are applied after inclusion filters)
// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance(ManagementFactory.getPlatformMBeanServer());
String excludeMBeanFilter = "mydomain:*";
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, excludeMBeanFilter, 30000).run();
Below is an example of how to sample all registered mbean servers:
// obtain SamplerFactory instance
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// find other registered mbean servers
ArrayList<MBeanServer> mlist = MBeanServerFactory.findMBeanServer(null);
for (MBeanServer server: mlist) {
Sampler jmxp = factory.newInstance(server);
jmxp.setSchedule(Sampler.JMX_FILTER_ALL, 30000).run();
}
Alternatively, Stream-JMX provides a helper class SamplingAgent that lets you schedule sampling for all registered MBeanServer instances.
SamplingAgent.newSamplingAgent().sample(Sampler.JMX_FILTER_ALL, Sampler.JMX_FILTER_NONE, 60000, TimeUnit.MILLISECONDS);
NOTE: Sampled MBean attributes and associated values are stored in a collection of Snapshot objects stored within Activity instance. Current Activity instance can be obtained via AttributeSample passed when calling listeners such as AttributeCondition, SampleListener. Snapshots can be accessed using Activity.getSnapshots() method call.
NOTE: Sampled output is written to underlying tnt4j event sink configured in tnt4j.properties file. Sink destinations could be a file, socket, log4j, user defined event sink implementations.
For more information on TNT4J and tnt4j.properties see TNT4J Wiki.
Command line to run
Example below runs SamplingAgent helper class as a standalone java application with a given MBean filter "*:*", sampling period in milliseconds (10000), and time to run in milliseconds (60000):
java -Dlog4j.configuration=file:.\config\log4j.properties -classpath "*;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent "*:*" "" 10000 60000
Stream-JMX configuration
Stream-JMX has configuration properties allowing to configure JMX sampler. It is possible to configure same stream JMX sampler parameter using System property or program argument. Depending on sampling environment used, in some cases it is easier to configure it using one approach or another. When both definitions available first System property defined value is assigned and then program argument value after.
JMX sampler configuration properties are:
forceObjectName- flag indicating to forcibly addobjectNameattribute if such is not present for a MBean. Default value -false.compositeDelimiter- delimiter used to tokenize composite/tabular type MBean properties keys. Default value -\;useObjectNameProperties- flag indicating to copy MBeanObjectNamecontained properties into sample snapshot properties. Default value -true.excludeOnError- flag indicating to auto-exclude failed to sample attributes. Default value -false.excludedAttributes- list of user chosen attribute names (may have wildcards*and?) to exclude, pattern:attr1,attr2,...,attrN@MBean1_ObjectName;...;attr1,attr2,...,attrN@MBeanN_ObjectName. Default value - ``.
See System properties used how to configure Stream-JMX using system properties. See Program arguments used how to configure Stream-JMX using program arguments.
System properties used
To define system property for application you can use common JVM argument -Dkey=value or SamplingAgent program argument -sp:key=value.
General use:
tnt4j.config- defines TNT4J properties file path. Example:-Dtnt4j.config=".\config\tnt4j.properties"log4j.configuration- definesstream-jmxlogging used LOG4J properties file path. Example:-Dlog4j.configuration="file:.\config\log4j.properties"com.jkoolcloud.tnt4j.stream.jmx.agent.forceObjectName- defines whether to forcibly addobjectNameattribute if such is not present for a MBean. Default value -false. Example:-Dcom.jkoolcloud.tnt4j.stream.jmx.agent.forceObjectName=truecom.jkoolcloud.tnt4j.stream.jmx.agent.compositeDelimiter- defines delimiter used to tokenize composite/tabular type MBean properties keys. Default value -\. Example:-Dcom.jkoolcloud.tnt4j.stream.jmx.agent.compositeDelimiter=.com.jkoolcloud.tnt4j.stream.jmx.agent.useObjectNameProperties- defines whether to copy MBeanObjectNamecontained properties into sample snapshot properties. Default value -true. Example:-Dcom.jkoolcloud.tnt4j.stream.jmx.agent.useObjectNameProperties=falsesjmx.serviceId- definesstream-jmxservice identifier used by TNT4JSourceFQNto distinguish monitored application instance. Example:-Dsjmx.serviceId=broker-0or-sp:sjmx.serviceId=broker-0com.jkoolcloud.tnt4j.stream.jmx.agent.excludeOnError- defines whether to auto-exclude failed to sample attributes. Default value -false. Example:-Dcom.jkoolcloud.tnt4j.stream.jmx.agent.excludeOnError=truecom.jkoolcloud.tnt4j.stream.jmx.agent.excludedAttributes- defines list of user chosen attribute names (may have wildcards*and?) to exclude, pattern:attr1,attr2,...,attrN@MBean1_ObjectName;...;attr1,attr2,...,attrN@MBeanN_ObjectName. Default value - ``. Example:-Dcom.jkoolcloud.tnt4j.stream.jmx.agent.excludedAttributes=javaVersion,javaVendor@WebSphere:mbeanIdentifier=cells*,*com.jkoolcloud.tnt4j.stream.jmx.sampler.factory- defines class name ofSamplerFactoryclass to be used by stream. Default value -com.jkoolcloud.tnt4j.stream.jmx.factory.DefaultSamplerFactory. Example:-Dcom.jkoolcloud.tnt4j.stream.jmx.sampler.factory=com.jkoolcloud.tnt4j.stream.jmx.impl.WASSamplerFactorytnt4j.stream.log.filename- defines name of stream log file. Default value -./logs/tnt4j-stream-jmx.log. Example:-Dtnt4j.stream.log.filename=./logs/tnt4j-stream-jmx_broker0.logtnt4j.activities.log.filename- defines name of streamed activities log file. Default value -./logs/tnt4j-stream-jmx_samples.log. Example:-Dtnt4j.activities.log.filename=./logs/tnt4j-stream-jmx_broker0_samples.log
Changes between versions:
- Prior to version
0.7stream-jmxwas writing logging messages toSystem.out/errprint streams. Since version0.7logging is performed overTNT4JLog Sink toslf4j-log4j12. Logger configuration is defined in./config/log4j.propertiesfile. If you where using system property-Dcom.jkoolcloud.tnt4j.stream.jmx.agent.trace=trueprior to0.7version, this is now configured over logger log level, by setting it toDEBUGvalue, e.g:
log4j.logger.com.jkoolcloud.tnt4j.stream.jmx=DEBUG
NOTE: when running multiple parallel instances of stream-jmx (e.g. collecting Kafka broker metrics using dedicated stream instance), it is recommended to define different log file names (using system properties tnt4j.stream.log.filename and tnt4j.activities.log.filename) for every stream-jmx ran JVM. Using same log file on different VMs may arise logging conflicts when VM ran logger holds different last log line references, resulting log entries to be written in random order. When monitoring multiple VM's over single stream-jmx instance, only one logger instance is ran (as there is one JVM running), and no logging conflicts shall occur.
Program arguments used
To define stream JMX sampler configuration property use program argument-slp:. One argument defines one property. To define multiple properties use as many argument definitions as there are required properties. For example:
-slp:forceObjectName=true
-slp:compositeDelimiter=.
-slp:useObjectNameProperties=false
-slp:excludeOnError=true
-slp:excludedAttributes=java*@WebSphere:mbeanIdentifier=cells*,*
JMX Sampling Agent sampler options
Agent options are defined using format: mbean-filter!exclude-filter!sample-ms!init-delay-ms
mbean-filter- MBean include name filter defined using object name pattern:domainName:keysSetexclude-filter- MBean exclude name filter defined using object name pattern:domainName:keysSetsample-ms- MBeans sampling rate in millisecondsinit-delay-ms- MBeans sampling initial delay in milliseconds. Optional, by default it is equal tosample-msvalue
Default sampling agent options value is: *:*!!30000
TNT4J Source fields configuration
Stream-JMS has couple additional features in comparison with basic TNT4J when building Source RootFQN value:
-
Has two dedicated value placeholders
sjmx.serverAddressandsjmx.serverNameto resolve remotely sampled machine IP address and host name. To map these sampled machine IP/host name values intosourcefield, put@symbol before placeholder name insourcefield definition. In this casetnt4j.propertiesconfiguration would be like this:; Remote machine IP address ... source.factory.DATACENTER: HQDC source.factory.SERVICE: broker-01 source.factory.SERVER: @sjmx.serverAddress source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=? ... ; Remote machine host name ... source.factory.DATACENTER: HQDC source.factory.SERVICE: broker-01 source.factory.SERVER: @sjmx.serverName source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=? ...
-
Can resolve values for
sourcefields from JMX MBean attributes. Two items are required to use intnt4j.propertiesconfiguration to enable this feature:- Set
source.factorytocom.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl - Put MBean attribute descriptor having prefix
@bean:assourcefield value. There are two types of MBean attribute descriptors:- Mbean attribute value descriptor pattern is
"@bean:MBean_ObjectName/?AttributeName. It also allows use of wildcard symbols*, e.g.:@bean:org.apache.ZooKeeperService:name0=*/?ClientPortresolvingClientPortvalue for first available (if more than one is running on same JVM, but usually it should be only one) ZooKeeper service instance. - MBean key property descriptor pattern is
@bean:MBean_ObjectName:PropertyKey=?(,OtherProperties)(part within()is optional), e.g.:@bean:kafka.server:id=?,type=app-info.
- Mbean attribute value descriptor pattern is
then putting all together,
tnt4j.propertiesconfiguration will be like this:... source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl source.factory.DATACENTER: HQDC source.factory.SERVICE: @bean:org.apache.activemq:type=Broker,brokerName=localhost/?BrokerId source.factory.SERVER: @bean:JMImplementation:type=MBeanServerDelegate/?MBeanServerId source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=? ...
- Set
Stream-JMX event data formatters
-
FactNameValueFormatter- this class provides key/value formatting for tnt4j activities, events and snapshots. The output format follows the following format:"OBJ:name-value-prefix1\name1=value1,name-value-prefix1\name2=value2,....,name-value-prefixN\nameN=valueN"Sample output:
OBJ:Streams\0, 0\HQDC\samtis\com.jkoolcloud.tnt4j.stream.jmx.impl.PlatformJmxSampler\Activities, Self\location=0, 0, Self\level=INFO, Self\id.count=0, Self\pid=7660, Self\tid=63, Self\snap.count=34, Self\elapsed.usec=9989365, JMImplementation:type\MBeanServerDelegate\MBeanServerId=samtis_1486645461070, JMImplementation:type\MBeanServerDelegate\SpecificationName=Java Management Extensions, JMImplementation:type\MBeanServerDelegate\SpecificationVersion=1.4, JMImplementation:type\MBeanServerDelegate\SpecificationVendor=Oracle Corporation, JMImplementation:type\MBeanServerDelegate\ImplementationName=JMX, JMImplementation:type\MBeanServerDelegate\ImplementationVersion=1.8.0_121-b13, JMImplementation:type\MBeanServerDelegate\ImplementationVendor=Oracle Corporation, org.apache.activemq:brokerName\localhost!type\Broker\Uptime=1 minute, org.apache.activemq:brokerName\localhost!type\Broker\MemoryPercentUsage=0, org.apache.activemq:brokerName\localhost!type\Broker\Persistent=true, org.apache.activemq:brokerName\localhost!type\Broker\DataDirectory=D:\tmp\apache-activemq-5.14.3\data, org.apache.activemq:brokerName\localhost!type\Broker\VMURL=vm://localhost, org.apache.activemq:brokerName\localhost!type\Broker\TempLimit=42844577792, org.apache.activemq:brokerName\localhost!type\Broker\MemoryLimit=668309914, org.apache.activemq:brokerName\localhost!type\Broker\StoreLimit=42846849471, org.apache.activemq:brokerName\localhost!type\Broker\MaxMessageSize=1024, org.apache.activemq:brokerName\localhost!type\Broker\AverageMessageSize=1024, org.apache.activemq:brokerName\localhost!type\Broker\TotalDequeueCount=0, org.apache.activemq:brokerName\localhost!type\Broker\TotalConsumerCount=0, org.apache.activemq:brokerName\localhost!type\Broker\StatisticsEnabled=true, org.apache.activemq:brokerName\localhost!type\Broker\JobSchedulerStoreLimit=0, org.apache.activemq:brokerName\localhost!type\Broker\MinMessageSize=1024, org.apache.activemq:brokerName\localhost!type\Broker\JobSchedulerStorePercentUsage=0, org.apache.activemq:brokerName\localhost!type\Broker\StorePercentUsage=0, org.apache.activemq:brokerName\localhost!type\Broker\TempPercentUsage=0, org.apache.activemq:brokerName\localhost!type\Broker\CurrentConnectionsCount=0, org.apache.activemq:brokerName\localhost!type\Broker\TotalMessageCount=0, org.apache.activemq:brokerName\localhost!type\Broker\TotalConnectionsCount=0, org.apache.activemq:brokerName\localhost!type\Broker\TotalEnqueueCount=1, org.apache.activemq:brokerName\localhost!type\Broker\TotalProducerCount=0, org.apache.activemq:brokerName\localhost!type\Broker\TransportConnectors=[openwire=tcp://samtis:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600, amqp=amqp://samtis:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600, mqtt=mqtt://samtis:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600, stomp=stomp://samtis:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600, ws=ws://samtis:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600], org.apache.activemq:brokerName\localhost!type\Broker\BrokerVersion=5.14.3, org.apache.activemq:brokerName\localhost!type\Broker\UptimeMillis=62787, org.apache.activemq:brokerName\localhost!type\Broker\BrokerName=localhost, org.apache.activemq:brokerName\localhost!type\Broker\Slave=false, org.apache.activemq:brokerName\localhost!type\Broker\BrokerId=ID:samtis-52769-1486645462150-0:1, SampleContext\noop.count=0, SampleContext\sample.count=6, SampleContext\total.error.count=6, SampleContext\total.exclude.count=204, SampleContext\mbean.count=37, SampleContext\condition.count=0, SampleContext\listener.count=1, SampleContext\total.action.count=0, SampleContext\total.metric.count=3582, SampleContext\last.metric.count=595, SampleContext\sample.time.usec=10054, SampleContext\listener.exclude.set.count=6, SampleContext\listener.trace.mode=false, ...NOTE: Entries are not sorted, sequence is same as returned by activity contained snapshots map entries and snapshot properties iterators.
-
FactPathValueFormatter- this class provides key/value formatting for tnt4j activities, events and snapshots. The output format follows the following format:"OBJ:object-path1\name1=value1,object-path1\name2=value2,....,object-pathN\nameN=valueN""Sample output:
OBJ:Streams\0, 0\HQDC\samtis\com.jkoolcloud.tnt4j.stream.jmx.impl.PlatformJmxSampler\Activities, Self\location=0, 0, Self\level=INFO, Self\id.count=0, Self\pid=3136, Self\tid=24, Self\snap.count=34, Self\elapsed.usec=9885208, JMImplementation\MBeanServerDelegate\ImplementationName=JMX, JMImplementation\MBeanServerDelegate\ImplementationVendor=Oracle Corporation, JMImplementation\MBeanServerDelegate\ImplementationVersion=1.8.0_121-b13, JMImplementation\MBeanServerDelegate\MBeanServerId=samtis_1486998422874, JMImplementation\MBeanServerDelegate\SpecificationName=Java Management Extensions, JMImplementation\MBeanServerDelegate\SpecificationVendor=Oracle Corporation, JMImplementation\MBeanServerDelegate\SpecificationVersion=1.4, SampleContext\condition.count=0, SampleContext\last.metric.count=633, SampleContext\listener.count=1, SampleContext\listener.exclude.set.count=6, SampleContext\listener.trace.mode=true, SampleContext\mbean.count=37, SampleContext\noop.count=0, SampleContext\sample.count=2, SampleContext\sample.time.usec=72856, SampleContext\total.action.count=0, SampleContext\total.error.count=6, SampleContext\total.exclude.count=66, SampleContext\total.metric.count=1272, org.apache.activemq\Broker\localhost\AverageMessageSize=1024, org.apache.activemq\Broker\localhost\BrokerId=ID:samtis-63754-1486998424125-0:1, org.apache.activemq\Broker\localhost\BrokerName=localhost, org.apache.activemq\Broker\localhost\BrokerVersion=5.14.3, org.apache.activemq\Broker\localhost\CurrentConnectionsCount=0, org.apache.activemq\Broker\localhost\DataDirectory=D:\tmp\apache-activemq-5.14.3\data, org.apache.activemq\Broker\localhost\DurableTopicSubscribers=[], org.apache.activemq\Broker\localhost\DynamicDestinationProducers=[], org.apache.activemq\Broker\localhost\InactiveDurableTopicSubscribers=[], org.apache.activemq\Broker\localhost\JMSJobScheduler=null, org.apache.activemq\Broker\localhost\JobSchedulerStoreLimit=0, org.apache.activemq\Broker\localhost\JobSchedulerStorePercentUsage=0, org.apache.activemq\Broker\localhost\MaxMessageSize=1024, org.apache.activemq\Broker\localhost\MemoryLimit=668309914, org.apache.activemq\Broker\localhost\MemoryPercentUsage=0, org.apache.activemq\Broker\localhost\MinMessageSize=1024, org.apache.activemq\Broker\localhost\Persistent=true, org.apache.activemq\Broker\localhost\QueueProducers=[], org.apache.activemq\Broker\localhost\QueueSubscribers=[], org.apache.activemq\Broker\localhost\Queues=[], org.apache.activemq\Broker\localhost\Slave=false, org.apache.activemq\Broker\localhost\StatisticsEnabled=true, org.apache.activemq\Broker\localhost\StoreLimit=42845775469, org.apache.activemq\Broker\localhost\StorePercentUsage=0, org.apache.activemq\Broker\localhost\TempLimit=42842071040, org.apache.activemq\Broker\localhost\TempPercentUsage=0, org.apache.activemq\Broker\localhost\TemporaryQueueProducers=[], org.apache.activemq\Broker\localhost\TemporaryQueueSubscribers=[], org.apache.activemq\Broker\localhost\TemporaryQueues=[], org.apache.activemq\Broker\localhost\TemporaryTopicProducers=[], org.apache.activemq\Broker\localhost\TemporaryTopicSubscribers=[], org.apache.activemq\Broker\localhost\TemporaryTopics=[], org.apache.activemq\Broker\localhost\TopicProducers=[], org.apache.activemq\Broker\localhost\TopicSubscribers=[], org.apache.activemq\Broker\localhost\Topics=[org.apache.activemq:type=Broker, brokerName=localhost, destinationType=Topic, destinationName=ActiveMQ.Advisory.MasterBroker], org.apache.activemq\Broker\localhost\TotalConnectionsCount=0, org.apache.activemq\Broker\localhost\TotalConsumerCount=0, org.apache.activemq\Broker\localhost\TotalDequeueCount=0, org.apache.activemq\Broker\localhost\TotalEnqueueCount=1, org.apache.activemq\Broker\localhost\TotalMessageCount=0, org.apache.activemq\Broker\localhost\TotalProducerCount=0, org.apache.activemq\Broker\localhost\TransportConnectors=[amqp=amqp://samtis:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600, mqtt=mqtt://samtis:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600, ws=ws://samtis:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600, openwire=tcp://samtis:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600, stomp=stomp://samtis:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600], org.apache.activemq\Broker\localhost\Uptime=13 minutes, org.apache.activemq\Broker\localhost\UptimeMillis=822599, org.apache.activemq\Broker\localhost\VMURL=vm://localhost, ...NOTE: Entries are sorted by key alphanumeric ordering and key representation is more common to be used for, e.g., tree model construction to represent JMX structure more like
JConsoledoes. -
LevelingJSONFormatter- this class provides JSON formatting for tnt4j activities, events and snapshots. Difference fromcom.jkoolcloud.tnt4j.format.JSONFormatteris only facts payload is present in produced JSON without major part of TNT4J metadata. Sample output:{ "source": "com.jkoolcloud.tnt4j.stream.jmx.impl.PlatformJmxSampler", "source-ssn": "tnt4j-stream-jmx", "type": "ACTIVITY", "snapshots": [ { "type": "SNAPSHOT", "name": "JMImplementation:type=MBeanServerDelegate", "properties": { "ImplementationName": "JMX", "ImplementationVendor": "Oracle Corporation", "ImplementationVersion": "1.8.0_152-b16", "MBeanServerId": "samtis_1517919369871", "SpecificationName": "Java Management Extensions", "SpecificationVendor": "Oracle Corporation", "SpecificationVersion": "1.4", "type": "MBeanServerDelegate" } }, { "type": "SNAPSHOT", "name": "SampleContext", "properties": { "condition.count": 0, "last.metric.count": 5688, "listener.compositeProperty.delimiter": "\\", "listener.count": 1, "listener.exclude.set.count": 0, "listener.forceObjectName.mode": false, "listener.silence.set.count": 6, "listener.trace.mode": true, "listener.useObjectNameProperties.mode": true, "mbean.count": 636, "noop.count": 0, "sample.count": 1, "sample.time.usec": 452552, "total.action.count": 0, "total.error.count": 6, "total.exclude.count": 0, "total.metric.count": 5688 } }, { "type": "SNAPSHOT", "name": "Self", "properties": { "elapsed.usec": 10442872, "id.count": 0, "level": "INFO", "pid": 13388, "snap.count": 637, "tid": 1 } }, { "type": "SNAPSHOT", "name": "java.lang:name=Code Cache,type=MemoryPool", "properties": { "CollectionUsage": null, "CollectionUsageThreshold": "<unsupported>", "CollectionUsageThresholdCount": "<unsupported>", "CollectionUsageThresholdExceeded": "<unsupported>", "CollectionUsageThresholdSupported": false, "MemoryManagerNames": "[CodeCacheManager]", "Name": "Code Cache", "ObjectName": "java.lang:type=MemoryPool,name=Code Cache", "PeakUsage.committed": 16384000, "PeakUsage.init": 2555904, "PeakUsage.max": 251658240, "PeakUsage.used": 16255488, "Type": "NON_HEAP", "UsageThreshold": 0, "UsageThresholdCount": 0, "UsageThresholdExceeded": false, "UsageThresholdSupported": true, "Usage.committed": 16384000, "Usage.init": 2555904, "Usage.max": 251658240, "Usage.used": 16062016, "Valid": true, "name": "Code Cache", "type": "MemoryPool" } }, { "type": "SNAPSHOT", "name": "java.lang:name=CodeCacheManager,type=MemoryManager", "properties": { "MemoryPoolNames": "[Code Cache]", "Name": "CodeCacheManager", "ObjectName": "java.lang:type=MemoryManager,name=CodeCacheManager", "Valid": true, "name": "CodeCacheManager", "type": "MemoryManager" } }, { "type": "SNAPSHOT", "name": "java.lang:name=Compressed Class Space,type=MemoryPool", "properties": { "CollectionUsage": null, "CollectionUsageThreshold": "<unsupported>", "CollectionUsageThresholdCount": "<unsupported>", "CollectionUsageThresholdExceeded": "<unsupported>", "CollectionUsageThresholdSupported": false, "MemoryManagerNames": ["Metaspace Manager"], "Name": "Compressed Class Space", "ObjectName": "java.lang:type=MemoryPool,name=Compressed Class Space", "PeakUsage.committed": 4325376, "PeakUsage.init": 0, "PeakUsage.max": 1073741824, "PeakUsage.used": 4108024, "Type": "NON_HEAP", "UsageThreshold": 0, "UsageThresholdCount": 0, "UsageThresholdExceeded": false, "UsageThresholdSupported": true, "Usage.committed": 4325376, "Usage.init": 0, "Usage.max": 1073741824, "Usage.used": 4108024, "Valid": true, "name": "Compressed Class Space", "type": "MemoryPool" } }, ... ] }NOTE: Entries are sorted by snapshot name and property key alphanumeric ordering.
Where do the streams go?
Stream-JMX streams all collected metrics based on a scheduled interval via TNT4J event streaming framework. All streams are written into TNT4J event sinks defined in tnt4j.properties file which is defined by -Dtnt4j.config=tnt4j.properties property.
To jKoolCloud: (Requires JESL libraries)
Below is an example of TNT4J stream configuration writing collected JMX samples to jKoolCloud server, formatted as JSON:
;Stanza used for Stream-JMX sources
{
source: com.jkoolcloud.tnt4j.stream.jmx
source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
source.factory.GEOADDR: New York
source.factory.DATACENTER: YourDC
source.factory.SERVICE: $sjmx.serviceId
source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GEOADDR=?
source.factory.RootSSN: tnt4j-stream-jmx
tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
dump.sink.factory: com.jkoolcloud.tnt4j.dump.DefaultDumpSinkFactory
event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl
; Event Sink configuration for streaming to jKoolCloud
; NOTE: Requires JESL libraries (http://nastel.github.io/JESL/)
event.sink.factory.EventSinkFactory: com.jkoolcloud.jesl.tnt4j.sink.JKCloudEventSinkFactory
event.sink.factory.EventSinkFactory.LogSink: file:./logs/stream-jmx_activities.json
event.sink.factory.EventSinkFactory.Url: https://data.jkoolcloud.com
event.sink.factory.EventSinkFactory.Token: YOUR-ACCESS-TOKEN
event.formatter: com.jkoolcloud.tnt4j.format.JSONFormatter
; Configure default sink filter based on level and time (elapsed/wait)
event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
event.sink.factory.Filter.Level: TRACE
tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}
To socket
Below is an example of TNT4J stream configuration writing collected JMX samples to a socket event sink com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory, formatted by com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter:
;Stanza used for Stream-JMX sources
{
source: com.jkoolcloud.tnt4j.stream.jmx
source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
source.factory.GEOADDR: New York
source.factory.DATACENTER: YourDC
source.factory.SERVICE: $sjmx.serviceId
source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GEOADDR=?
source.factory.RootSSN: tnt4j-stream-jmx
tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
dump.sink.factory: com.jkoolcloud.tnt4j.dump.DefaultDumpSinkFactory
event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl
event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory
; If socket sent data should no be logged anywhere else
event.sink.factory.EventSinkFactory.LogSink: null
; If socket sent data should be logged to file
#event.sink.factory.EventSinkFactory.LogSink: file:./logs/tnt4j-stream-jmx_samples_socket.log
event.sink.factory.EventSinkFactory.Host: localhost
event.sink.factory.EventSinkFactory.Port: 6060
; NOTE: DO NOT define "event.formatter" property value if have no need for custom formatter.
; SamplerFactory will take care to set appropriate one for a context.
#event.formatter: com.jkoolcloud.tnt4j.format.JSONFormatter
; If JMX attributes should be formatted as JMX object names
event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter
; If JMX attributes should be formatted as JMX object paths
#event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactPathValueFormatter
; If JMX attributes should be formatted as JMX object paths for IBM WAS and Liberty
#event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.SLIFactPathValueFormatter
; Mapping of attribute key string symbol replacements
#event.formatter.KeyReplacements: " "->"_" "\""->"'" "/"->"%" "="->"\\" ","->"!'" "\\\\"->"\\"
; Mapping of attribute value string symbol replacements
#event.formatter.ValueReplacements: "\r"->"\\r" "\n"->"\\n" ";"->"|" ","->"|" "["->"{(" "]"->")}" "\""->"'"
; Definitions ObjectName attributes sets used when building path: ';' is level set delimiter and ',' is set attribute names delimiter
#event.formatter.PathLevelAttributes: domain; type; name, brokerName; service, connector, destinationType; instanceName, connectorName, destinationName
; Defines JMX sample attribute key suffix to be added when duplicate keys for "branch" and "leaf" nodes are found.
; NOTE: AP does not allow to have same name for "branch" and "leaf" nodes at same tree level
#event.formatter.DuplicateKeySuffix: ___
; Configure default sink filter based on level and time (elapsed/wait)
event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
event.sink.factory.Filter.Level: TRACE
tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}
To file
Below is an example of TNT4J stream configuration writing collected JMX samples to a log file MyStream.log, formatted by com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter:
;Stanza used for Stream-JMX sources
{
source: com.jkoolcloud.tnt4j.stream.jmx
source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
source.factory.GEOADDR: New York
source.factory.DATACENTER: YourDC
source.factory.SERVICE: $sjmx.serviceId
source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GEOADDR=?
source.factory.RootSSN: tnt4j-stream-jmx
tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
dump.sink.factory: com.jkoolcloud.tnt4j.dump.DefaultDumpSinkFactory
event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl
event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.FileEventSinkFactory
event.sink.factory.EventSinkFactory.FileName: ./MyStream.log
; NOTE: DO NOT define "event.formatter" property value if have no need for custom formatter.
; SamplerFactory will take care to set appropriate one for a context.
#event.formatter: com.jkoolcloud.tnt4j.format.JSONFormatter
; If JMX attributes should be formatted as JMX object names
event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter
; If JMX attributes should be formatted as JMX object paths
#event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactPathValueFormatter
; If JMX attributes should be formatted as JMX object paths for IBM WAS and Liberty
#event.formatter: com.jkoolcloud.tnt4j.stream.jmx.format.SLIFactPathValueFormatter
; Mapping of attribute key string symbol replacements
#event.formatter.KeyReplacements: " "->"_" "\""->"'" "/"->"%" "="->"\\" ","->"!'" "\\\\"->"\\"
; Mapping of attribute value string symbol replacements
#event.formatter.ValueReplacements: "\r"->"\\r" "\n"->"\\n" ";"->"|" ","->"|" "["->"{(" "]"->")}" "\""->"'"
; Definitions ObjectName attributes sets used when building path: ';' is level set delimiter and ',' is set attribute names delimiter
#event.formatter.PathLevelAttributes: domain; type; name, brokerName; service, connector, destinationType; instanceName, connectorName, destinationName
; Defines JMX sample attribute key suffix to be added when duplicate keys for "branch" and "leaf" nodes are found.
; NOTE: AP does not allow to have same name for "branch" and "leaf" nodes at same tree level
#event.formatter.DuplicateKeySuffix: ___
; Configure default sink filter based on level and time (elapsed/wait)
event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
event.sink.factory.Filter.Level: TRACE
tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}
And even more
You can also write your own custom event sinks (HTTPS, HTTP, etc) and your own stream formatters without having to change Stream-JMX code or your application. TNT4J comes with a set of built-in event sink implementations such as:
com.jkoolcloud.tnt4j.logger.Log4JEventSinkFactory-- log4jcom.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory-- buffered sinkcom.jkoolcloud.tnt4j.sink.impl.FileEventSinkFactory- standard log filecom.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory-- socket (tcp/ip)com.jkoolcloud.tnt4j.sink.impl.NullEventSinkFactory-- null (empty)
Streaming to multiple endpoints
Consider one of most common cases when Stream-JMX produced samples shall appear on both AutoPilot and jKool at same time. To achieve that, BroadcastingEventSinkFactory shall be used and have two sinks bound: for AutoPilot endpoint - ap (SocketEventSinkFactory) and for jKool endpoint - jkool (JKCloudEventSinkFactory). This way same JMX sample will be sent to AutoPilot over configured socket having sample data formatted by FactNameValueFormatter and to jKool repo defined by access token having sample data formatted by JSONFormatter.
;Stanza used for Stream-JMX sources
{
source: com.jkoolcloud.tnt4j.stream.jmx
source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
source.factory.GEOADDR: New York
source.factory.DATACENTER: YourDC
source.factory.SERVICE: $sjmx.serviceId
source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GEOADDR=?
source.factory.RootSSN: tnt4j-stream-jmx
tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
dump.sink.factory: com.jkoolcloud.tnt4j.dump.DefaultDumpSinkFactory
event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BroadcastingEventSinkFactory
event.sink.factory.BroadcastSequence: ap,jkool
event.sink.factory.EventSinkFactory.ap: com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory
; If socket sent data should no be logged anywhere else
event.sink.factory.EventSinkFactory.ap.LogSink: null
; If socket sent data should be logged to file
#event.sink.factory.EventSinkFactory.ap.LogSink: file:./logs/tnt4j-stream-jmx_samples_socket.log
event.sink.factory.EventSinkFactory.ap.Host: localhost
event.sink.factory.EventSinkFactory.ap.Port: 6060
event.sink.factory.EventSinkFactory.ap.Formatter: com.jkoolcloud.tnt4j.stream.jmx.format.FactNameValueFormatter
event.sink.factory.EventSinkFactory.ap.Formatter.KeyReplacements: " "->"_" "\""->"'" "/"->"%" "="->"\\" ","->"!'" "\\\\"->"\\"
event.sink.factory.EventSinkFactory.ap.Formatter.ValueReplacements: "\r"->"\\r" "\n"->"\\n" ";"->"&" ","->"&" "["->"{(" "]"->")}" "\""->"'"
event.sink.factory.EventSinkFactory.jkool: com.jkoolcloud.jesl.tnt4j.sink.JKCloudEventSinkFactory
event.sink.factory.EventSinkFactory.jkool.LogSink: file:./logs/stream-jmx_activities.json
event.sink.factory.EventSinkFactory.jkool.Url: https://data.jkoolcloud.com
event.sink.factory.EventSinkFactory.jkool.Token: YOUR-ACCESS-TOKEN
event.sink.factory.EventSinkFactory.jkool.Formatter: com.jkoolcloud.tnt4j.format.JSONFormatter
; Configure default sink filter based on level and time (elapsed/wait)
event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
event.sink.factory.Filter.Level: TRACE
tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}
Auto-generating application state dump
Stream-JMX is utilizing TNT4J state dump capability to generate application state dumps
(1) Dump on VM shut-down:
java -Dtnt4j.dump.on.vm.shutdown=true -Dtnt4j.dump.provider.default=true -Dtnt4j.dump.folder=./ ...
(2) Dump on uncaught thread exceptions:
java -Dtnt4j.dump.on.exceptionn=true -Dtnt4j.dump.provider.default=true -Dtnt4j.dump.folder=./ ...
-Dtnt4j.dump.folder=./ specifies the destination folder where dump (.dump) files will be created (default is current working directory).
By default Stream-JMX will generate dumps with the following info:
- Java Properties Dump --
PropertiesDumpProvider - Java Runtime Dump --
MXBeanDumpProvider - Thread Stack Dump --
ThreadDumpProvider - Thread Deadlock Dump --
ThreadDumpProvider - Logging Statistics Dump --
LoggerDumpProvider
You may create your own dump providers and handlers
Overriding default SamplerFactory
SamplerFactory instances are used to generate Sampler implementation for a specific runtime environment. Stream-JMX supplies sampler and sampler factories for standard JVMs, J2EE, JBoss, IBM WebSphere Application Server, IBM WebSphere Liberty server. You may want to override default SamplerFactory with your own or an alternative by specifying:
java -Dcom.jkoolcloud.tnt4j.stream.jmx.sampler.factory=com.jkoolcloud.tnt4j.stream.jmx.impl.PlatformSamplerFactory ...
SamplerFactory is used to generate instances of the underlying sampler implementations (objects that provide sampling of underlying mbeans).
// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
...
Managing Sample Behavior
Stream-JMX provides a way to intercept sampling events such as pre, during an post for each sample run and control sample behavior. See SampleListener interface for more details. Applications may register more than one listener per Sampler. Each listener is called in registration order.
In addition to intercepting sample events, applications may want to control how one ore more attributes are sampled and whether each sample is reported/logged. See example below:
// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).addListener(new MySampleListener())).run();
Below is a sample of what MySampleListener may look like:
class MySampleListener implements SampleListener {
@Override
public void getStats(SampleContext context, Map<String, Object> stats) {
// add your own stats to the map
}
@Override
public void register(SampleContext context, ObjectName oname) {
System.out.println("Register mbean: " + oname + ", mbean.server=" + context.getMBeanServer());
}
@Override
public void unregister(SampleContext context, ObjectName oname) {
System.out.println("Unregister mbean: " + oname + ", mbean.server=" + context.getMBeanServer());
}
@Override
public void pre(SampleContext context, Activity activity) {
// called once per sample, beginning of each sample
// set activity to NOOP to disable further sampling
// no other attribute will be sampled during current sample
if (some-condition) {
activity.setType(OpType.NOOP);
}
}
@Override
public void pre(SampleContext context, AttributeSample sample) {
// called once before attribute is sampled
// set exclude to true to skip sampling this attribute
sample.excludeNext(sample.getAttributeInfo().isReadable());
}
@Override
public void post(SampleContext context, AttributeSample sample) {
// called once after attribute is sampled
Object value = sample.get();
}
@Override
public void post(SampleContext context, Activity activity) {
// called once per sample, end of each sample
// set activity to NOOP to disable sampling reporting
if (some-condition) {
activity.setType(OpType.NOOP);
}
}
@Override
public void error(SampleContext context, Throwable ex) {
// called once for every exception that occurs not associated with a sample
ex.printStackTrace();
}
@Override
public void error(SampleContext context, AttributeSample sample) {
// called once for every exception that occurs during each sample
Throwable ex = sample.getError();
ex.printStackTrace();
}
}
Conditions and Actions
Stream-JMX allows you to associate conditions with user defined actions based on values of MBean attributes on each sampling interval. For example, what if you wanted to setup an action when a specific MBean attribute exceeds a certain threshold?
Stream-JMX AttributeCondition and AttributeAction interfaces allow you to call your action at runtime every time a condition is evaluated to true. See example below:
// return default or user defined SamplerFactory implementation
SamplerFactory factory = DefaultSamplerFactory.getInstance();
// create an instance of the sampler that will sample mbeans
Sampler sampler = factory.newInstance();
// create a condition when ThreadCount > 100
AttributeCondition myCondition = new SimpleCondition("java.lang:type=Threading", "ThreadCount", 100, ">");
// schedule collection (ping) for given MBean filter and 30000 ms sampling period
sampler.setSchedule(Sampler.JMX_FILTER_ALL, 30000).register(myCondition, new MyAttributeAction()).run();
Below is a sample of what MyAttributeAction may look like:
public class MyAttributeAction implements AttributeAction {
@Override
public Object action(SampleContext context, AttributeCondition cond, AttributeSample sample) {
Activity activity = sample.getActivity();
// obtain a collection of all sampled metrics
Collection<Snapshot> metrics = activity.getSnapshots();
System.out.println("MyAction called with value=" + sample.get()
+ ", age.usec=" + sample.ageUsec()
+ ", count=" + metrics.size());
return null;
}
}
How to Build TNT4J-Stream-JMX
Modules
- (M) marked modules are mandatory
- (O) marked modules are optional
- (U) marked modules are utility modules (e.g., performs compiled assemblies packaging)
Modules list:
Core(M) - implements core JMX sampling, processing and dissemination features.J2EE(O) - J2EE API addition to handle J2EE API defined attributes and implements abstract Stream-JMX servlet API.WAS(O, requiresJ2EE) - IBM WebSphere Application Server (WAS) root module.Api(O) - builds specific API used to sample WAS JMX (jar).War(O) - builds WAS compliant web application package (war).Ear(O) - builds WAS compliant enterprise application package (ear).
Liberty(O, requiresJ2EE) - IBM WebSphere Liberty root module.Api(O) - builds specific API used to sample Liberty JMX (jar).War(O) - builds Liberty compliant web application package (war).
ZK(O) - JMX connections resolution for VMs orchestrated by ZooKeeper.Distribution(OU) - build distribution packages, see../build/tnt4j-stream-jmxdirectory.
All optional modules (extensions) depends to core module and can't be build and run without it.
NOTE: Distribution module performs maven post build release assemblies delivery to ../build/tnt4j-stream-jmx directory.
Requirements
- JDK 1.8+
- Apache Maven 3
- TNT4J
- JESL
All other required dependencies are defined in project modules pom.xml files. If maven is running online mode it should download these defined dependencies automatically.
Manually installed dependencies
Some of required and optional dependencies may be not available in public Maven Repository. In this case we would recommend to download those dependencies manually into module's lib directory and install into local maven repository by running maven script lib/pom.xml with install goal. For example see tnt4j-stream-jmx/tnt4j-stream-jmx-was/tnt4j-stream-jmx-was-api/lib/pom.xml how to do this.
Core module
This module does not require manually downloaded dependencies, but depends on JDK contained instrumentation library tools.jar. Dependency is defined in:
- Maven POM script by
<dependency>
<groupId>com.sun</groupId>
<artifactId>tools</artifactId>
<version>${project.java.version}</version>
<scope>system</scope>
<systemPath>${java.home}/../lib/tools.jar</systemPath>
</dependency>
- System executables
bin/stream-jmx*.batorbin/stream-jmx*.shby environment variableTOOLS_PATH
set TOOLS_PATH="%JAVA_HOME%\lib\tools.jar"
TOOLS_PATH="$JAVA_HOME/lib/tools.jar"
NOTE: you may need to change paths if these do not match your environment.
WAS module
NOTE: Because this module requires manually downloaded libraries, it is commented out in main project pom file tnt4j-stream-jmx/pom.xml by default. If you want to use it uncomment this line of pom.xml file. But WAS module will be ready to build only when manually downloaded libraries will be installed to local maven repository.
What to download manually or copy from your existing IBM MQ installation:
- IBM WAS (e.g., version 8.5) client libs
- j2ee API library
Download the above libraries and place into the tnt4j-stream-jmx/tnt4j-stream-jmx-was/tnt4j-stream-jmx-was-api/lib directory like this:
lib
+ was
|- com.ibm.ws.admin.client_8.5.0.jar
|- com.ibm.ws.ejb.thinclient_8.5.0.jar
|- com.ibm.ws.orb_8.5.0.jar
(O) marked libraries are optional
Additionally alter system executables bin/stream-jmx*.bat or bin/stream-jmx*.sh by extending environment variable LIBPATH value:
set WAS_HOME=C:\IBM\WebSphere\AppServer
set WAS_PATH=%WAS_HOME%\runtimes\*;%WAS_HOME%\lib\webadmin\management.jar;%WAS_HOME%\plugins\com.ibm.ws.runtime.jar
set LIBPATH=%RUNDIR%..\*;%RUNDIR%..\lib\*;%TOOLS_PATH%;%WAS_PATH%
WAS_HOME="/opt/IBM/WebSphere/AppServer"
WAS_PATH="$WAS_HOME/runtimes/*:$WAS_HOME/lib/webadmin/management.jar:$WAS_HOME/plugins/com.ibm.ws.runtime.jar"
LIBPATH="$SCRIPTPATH/../*:$SCRIPTPATH/../lib/*:$TOOLS_PATH:$WAS_PATH"
If you don't have actual WAS installation, WAS_PATH may also refer to jars located in tnt4j-stream-jmx/tnt4j-stream-jmx-was/lib/* directory.
Building
- To build the project, run maven goals
clean package - To build the project and install to local repo, run maven goals
clean install - To make distributable release assemblies use one of profiles:
pack-binorpack-all:- containing only binary distribution: run
mvn -P pack-bin - containing binary, source and javadoc distribution: run
mvn -P pack-all
- containing only binary distribution: run
By default maven will build all modules defined in tnt4j-stream-jmx/pom.xml file.
If you do not want to build some of optional modules, comment those out like WAS module is. Or you can define maven to build your preferred set of modules using -pl, --projects argument (comma separated modules list) together with -am, --also-make argument, e.g.:
mvn -pl tnt4j-stream-jmx-core,tnt4j-stream-jmx--distribution -am clean install -P pack-bin
or
mvn --projects tnt4j-stream-jmx-core,tnt4j-stream-jmx--distribution --also-make clean install -P pack-bin
NOTE: modules list should be without spaces after comma!
Issuing these commands, maven will build only tnt4j-stream-jmx-core and tnt4j-stream-jmx--distribution modules.
Release assemblies are built to ../build/tnt4j-stream-jmx directory.
NOTE: sometimes maven fails to correctly handle dependencies. If dependency configuration looks fine, but maven still complains about missing dependencies try to delete local maven repository by hand: e.g., on MS Windows delete contents of c:\Users\[username]\.m2\repository directory.
Running samples
See Using Stream-JMX section.
