JarCasting
PerfMon Server Agent
Download
(previous ServerAgent-2.2.1.zip)
Installation
You do not need any root/admin privilege. You can just unzip the the ServerAgent-X.X.X.zip somewhere on the server. Then launch the agent using startAgent.sh script on Unix, or startAgent.bat script on Windows.
The agent is written in Java, you will need JRE 1.5 or later to run it. Note you can create yourself the agent package which includes its own JRE so you don't have to install java on the server (We tested it on windows platform). To do this, just create a JRE folder in the agent folder and copy one installed JRE inside. Change the java command inside the .bat file to the path to the java executable and it will work.
Usage
To start the agent, simply run startAgent.bat/sh file. It will open UDP/TCP server ports where clients can connect and query the metrics. Some information will be printed to standard output, informing you on problems gathering metrics (logging verbosity regulated with --loglevel parameter).
You can specify the listening ports as arguments (0 disables listening), default is 4444:
$ ./startAgent.sh --udp-port 0 --tcp-port 3450
INFO 2011-11-25 19:02:14.315 [kg.apc.p] (): Binding TCP to 3450
You can use the --auto-shutdown option when starting the agent to automatically stop it once the last client is disconnected. It is recommended to use this feature only with TCP connections:
$ undera@undera-HP:/tmp/serverAgent$ ./startAgent.sh --udp-port 0 --auto-shutdown
INFO 2011-11-25 19:48:59.321 [kg.apc.p] (): Agent will shutdown when all clients disconnected
INFO 2011-11-25 19:48:59.424 [kg.apc.p] (): Binding TCP to 4444
You can use the --sysinfo option to view available system objects:
$ ./startAgent.sh --sysinfo
INFO 2011-11-25 19:51:25.517 [kg.apc.p] (): *** Logging available processes ***
INFO 2011-11-25 19:51:25.542 [kg.apc.p] (): Process: pid=24244 name=bash args=-bash
INFO 2011-11-25 19:51:25.543 [kg.apc.p] (): Process: pid=25086 name=dash args=/bin/sh ./startAgent.sh --sysinfo
INFO 2011-11-25 19:51:25.543 [kg.apc.p] (): Process: pid=25088 name=java args=java -jar ./CMDRunner.jar --tool PerfMonAgent --sysinfo
INFO 2011-11-25 19:51:25.549 [kg.apc.p] (): *** Logging available filesystems ***
INFO 2011-11-25 19:51:25.551 [kg.apc.p] (): Filesystem: fs=/dev type=devtmpfs
INFO 2011-11-25 19:51:25.551 [kg.apc.p] (): Filesystem: fs=/ type=ext4
INFO 2011-11-25 19:51:25.551 [kg.apc.p] (): Filesystem: fs=/var/run type=tmpfs
INFO 2011-11-25 19:51:25.551 [kg.apc.p] (): Filesystem: fs=/home type=ext4
INFO 2011-11-25 19:51:25.552 [kg.apc.p] (): Filesystem: fs=/var/lock type=tmpfs
INFO 2011-11-25 19:51:25.552 [kg.apc.p] (): Filesystem: fs=/proc type=proc
INFO 2011-11-25 19:51:25.553 [kg.apc.p] (): *** Logging available network interfaces ***
INFO 2011-11-25 19:51:25.554 [kg.apc.p] (): Network interface: iface=lo addr=127.0.0.1 type=Local Loopback
INFO 2011-11-25 19:51:25.554 [kg.apc.p] (): Network interface: iface=eth0 addr=192.168.0.1 type=Ethernet
INFO 2011-11-25 19:51:25.555 [kg.apc.p] (): *** Done logging sysinfo ***
INFO 2011-11-25 19:51:25.555 [kg.apc.p] (): Binding UDP to 4444
INFO 2011-11-25 19:51:26.560 [kg.apc.p] (): Binding TCP to 4444
The --interval <seconds> argument can be used to change metrics collection frequency.
Using Server Agent With Other Applications
Server Agent uses simple plain-text protocol, anyone can use agent's capabilities implementing client, based on kg.apc.perfmon.client.Transport interface. If anyone's interested, start the topic on the support forums and I'll describe how to connect third-party client app to agent.
ServerAgent has simple text protocol and can work on UDP and TCP transports. Most of cases will use TCP.
To have your first talk with the agent, start it locally. Then use telnet utility to connect to it:
user@ubuntu:~$ telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
If connection has succeeded, you should see "Accepting new TCP connection" message in ServerAgent console log. Type "test" and press Enter in telnet chat, server should answer with short "Yep":
test
Yep
Type "exit":
exit
Connection closed by foreign host.
That's it. You sending a command line, server answering. Command line consists of command, sometimes with parameters. Parameters are separated from command with a colon sign.
Possible commands are:
- exit - terminates current client session and closes connection to agent, no parameters
- test - test if server is alive, no parameters
- shutdown - terminate all client connections and shutdown agent process, no parameters
- interval - change metrics reporting interval used in 'metrics' command, single parameter is integer value in seconds. Interval can be changed in the middle of metrics reporting. Example:
interval:5 - metrics - starts automatic metrics collection, parameters are metrics list to collect, described below. Example:
metrics:cpu - metrics-single - calls single metric collection iteration. Example:
metrics-single:memory
Specifying Metrics
Metrics list consists of metric specifications, separated by TAB character. Metric collection output consists of float values, TAB separated. Example:
metrics-single:cpu<TAB>memory
22.05736894164194<TAB>57.52359562205553
Each metric specification consists of several fields, colon-separated. Short example:
metrics-single:cpu:idle memory:free
80.02238388360381 57.52359562205553
Fields number is metric-type specific. Possible metric types are:
- cpu
- memory
- swap
- disks
- network
- tcp
- tail
- exec
- jmx
Fields corresponding to each metric type are described below. Last example (Yep, ServerAgent can be shell exec vulnerability. If you have issue with this, ask me and I'll introduce 'secure' mode, disabling insecure metric types):
metrics-single:exec:/bin/sh:-c:free | grep Mem | awk '{print $7}'
1152488
Viewing Available System Objects
Server Agent has special command-line option --sysinfo for printing available processes, filesystems and network interfaces together with their selectors.
Supported Metrics
CPU Metrics
Total
Primary
- combined
- idle
- system
- user
- iowait
Additional
- irq
- nice
- softirq
- stolen
Per Process
- percent
- total
- system
- user
Memory Metrics
Total
Primary
- usedperc - relative memory usage in percents
- freeperc
- used
- free
Additional
- actualfree
- actualused
- ram
- total
Per Process
Primary
- resident
- virtual
- shared
Additional
- pagefaults
- majorfaults
- minorfaults
Disk I/O Metrics
Primary
- queue
- reads
- writes
- readbytes
- writebytes
Additional
- available
- service
- files
- free
- freefiles
- total
- useperc
- used
Network I/O Metrics
Primary
- bytesrecv
- bytessent
- rx
- tx
Additional
- used
- speed
- rxdrops
- rxerr
- rxframe
- rxoverruns
- txcarrier
- txcollisions
- txdrops
- txerr
- txoverruns
JMX Metrics
Since version 0.5.2 Server Agent has ability to monitor some JMX values inside Java Virtual Machine. To enable JMX monitoring you must start Java with some special properties enabled, like described here. Here's simplest (and not so secure) options to start Java with JMX enabled:
java -Dcom.sun.management.jmxremote.port=4711 -Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false <other options>
By default the Server Agent will try to connect to JMX server at localhost with port 4711. If you started JMX server at different host/port or using authentication with username/password, please, use following additional parameters:
url=<hostname>\:<port>user=<username>password=<password>
Available JMX metric types:
- gc-time - time spent in garbage collection, milliseconds (used method)
- memory-usage - heap memory used by VM, bytes (method used)
- memory-committed - heap memory committed by VM, bytes (method used)
- memorypool-usage - heap memory pool usage, bytes (method used)
- memorypool-committed - heap memory pool committed size, bytes (method used)
- class-count - loaded class count in VM (used method)
- compile-time - time spent in compilation, milliseconds (used method)
Examples:
gc-time - monitor GC time at localhost:4711
memory-usage:url=somehost.com\:4715 - use alternative hostname/password
class-count:url=somehost.com\:4715:user=apc:password=SecurityPlease123 - some secure setup access
TCP Metrics
TCP metrics represents TCP socket state statistics (like open ports). Primary
- estab - established connections
- time_wait
- close_wait
Additional
- bound
- close
- closing
- fin_wait1
- fin_wait2
- idle
- inbound
- last_ack
- listen
- outbound
- syn_recv
Swap Metrics
- used
- pagein
- pageout
- free
- total
Custom Metrics
If you haven't found appropriate metric in above lists, you can set up collecting any value with custom metrics.
Security note: Both exec and tail metric types can be turned off by providing --no-exec flag to server agent. This enables more secure service with only strict KPI fetching.
EXEC
This metric type interprets parameter string as path to process to start and arguments to pass to the process. Parameters separated with colon (not space), see examples below. The process must print out to standard output single line containing single numeric metric value.
In most cases you'll need to start system interpreter (cmd.exe, /bin/sh) to run complex sequences of parameterized calls.
Example1: Monitoring Linux cached memory size (unavailable via SIGAR lib in default metrics), used free utility output:
/bin/sh:-c:free | grep Mem | awk '{print $7}'
Example2: Monitoring MySQL select query count:
/bin/sh:-c:echo "show global status like 'Com_select'" | mysql -u root | awk ' $1 =="Com_select" {print $2}'
TAIL
Another way to collect cutom metrics is to read lines off the end of the file. Metric parameter for tail type is path to file to read. Lines added to file must contain single numeric metric. Example:
123
342
333
297