Java程序监控---Metrics

java

概念

Metrics是一个给JAVA服务的各项指标提供度量工具的包，在JAVA代码中嵌入Metrics代码，可以方便的对业务代码的各个指标进行监控

目前最为流行的 metrics 库是来自 Coda Hale 的 dropwizard/metrics，该库被广泛地应用于各个知名的开源项目中。例如 Hadoop，Kafka，Spark，JStorm 中。

有一些优点：

  • 提供了对Ehcache、Apache HttpClient、JDBI、Jersey、Jetty、Log4J、Logback、JVM等的集成
  • 支持多种Metric指标：Gauges、Counters、Meters、Histograms和Timers
  • 支持多种Reporter发布指标
    • JMX、Console，CSV文件和SLF4J loggers
    • Ganglia、Graphite，用于图形化展示

MetricRegistry

MetricRegistry类是Metrics的核心，它是存放应用中所有metrics的容器。也是我们使用 Metrics 库的起点。其中maven依赖添加在文末。

1
static final MetricRegistry metrics = new MetricRegistry();

Reporter

指标获取之后需要上传到各种地方，就需要用到Reporter。

控制台

监控指标直接打印在控制台

1
2
3
4
5
6
7
pravite static void startReportConsole() {
    ConsoleReporter reporter = ConsoleReporter.forRegistry(metrics)
            .convertRatesTo(TimeUnit.SECONDS)
            .convertDurationsTo(TimeUnit.MILLISECONDS)
            .build();
    reporter.start(1, TimeUnit.SECONDS);
}

JMX

将监控指标上报到JMX中，后续可以通过其他的开源工具上传到Graphite等供图形化展示。从Jconsole中MBean中能看到。

1
2
3
4
pravite static void startReportJmx(){
    JmxReporter reporterJmx = JmxReporter.forRegistry(metrics).build();
    reporterJmx.start();
}

Graphite

将监控指标上传到Graphite，从Graphite-web中能看到上传的监控指标。

1
2
3
4
5
6
7
8
9
10
pravite static void startReportGraphite(){
    Graphite graphite = new Graphite(new InetSocketAddress("graphite.xxx.com", 2003));
    GraphiteReporter reporter = GraphiteReporter.forRegistry(metrics)
            .prefixedWith("test.metrics")
            .convertRatesTo(TimeUnit.SECONDS)
            .convertDurationsTo(TimeUnit.MILLISECONDS)
            .filter(MetricFilter.ALL)
            .build(graphite);
    reporter.start(1, TimeUnit.MINUTES);
}

封装各种Reporter

调用方式MetricCommon.getMetricAndStartReport();

1
2
3
4
5
6
7
8
9
10
11
12
13
public class MetricCommon {
    private static final MetricRegistry metricRegistry = new MetricRegistry();
    public static MetricRegistry getMetricAndStartReport(){
        startReportConsole();
        startReportJmx();
        startReportGraphite();
        return metricRegistry;
    }
    pravite static void startReportConsole() {...}
    pravite static void startReportJmx(){...}
    pravite static void startReportGraphite(){...}
}

Metics指标

Metrics 有如下监控指标：

  • Gauges：记录一个瞬时值。例如一个待处理队列的长度。
  • Histograms：统计单个数据的分布情况，最大值、最小值、平均值、中位数，百分比（75%、90%、95%、98%、99%和99.9%）
  • Meters：统计调用的频率（TPS），总的请求数，平均每秒的请求数，以及最近的1、5、15分钟的平均TPS
  • Timers：当我们既要统计TPS又要统计耗时分布情况，Timer基于Histograms和Meters来实现
  • Counter：计数器，自带inc()和dec()方法计数，初始为0。
  • Health Checks：用于对Application、其子模块或者关联模块的运行是否正常做检测

Gauges

最简单的度量指标，只有一个简单的返回值，例如，我们想衡量一个待处理队列中任务的个数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
public class GaugeTest {
    private static final MetricRegistry registry = MetricCommon.getMetricAndStartReport();
    private static final Random random = new Random();
    @Test
    public void testOneGuage() throws InterruptedException {
        Queue queue= new LinkedList<String>();
        registry.register(MetricRegistry.name(GaugeTest.class, "testGauges-queue-size", "size"),
                (Gauge<Integer>) () -> queue.size());
        while(true){
            Thread.sleep(1000);
            queue.add("Job-xxx");
        }
    }
    @Test
    public void testMultiGuage() throws InterruptedException {
        Map<Integer, Integer> map = new ConcurrentHashMap<>();
        while(true){
            int i = random.nextInt(100);
            int j = i % 10;
            if(!map.containsKey(j)){
                map.put(j,i);
                registry.register(MetricRegistry.name(GaugeTest.class, "testGauges-number", String.valueOf(j)),
                        (Gauge<Integer>) () -> map.get(j));
            }else{
                map.put(j,i);
            }
            Thread.sleep(1000);
        }
    }
}

第一个测试用例，是用一个guage记录队列的长度

1
2
3
-- Gauges ----------------------------------------------------------------------
GaugeTest.testGauges-queue-size.size
             value = 4

第二个测试用例，每次产生一个100以内的随机数，将这些数以个位数的数字分组，guage记录每一组现在是什么数。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
-- Gauges ----------------------------------------------------------------------
GaugeTest.testGauges-number.0
             value = 60
GaugeTest.testGauges-number.1
             value = 1
GaugeTest.testGauges-number.2
             value = 82
GaugeTest.testGauges-number.3
             value = 23
GaugeTest.testGauges-number.4
             value = 74
GaugeTest.testGauges-number.5
             value = 25
GaugeTest.testGauges-number.7
             value = 17
GaugeTest.testGauges-number.8
             value = 78
GaugeTest.testGauges-number.9
             value = 69

Histogram

Histogram统计数据的分布情况。比如最小值，最大值，中间值，还有中位数，75百分位, 90百分位, 95百分位, 98百分位, 99百分位, 和 99.9百分位的值(percentiles)。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
public class HistogramTest {
    private static final MetricRegistry registry = MetricCommon.getMetricAndStartReport();
    public static Random random = new Random();
    @Test
    public void test() throws InterruptedException {
        Histogram histogram = new Histogram(new ExponentiallyDecayingReservoir());
        registry.register(MetricRegistry.name(HistogramTest.class, "request", "histogram"), histogram);
        while(true){
            Thread.sleep(1000);
            histogram.update(random.nextInt(100000));
        }
    }
}

运行很长时间之后，相当于随机值取极限，会趋向于统计值，75%肯定是要<=75000,99.9%肯定是要<=999000。

1
2
3
4
5
6
7
8
9
10
11
12
13
-- Histograms ------------------------------------------------------------------
HistogramTest.request.histogram
             count = 1336
               min = 97
               max = 99930
              mean = 49816.49
            stddev = 29435.27
            median = 49368.00
              75% <= 75803.00
              95% <= 95340.00
              98% <= 98096.00
              99% <= 98724.00
            99.9% <= 99930.00

Meters

Meter度量一系列事件发生的速率(rate)，例如TPS。Meters会统计最近1分钟，5分钟，15分钟，还有全部时间的速率。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
public class MetersTest {
    MetricRegistry registry = MetricCommon.getMetricAndStartAllReport("nc110x.corp.youdao.com","test.metrics");
    public static Random random = new Random();
    @Test
    public void testOne() throws InterruptedException {
        Meter meterTps = registry.meter(MetricRegistry.name(MetersTest.class,"request","tps"));
        while(true){
            meterTps.mark();
            Thread.sleep(random.nextInt(1000));
        }
    }
    @Test
    public void testMulti() throws InterruptedException {
        while(true){
            int i = random.nextInt(100);
            int j = i % 10;
            Meter meterTps = registry.meter(MetricRegistry.name(MetersTest.class,"request","tps",String.valueOf(j)));
            meterTps.mark();
            Thread.sleep(10);
        }
    }
}

这里，多个注册多个meter与注册多个guage、Histograms用法会有不同，meter方法是getOrAdd

1
2
3
public Meter meter(String name) {
        return (Meter)this.getOrAdd(name, MetricRegistry.MetricBuilder.METERS);
}

一个meter的测试用例，运行结果如下。可以看到随着次数的增多，各种rate无限趋近于2次。

1
2
3
4
5
6
7
-- Meters ----------------------------------------------------------------------
MetersTest.request.tps
             count = 452
         mean rate = 1.99 events/second
     1-minute rate = 2.03 events/second
     5-minute rate = 2.00 events/second
    15-minute rate = 2.00 events/second

多个meter的测试用例，运行结果取了数字个位数为6/7/8的三个如下。最后都会无限趋近于10。sleep时间为10ms，每秒有100份，平均到尾数不同的，每组就有10份。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
MetersTest.request.tps.6
             count = 905
         mean rate = 9.74 events/second
     1-minute rate = 9.76 events/second
     5-minute rate = 9.94 events/second
    15-minute rate = 9.98 events/second
MetersTest.request.tps.7
             count = 935
         mean rate = 10.07 events/second
     1-minute rate = 10.62 events/second
     5-minute rate = 11.82 events/second
    15-minute rate = 12.19 events/second
MetersTest.request.tps.8
             count = 937
         mean rate = 10.09 events/second
     1-minute rate = 10.09 events/second
     5-minute rate = 10.31 events/second
    15-minute rate = 10.37 events/second

Timer

Timer其实是 Histogram 和 Meter 的结合， histogram 某部分代码/调用的耗时， meter统计TPS。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
public class TimerTest {
    public static Random random = new Random();
    private static final MetricRegistry registry = MetricCommon.getMetricAndStartAllReport("nc110x.corp.youdao.com","test.metrics");
    private static final Map<Integer,Timer> timerMap = new ConcurrentHashMap<>();
    @Test
    public void testOneTimer() throws InterruptedException {
        Timer timer = registry.timer(MetricRegistry.name(TestTimer.class,"get-latency"));
        Timer.Context ctx;
        while(true){
            ctx = timer.time();
            Thread.sleep(random.nextInt(1000));
            ctx.stop();
        }
    }
    @Test
    public void testMultiTimer() throws InterruptedException {
        while(true){
            int i = random.nextInt(100);
            int j = i % 10;
            Timer timer = registry.timer(MetricRegistry.name(TestTimer.class,"get-latency",String.valueOf(j)));
            Timer.Context ctx;
            ctx = timer.time();
            Thread.sleep(random.nextInt(1000));
            ctx.stop();
            Thread.sleep(1000);
        }
    }
}

测试用例1是单个timer，结果如下。最后的时间都趋近于统计值。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
-- Timers ----------------------------------------------------------------------
com.testmetrics.TestTimer.get-latency
             count = 657
         mean rate = 2.05 calls/second
     1-minute rate = 1.98 calls/second
     5-minute rate = 2.02 calls/second
    15-minute rate = 2.01 calls/second
               min = 4.98 milliseconds
               max = 998.93 milliseconds
              mean = 496.79 milliseconds
            stddev = 297.46 milliseconds
            median = 501.02 milliseconds
              75% <= 765.09 milliseconds
              95% <= 952.03 milliseconds
              98% <= 974.12 milliseconds
              99% <= 989.02 milliseconds
            99.9% <= 998.93 milliseconds

Counters

Counter 就是计数器，Counter 只是用 Gauge 封装了 AtomicLong 。我们可以使用如下的方法，使得获得队列大小更加高效。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
public class CounterTest {
    private static final MetricRegistry registry = MetricCommon.getMetricAndStartReport();
    public static Queue<String> q = new LinkedBlockingQueue<String>();
    public static Counter pendingJobs;
    public static Random random = new Random();
    public static void addJob(String job) {
        pendingJobs.inc();
        q.offer(job);
    }
    public static String takeJob() {
        pendingJobs.dec();
        return q.poll();
    }
    @Test
    public void test() throws InterruptedException {
        pendingJobs = registry.counter(MetricRegistry.name(Queue.class,"pending-jobs","size"));
        int num = 1;
        while(true){
            Thread.sleep(200);
            if (random.nextDouble() > 0.7){
                String job = takeJob();
                System.out.println("take job : "+job);
            }else{
                String job = "Job-"+num;
                addJob(job);
                System.out.println("add job : "+job);
            }
            num++;
        }
    }
}

job会越来越多，因为每次取走只取一个job，但是加入job是加入num个，num会一直增加，而概率是7:3。

1
2
3
-- Counters --------------------------------------------------------------------
java.util.Queue.pending-jobs.size
             count = 36

HeathChecks

Metrics提供了一个独立的模块：Health Checks，用于对Application、其子模块或者关联模块的运行是否正常做检测。该模块是独立metrics-core模块的，使用时则导入metrics-healthchecks包。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
public class HeathChecksTest extends HealthCheck {
    @Override
    protected Result check() throws Exception {
        Random random = new Random();
        if(random.nextInt(10)!=9){
            return Result.healthy();
        }else{
            return Result.unhealthy("oh,unhealthy");
        }
    }
    @Test
    public void test() throws InterruptedException {
        HealthCheckRegistry registry = new HealthCheckRegistry();
        registry.register("check1",new HeathChecksTest());
        registry.register("check2", new HeathChecksTest());
        while (true) {
            for (Map.Entry<String, Result> entry : registry.runHealthChecks().entrySet()) {
                if (entry.getValue().isHealthy()) {
                    System.out.println(entry.getKey() + ": OK, message:"+entry.getValue());
                } else {
                    System.err.println(entry.getKey() + ": FAIL, error message: " + entry.getValue());
                }
            }
            Thread.sleep(1000);
        }
    }
}

注册两个HeathChecks，重写其check()方法为取随机数，只要不是9就为healthy，输出结果如下：

1
2
3
4
5
6
7
8
9
check1: OK, message:Result{isHealthy=true}
check2: FAIL, error message: Result{isHealthy=false, message=oh,unhealthy}
check1: OK, message:Result{isHealthy=true}
check2: OK, message:Result{isHealthy=true}
check1: OK, message:Result{isHealthy=true}
check2: OK, message:Result{isHealthy=true}
check1: OK, message:Result{isHealthy=true}
check2: OK, message:Result{isHealthy=true}
check1: OK, message:Result{isHealthy=true}

maven依赖

  • metrics-core：必须添加
  • metrics-healthchecks：用到healthchecks时添加
  • metrics-graphite：用到graphite时添加
  • org.slf4j：不添加看不到metrics-graphite包出错的log
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    <properties>
        <metrics.version>3.1.0</metrics.version>
        <sl4j.version>1.7.22</sl4j.version>
    </properties>
    <dependency>
        <groupId>io.dropwizard.metrics</groupId>
        <artifactId>metrics-core</artifactId>
        <version>${metrics.version}</version>
    </dependency>
    <dependency>
        <groupId>io.dropwizard.metrics</groupId>
        <artifactId>metrics-healthchecks</artifactId>
        <version>${metrics.version}</version>
    </dependency>
    <dependency>
        <groupId>io.dropwizard.metrics</groupId>
        <artifactId>metrics-graphite</artifactId>
        <version>${metrics.version}</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>${sl4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-simple</artifactId>
        <version>${sl4j.version}</version>
    </dependency>

参考

http://metrics.dropwizard.io/3.1.0/getting-started/
http://www.cnblogs.com/nexiyi/p/metrics_sample_1.html
http://wuchong.me/blog/2015/08/01/getting-started-with-metrics/

本文标题:Java程序监控---Metrics

文章作者:小懒

发布时间:2016-12-14, 20:41:00

最后更新:2017-01-06, 22:11:49

原始链接:https://fangyeqing.github.io/2016/12/14/Java程序监控---Metrics/

许可协议: "署名-非商用-相同方式共享 4.0" 转载请保留原文链接及作者。

文章目录
  1. 1. 概念
  2. 2. MetricRegistry
  3. 3. Reporter
    1. 3.1. 控制台
    2. 3.2. JMX
    3. 3.3. Graphite
    4. 3.4. 封装各种Reporter
  4. 4. Metics指标
    1. 4.1. Gauges
    2. 4.2. Histogram
    3. 4.3. Meters
    4. 4.4. Timer
    5. 4.5. Counters
    6. 4.6. HeathChecks
  5. 5. maven依赖
  6. 6. 参考
|