What is System Cube

For better supporting self-monitoring, a set of system Cubes are created under the system project, called “KYLIN_SYSTEM”. Currently, there are five Cubes. Three are for query metrics, “METRICS_QUERY_EXECUTION”, “METRICS_QUERY_SPARK_JOB”, “METRICS_QUERY_SPARK_STAGE”. And the other two are for job metrics, “METRICS_JOB”, “METRICS_JOB_EXCEPTION”.

How to Set Up System Cube

Please check document: How to use System Cube in Kylin4.0

Details of System Cube

Common Dimension

For all of these Cube, admins can query at four time granularities. From higher level to lower, it’s as follows:

KYEAR_BEGIN_DATE year
KMONTH_BEGIN_DATE month
KWEEK_BEGIN_DATE week
KDAY_DATE date

METRICS_QUERY

This Cube is for collecting query metrics at the Spark Execution level.

METRICS_QUERY_SPARK_JOB

This Cube is for collecting Spark Job level query metrics.

METRICS_QUERY_SPARK_STAGE

This Cube is for collecting Spark Stage level query metrics.

METRICS_JOB

In Kylin, there are mainly three types of job:
- “BUILD”, for building Cube segments from HIVE.
- “MERGE”, for merging Cube segments in storage engine.
- “OPTIMIZE”, for dynamically adjusting the precalculated cuboid tree base on the base cuboid in storage engine.

This Cube is for collecting job metrics. The details are as follows:

Dimension
PROJECT
CUBE_NAME
JOB_TYPE
CUBING_TYPE in kylin, there are two cubing algorithms, Layered & Fast(InMemory)
Measure
COUNT
MIN, MAX, SUM of DURATION the duration from a job start to finish
MIN, MAX, SUM of TABLE_SIZE the size of data source in bytes
MIN, MAX, SUM of CUBE_SIZE the size of created Cube segment in bytes
MIN, MAX, SUM of PER_BYTES_TIME_COST = DURATION / TABLE_SIZE
MIN, MAX, SUM of WAIT_RESOURCE_TIME a job may includes several MR(map reduce) jobs. Those MR jobs may wait because of lack of Hadoop resources.

METRICS_JOB_EXCEPTION

This Cube is for collecting job exception metrics. The details are as follows:

Dimension
PROJECT
CUBE_NAME
JOB_TYPE
CUBING_TYPE
EXCEPTION when running a job, exceptions may happen. It's for classifying different exception types
Measure
COUNT