System Log

After being successfully started, Kylin will create a directory named logs/ by default, all logs generated during Kylin runtime will be stored in this directory.

Log files

Log files generated by Kylin is as followings.

`kylin.log`

This file is Kylin's main log file, whose default logging level is DEBUG.

`kylin.out`

The standard output of Kylin process will be redirected to this file, including the output of Tomcat and Hive.

`kylin.gc`

This file is the GC(Garbage Collection) log of Kylin Java process. And it appends pid as filename suffix to avoid being overwritten.

`access.log`

This file stores the Tomcat access log. It records all HTTP request response information.For example, User-Agent, access URL, etc.

`jstack.timed.log`

This file records Java stack traces of Java threads of Kylin, which is used to record some threads running status. To avoid the storage overused, only 20 log files can be kept. The new file will replace the oldest one, when exceeding the maximum number.

Note:Because the execution of jstack depends on the .java_pid file written by jvm in the /tmp directory, if the file is deleted (for example, a scheduled clean up script), it will cause the jstack to not run properly, so that the jstack.timed .log will not be generated

`check-env.out`

The standard output of executing check-env.sh script will be redirected to this file.

`check-env.error`

The error message of executing check-env.sh script will be redirected to this file.

`shell.stderr`

The result of running command lines will be stored in this file.

`shell.stdout`

The standard output of running command lines will be redirected to this file.

`kylin.security.log`

The log of system start, stop, upgrade, login and logout will be redirected to this file.

Notes: When using LDAP service to implement user authentication, two logs will be recorded for each login failure. Because when using the LDAP service, if the login fails, another method will be used for authentication.

`kylin.schedule.log`

This file records logs related to task scheduling, whose default logging level is DEBUG.

`kylin.query.log`

This file records query related logs, whose default logging level is DEBUG.

`kylin.build.log`

This file records build-related logs, whose default logging level is DEBUG.

`kylin.metadata.log`

This file records metadata and transaction operations related logs, whose default logging level is DEBUG.

`dump.hprof`

When Out of Memory (OOM) occurs in Kylin, it will dump the entire heap, which is convenient for checking the cause.

Note: When you have a large memory setting and Out of Memory OOM occurs, the file dump.hprof will occupy a large storage space, which may cause your disk space to be insufficient and the node to be abnormal. You can manually clean up the historical file.

Logging Analysis

Take query as an example, submitting a query on Web UI, and we'll see the following information in kylin.query.log

==========================[QUERY]===============================
Query Id: 8586e718-67b4-c840-61b4-a8898415a154
SQL: select lo_revenue as from p_lineorder;
User: ADMIN
Success: true
Duration: 1.243
Project: ssb100_10
Realization Names: [P_LINEORDER_1]
Index Layout Ids: [30001]
Snapshot Names: []
Is Partial Match Model: [false]
Scan rows: [35000]
Total Scan rows: 35000
Scan bytes: [246530]
Total Scan Bytes: 246530
Result Row Count: 280
Shuffle partitions: 1
Hit Exception Cache: false
Storage Cache Used: false
Storage Cache Type: null
Is Query Push-Down: false
Is Prepare: false
Is Timeout: false
Time Line Schema: massage,end calcite parse sql,end_convert_to_relnode,end_calcite_optimize,end_plan,collect_olap_context_info,end select realization,end_rewrite,to_spark_plan,seg_pruning,fetch_file_status,shard_pruning,executed_plan,collect_result
Time Line: 6,1,4,11,0,0,1,1,14,6,0,0,1,1198
Message: null
Is forced to Push-Down: false
User Agent: null
Scan Segment Count: 1
Scan File Count: 1
Is Refused: false
==========================[QUERY]===============================

The main fields in the above clip are described as follows:

Query Id： Query id
SQL： Query statement
User： The user name to execute the query
Success： Status flag of query result, true execution succeeded, false execution failed
Duration： Query time (unit: seconds)
Project： The name of the project used in the query
Realization Names： The name of the model hit by the query
Index Layout Ids： ID of the layout hit by the query
Snapshot Names： Query the hit snapshot
Is Partial Match Model： Partial match model, such as a left B, you can check table a alone
Scan rows： Query the number of data rows scanned
Total Scan rows： Total rows of data scanned by query
Scan bytes： Query the number of data bytes scanned
Total Scan Bytes ： Query the total number of bytes of scanned data
Result Row Count： The number of data rows returned by the query
Shuffle partitions： A spark query parameter that affects how many partitions / tasks are generated after a shuffle. It is calculated by kylin.query.engine.sparkl-sql-shuffle-parittions or dynamic calculation. The calculation formula is min (the estimated value of data size and the total number of cores of spark cluster)
Hit Exception Cache： Whether to hit the cache of failed queries
Storage Cache Used： Whether to hit the cache successfully queried
Storage Cache Type： The cache type of the hit query
Is Query Push-Down： Is it a push down query
Is Prepare： Whether it is a probe query (this item will be true for the query sent by BI)
Is Timeout： Whether to timeout
Time Line Schema： Steps in query module
Time Line： Time spent in each step of the query module (MS)
Message：Query the prompt information on the page. The query is successful. This item is null
Is forced to Push-Down： Whether to force down
User Agent： The environment information used to submit the query
Scan Segment Count： Number of scanned segments
Scan File Count： Number of scanned files
Is Refused: Whether to refuse to query

Logging Configuration

Kylin uses log4j2 to configure logs. Users can edit the kylin-server-log4j.xml file in the $KYLIN_HOME/server/conf/ directory to modify the log level, path, etc. After modification, you need to restart Kylin for the configuration to take effect.

The configuration of all logs starting with kylin and ending with log is in kylin-server-log4j.xml, the configuration code is as follows.

        <Routing name="routing">
            <Routes pattern="$${ctx:logCategory}">
                <Route>
                    <RollingFile name="rolling-${ctx:logCategory}"
                                 fileName="${env:KYLIN_HOME}/logs/kylin.${ctx:logCategory}.log"
                                 filePattern="${env:KYLIN_HOME}/logs/kylin.${ctx:logCategory}.log.%i">
                        <Policies>
                            <SizeBasedTriggeringPolicy size="268435456"/>
                        </Policies>
                        <DefaultRolloverStrategy max="10"/>
                        <PatternLayout pattern="%d{ISO8601} %-5p %X{request.project}[%t] %c{2} : %mask{%m}%n"/>
                    </RollingFile>
                </Route>

                <Route ref="server" key="$${ctx:logCategory}"/>
            </Routes>
        </Routing>

In the default configuration, log rolling is triggered when the log file reaches 256MB, keeping the last 10 log files.

If you need to configure one of the log files (such as kylin.query.log) separately, you need to add a new Route under the Routes configuration in the above configuration code, and configure the key as the corresponding log file name (query, schedule). It should be noted that the new route needs to be configured before the existing route, otherwise it will not take effect.

The following is an example, modify the rolling strategy of kylin.query.log to trigger at 0:00 every day, back up the last 5 logs.

<Route key="query">
    <RollingFile name="rolling-${ctx:logCategory}" fileName="${env:KYLIN_HOME}/logs/kylin.${ctx:logCategory}.log" filePattern="${env:KYLIN_HOME}/logs/kylin.${ctx:logCategory}.log.%i">
        <Policies>
            <CronTriggeringPolicy schedule="0 0 0 * * ?"/>
        </Policies>
        <DefaultRolloverStrategy max="5" />
        <PatternLayout pattern="%d{ISO8601} %-5p %X{request.project}[%t] %c{2} : %mask{%m}%n" />
    </RollingFile>
</Route>

If you need to configure kylin.log, you can modify the RollingRandomAccessFile configuration, for example, change the number of reserved files to 5

<RollingRandomAccessFile name="server" fileName="${env:KYLIN_HOME}/logs/kylin.log" append="true"
                         filePattern="${env:KYLIN_HOME}/logs/kylin.log.%i" immediateFlush="false" >
    <Policies>
        <SizeBasedTriggeringPolicy size="268435456"/>
    </Policies>
    <DefaultRolloverStrategy max="5"/>
    <PatternLayout pattern="%d{ISO8601} %-5p %X{request.project}[%t] %c{2} : %mask{%m}%n"/>
</RollingRandomAccessFile>

Error Code In Log

The format of log error code is KE-AABBBCCC, AA refers to the error reporting module, BBB refers to the more detailed business error reporting and CCC refers to the error number.

AA	Description
00	common
10	server
20	query
30	build
40	system
50	tool

BBB	Description
000	general
001	project
002	model
003	user
004	user group
005	password
006	column
007	table
008	database
009	measure
010	dimension
011	cc
012	index
013	job
014	sql expression
015	license
016	email
017	file
018	kerberos
019	catalog
020	recommendation
021	server
022	segment
023	diag
024	auth
025	shell
026	metadata
027	frequency query
028	json

Log files​

kylin.log​

kylin.out​

kylin.gc​

access.log​

jstack.timed.log​

check-env.out​

check-env.error​

shell.stderr​

shell.stdout​

kylin.security.log​

kylin.schedule.log​

kylin.query.log​

kylin.build.log​

kylin.metadata.log​

dump.hprof​

Logging Analysis​

Logging Configuration​

Error Code In Log​