Model Management
Kylin provides REST APIs on checking model information, index build and model management.
- On how authentication works, see Access and Authentication REST API.
- On Curl command line, don't forget to quote the URL if it contains the special char
&
.- On Curl command line, escape
%25
if it contains the special char%
.
Create Model
-
POST http://host:port/kylin/api/models
-
Request Permission: MANAGEMENT permission and above
-
HTTP Header
Accept: application/vnd.apache.kylin-v4-public+json
Accept-Language: cn
Content-Type: application/json;charset=utf-8
-
HTTP Body: JSON Object
project
-required
string
, project namefact_table
-required
string
, fact table nameuuid
-optional
string
, model ID, default auto generatedalias
-required
string
, model alias namemanagement_type
-required
string
, for model creation please use: MODEL_BASEDsimplified_measures
-required
JSON Object[]
, measures-
name
-required
string
, measure name -
expression
-required
string
, function, including: SUM,MIN,MAX,TOP_N,COUNT,COUNT_DISTINCT,CORR,PERCENTILE_APPROX,COLLECT_SET -
parameter_value
-required
JSON Object[]
, measure parameterstype
-required
string
, parameter type: constant,columnvalue
-required
int|string
, parameter value (Set value 1 when 'expression' is 'COUNT' and 'type' is 'constant', while others should be: TABLE.COLUMN)
-
return_type
-required
string
, function return type with arguments including: topn(10)、topn(100)、percentile(100) etc., using empty string("") when no return type neededSupported return_type values:
TOP_N:
* Top 10: topn(10)
* Top 100: topn(100)
* Top 1000: topn(1000)
COUNT_DISTINCT:
* Error Rate < 9.75%: hllc(10)
* Error Rate < 4.88%: hllc(12)
* Error Rate < 2.44%: hllc(14)
* Error Rate < 1.72%: hllc(15)
* Error Rate < 1.22%: hllc(16)
* Precisely: bitmap
PERCENTILE_APPROX:
* percentile(100)
* percentile(1000)
* percentile(10000) -
comment
-optional
string
, comments
-
simplified_dimensions
-required
JSON Object[]
, dimensions infocolumn
-required
string
, format: TABLE.COLUMNname
-required
string
, column alias, may set the same as column namestatus
-required
string
, fixed value: DIMENSION
computed_columns
-optional
JSON Object[]
, computed columns infocolumnName
-required
string
, the new column nameexpression
-required
string
, expressiondatatype
-required
string
, data type: VARCHAR,INT,BIGINT ...standard sql typestableIdentity
-required
string
, format: SCHEMA.TABLEtableAlias
-required
string
, table alias
join_tables
-required
JSON Object[]
, join infotable
-required
string
, table namejoin
-required
JSON Object
, join info-
type
-required
string
, join type: INNER,LEFT -
foreign_key
-required
string[]
, foreign key -
primary_key
-required
string[]
, primary key -
simplified_non_equi_join_conditions
-optional
JSON Object
, non-equivalent join conditions(note1: The support of this settings should have 'Support History Table' enabled in advance. Seeing Slowly Changing Dimension
(note2: Join relationship >= and < must be used in pairs, and same column must be joint in both conditions)
foreign_key
-string
, foreign keyprimary_key
-string
, primary keyop
-string
, join relationship: LESS_THAN,GREATER_THAN_OR_EQUAL
-
kind
-optional
string
, table kind: FACT,LOOKUP, default: LOOKUPalias
-optional
string
, aliasflattenable
-required
string
, precomputing associations(precomputing: flatten, no-precomputing: normalized), flatten is suggestedjoin_relation_type
-optional
string
, join type: MANY_TO_ONE,MANY_TO_MANY, default: MANY_TO_ONE
partition_desc
-optional
JSON Object
, partition columns infopartition_date_column
-required
string
, partition date column, format: TABLE.COLUMNpartition_date_format
-required
string
, partition date column format, including: yyyy-MM-dd, yyyyMMdd... Supported date format please check "Model Design Basics"partition_type
-optional
string
, partition type, including: APPEND, default: APPEND
owner
-optional
string
, the owner of model, default current userdescription
-optional
string
, model descriptioncapacity
-optional
string
, model capacity, including: SMALL,MEDIUM,LARGE, default: MEDIUMfilter_condition
-optional
string
, data filter condition. Data filter condition is an additional data filter during data loading.with_base_index
-optional
boolean
, adding base indexes or not, including base aggregate index and base table index. Base indexes include all dimensions and measures of the model and automatically update as the model changes by default. default: false; It is recommended to usebase_index_type
after version 4.6.6,with_base_index
does not take effect after configuringbase_index_type
base_index_type
-optional
Array[String]
, select the base index type to be added, optional valuesBASE_AGG_INDEX
,BASE_TABLE_INDEX
, effective version: 4.6.6[BASE_AGG_INDEX, BASE_TABLE_INDEX]
Both the base aggregate index and the base table index are added[BASE_AGG_INDEX]
only add the base aggregate index[BASE_TABLE_INDEX]
only add the base table index[]
does not add base indexes
-
Curl Request Example
curl -X POST \
'http://host:port/kylin/api/models' \
-H 'Accept: application/vnd.apache.kylin-v4-public+json' \
-H 'Accept-Language: cn' \
-H 'Authorization: Basic QURNSU46S1lMSU4=' \
-H 'Content-Type: application/json;charset=utf-8' \
### Stringify the following JSON Object when use
-d '{
"project": "pj01",
"fact_table": "SSB.P_LINEORDER",
"alias": "test_model_01",
"management_type": "MODEL_BASED",
"simplified_measures": [
{
"name": "COUNT_ALL",
"expression": "COUNT",
"parameter_value": [
{
"type": "constant",
"value": 1
}
],
"return_type": ""
}
],
"simplified_dimensions": [
{
"column": "P_LINEORDER.LO_ORDERKEY",
"name": "LO_ORDERKEY",
"status": "DIMENSION"
},
{
"column": "P_LINEORDER.LO_CUSTKEY",
"name": "LO_CUSTKEY",
"status": "DIMENSION"
},
{
"column": "P_LINEORDER.LO_ORDERDATE",
"name": "LO_ORDERDATE",
"status": "DIMENSION"
},
{
"column": "P_LINEORDER.LO_ORDERPRIOTITY",
"name": "LO_ORDERPRIOTITY",
"status": "DIMENSION"
},
{
"column": "P_LINEORDER.V_REVENUE",
"name": "V_REVENUE",
"status": "DIMENSION"
},
{
"column": "DATES.D_YEAR",
"name": "D_YEAR",
"status": "DIMENSION"
},
{
"column": "CUSTOMER.C_NAME",
"name": "C_NAME",
"status": "DIMENSION"
},
{
"column": "CUSTOMER.C_PHONE",
"name": "C_PHONE",
"status": "DIMENSION"
}
],
"join_tables": [
{
"table": "SSB.DATES",
"join": {
"type": "LEFT",
"foreign_key": [
"P_LINEORDER.LO_ORDERDATE"
],
"primary_key": [
"DATES.D_DATE"
]
},
"alias": "DATES",
"flattenable": "flatten",
"join_relation_type": "MANY_TO_ONE"
},
{
"table": "SSB.CUSTOMER",
"join": {
"type": "LEFT",
"foreign_key": [
"P_LINEORDER.LO_CUSTKEY"
],
"primary_key": [
"CUSTOMER.C_CUSTKEY"
]
},
"alias": "CUSTOMER",
"flattenable": "flatten",
"join_relation_type": "MANY_TO_ONE"
}
],
"with_base_index": true,
"base_index_type": ["BASE_AGG_INDEX", "BASE_TABLE_INDEX"]
}' -
Response Details
code
-string
, response code, succeed:000
, failed:999
msg
-string
, response messagedata
-JSON Object
, response database_table_index
-JSON Object
, base index infodimension_count
-int
, dimension countmeasure_count
-int
, measure countlayout_id
-long
, layout idoperate_type
-string
, operation type, including: UPDATE,CREATE
base_agg_index
-JSON Object
, base aggregation index info, same structure asbase_table_index
warn_code
-string
, warning code message
-
Response Example
{
"code": "000",
"msg": "",
"data": {
"base_table_index": {
"dimension_count": 8,
"measure_count": 0,
"layout_id": 20000000001,
"operate_type": "CREATE"
},
"base_agg_index": {
"dimension_count": 8,
"measure_count": 1,
"layout_id": 1,
"operate_type": "CREATE"
},
"warn_code": null
}
} -
Failed Response Example
{
"code": "999",
"data": null,
"msg": "KE-010001002(Empty Project Name):Can’t find project information. Please select a project.",
"stacktrace": "KE-010001002(Empty Project Name) \norg.apache.kylin.common.exception.KylinException: KE-010001002(Empty Project Name):Can’t find project information. ...",
"exception": "KE-010001002(Empty Project Name):Can’t find project information. Please select a project.",
"url": "http://host:port/kylin/api/models"
} -
Error Code (Specific error message please check real api return.)
KE-010001001
: Project Not ExistKE-010001002
: Empty Project NameKE-010006002
: Invalid Partition ColumnKE-010000003
: Invalid ParameterKE-010000002
: Invalid RangeKE-010000004
: Invalid NameKE-010006005
: Timestamp Column Not ExistKE-010002010
: Failed to Add ModelsKE-010011001
: Duplicate Computed Column NameKE-010007001
: Table Not Exist
Get Model List
Call this API to get a list of models under specified project that satisfies certain conditions.
-
GET http://host:port/kylin/api/models
-
URL Parameters
project
-required
,string
, project name.page_offset
-optional
,int
, offset of returned result,0
by default.page_size
-optional
,int
, quantity of returned result per page,10
by default.status
-optional
,string
, model status.model_name
-optional
,string
, model name.exact
-optional
,boolean
, whether exactly match the model name,true
by default.
-
HTTP Header
Accept: application/vnd.apache.kylin-v4-public+json
Accept-Language: cn
Content-Type: application/json;charset=utf-8
-
Curl Request Example
curl -X GET \
'http://host:port/kylin/api/models?project=doc_expert' \
-H 'Accept: application/vnd.apache.kylin-v4-public+json' \
-H 'Accept-Language: cn' \
-H 'Authorization: Basic QURNSU46S1lMSU4=' \
-H 'Content-Type: application/json;charset=utf-8' -
Response Details
Code
:String
, response code. Returned value:000
(request processing success),999
(request processing failed)data
:JSON
, returned datavalue
: details of the returned data, which consists of:name
:String
, model namelookups
:JSON
, detailed list of all dimension tablessize_kb
:Long
, total size of all segments of the modelsinput_records_count
:Long
, number of flattened tables within all segmentsinput_records_size
:Long
, size of flattened tables within all segmentsproject
:String
, project nameuuid
:String
, model IDlast_modified
:Long
, last modified time of the modelcreate_time
:Long
, model creation timemvcc
:Long
, version number with metadata modifiedalias
:String
, model aliasowner
:String
, model ownerconfig_last_modifier
:String
, last user who modified the configurationfact_table
:String
, fact table (one model contains only one fact table)fact_table_alias
:String
, fact table aliasjoin_tables
:JSON
, joined tablespartition_desc
:JSON
, partition columnall_measures
:JSON
, all measures within the modelmulti_partition_desc
:JSON
, multiple partitionscomputed_columns
:JSON
, computed columnscanvas
:JSON
, position of model canvasstatus
:String
, model status; Returned value:online
,offline
,broken
,warning
last_build_end
:String
, building time of the last segmentstorage
:Long
, total storage size of the model; Unit: bytesource
:Long
, sourceByte sum of all segments in the modelexpansion_rate
:String
, model expansion rate; Unit: %usage
:Long
, queried times of the modelmodel_broken
:Boolean
, if the model is broken or notroot_fact_table_deleted
:Boolean
, if the fact table is deletedsegments
:JSON
, segment listsrecommendations_count
:Integer
, number of recommendationssimplified_measures
:JSON
, measure list of the modelssimplified_dimensions
:JSON
, dimension list of the modelssimplified_tables
:JSON
, all tables in the model
offset
: page numberlimit
: models listed in each pagetotal_size
: total number of models
-
Response Example
{
"code":"000",
"data":{
"value":[
{
"name":"Model_03",
"lookups":Array[1],
"is_streaming":false,
"size_kb":0,
"input_records_count":0,
"input_records_size":0,
"project":null,
"uuid":"0464241b-fd7d-49a9-a3c9-b4f7e32cf489",
"last_modified":1574750372949,
"create_time":1574761225505,
"version":"4.0.0.0",
"mvcc":4,
"alias":"Model_03",
"owner":"ADMIN",
"config_last_modifier":null,
"config_last_modified":0,
"description":"",
"fact_table":"SSB.LINEORDER",
"fact_table_alias":null,
"management_type":"MODEL_BASED",
"join_tables":Array[1],
"filter_condition":"",
"partition_desc":Object{...},
"capacity":"MEDIUM",
"segment_config":Object{...},
"data_check_desc":null,
"semantic_version":0,
"all_named_columns":Array[26],
"all_measures":Array[2],
"column_correlations":Array[0],
"multilevel_partition_cols":Array[0],
"computed_columns":Array[0],
"canvas":Object{...},
"status":"ONLINE",
"last_build_end":"902016000000",
"storage":24694,
"source":5585164,
"expansion_rate":"0.44",
"usage":0,
"model_broken":false,
"root_fact_table_deleted":false,
"segments":null,
"recommendations_count":0,
"simplified_measures":Array[2],
"simplified_dimensions":Array[9],
"simplified_tables":Array[2],
"multi_partition_desc": {
"columns": ["KYLIN_SALES.LSTG_SITE_ID"],
"partitions": [
{
"id": 0,
"values": [
"1"
]
},
{
"id": 1,
"values": [
"2"
]
}
],
"max_partition_id": 1
}
},
Object{...},
Object{...}
],
"offset":0,
"limit":10,
"total_size":3
},
"msg":""
}