Prerequisite

Kylin Real-Time supports querying real-time streaming data, which achieves lower latency from data loading to query.

This article introduces hardware and software requirements for Kylin Real-Time.

Network requirements

Connection between Kylin and Kafka clusters is enabled.

info

If there's no DNS server in LAN, you need to manually add the IP address and Hostname of Kafka nodes to the /etc/hosts file of the Kylin server.

Kylin and Kafka versions

Product	Version
Kylin	5.0 or higher version
Kafka	0.11.0.X or higher version (manual deployment required)

Other requirements

• The time across Kylin and Kafka clusters are synchronized. On how to change the time for Kylin, see Basic Configuration.

• When authentication mechanism is enabled on Hadoop or Kafka cluster, further configurations are needed:

When Kerberos is enabled on Hadoop and Kafka

Note

Make sure the Kerberos ticket to use is still valid during index building, or Kafka data cannot be consumed.

1. Add the following Kafka configuration settings In the kylin.properties file.

   kylin.kafka-jaas.enabled=true
   kylin.streaming.kafka-conf.security.protocol=SASL_PLAINTEXT
   kylin.streaming.kafka-conf.sasl.mechanism=GSSAPI
   kylin.streaming.kafka-conf.sasl.kerberos.service.name=kafka

2. Create Kafka authentication file in path $KYLIN_HOME/conf/kafka_jaas.conf with the following configuration.

   KafkaClient {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=false
        useTicketCache=true
        serviceName="${serviceName}";
   };

   ${serviceName}: for example, kafka

When Kerberos is enabled on Hadoop while SASL/PLAIN is enabled on Kafka

1. Add the following Kafka configuration in the kylin.properties file.

   kylin.kafka-jaas.enabled=true
   kylin.streaming.kafka-conf.security.protocol=SASL_PLAINTEXT
   kylin.streaming.kafka-conf.sasl.mechanism={mechanism}

   {mechanism}: encryption algorithm, for example, PLAIN or SCRAM-SHA-256

2. Create Kafka authentication file in path $KYLIN_HOME/conf/kafka_jaas.conf with the following configuration.

   KafkaClient {
      {LoginModule} required
      username="{username}"
      password="{password}";
   };
   • {LoginModule}: class name of the login module, for example, org.apache.kafka.common.security.scram.ScramLoginModule
   • {username}: Kafka username
   • {password}: Kafka password