Date Published: October 23, 2017

Spark and Qlik Integration

Steps:

  • Start Spark Thrift Server on Datastax Cluster
$ dse -u cassandra -p <password> spark-sql-thriftserver start --conf spark.cores.max=4 --conf spark.executor.memory=2G --conf spark.driver.maxResultSize=1G --conf spark.kryoserializer.buffer.max=512M --conf spark.sql.thriftServer.incrementalCollect=true
  • Enable Qlik Server’s Security Group on AWS to access port 10000 (basically from qlik, need to connect to thrift server port 10000)
  • Install Simba ODBC Driver for Spark on the Qilk Windows EC2 Instance
    Create System DSN as follows:
Spark Server Type:SparkThriftServer
Host:internal-spark-thriftserver-prod-lb-861234576.ap-southeast-1.elb.amazonaws.com (DNS name of spark thrift server ELB)
Port:10000
Database:avm_analytics
Authentication Mechanism:Username
Thrift Transport:SASL
  • Now go to Qlik Admin UI -> Data Connections, click on above DSN, it gets connected
  • In the Data Editor, give below to execute query
LIB connect TO ‘Simba Spark(Qlik-sense-administration)’  
select txn_id,txn_date from transactions where txn_date>=‘2017-06-05’ and txn_date<‘2017-06-06’  
  • Observe the execution of spark job in Spark Web UI

Comments

Write a Reply or Comment

Your email address will not be published. Required fields are marked *