Kong doesn't keep track of changing Cassandra cluster nodes.

[angrosz]

Hi,
I'm working with Kong 0.13.1 that uses lua-cassandra 1.2.3 and OpenResty 1.11.2.5. Kong talks to Cassandra 3.11.2. My problem is that Kong doesn't keep track of changing Cassandra cluster nodes.
Details are following:

1. One or two nodes of the Cassandra cluster are set in the Kong configuration.
2. At the startup Kong gets complete topology, i.e. addresses of all nodes, from Cassandra.
3. Once Kong is done with its initialization procedure it doesn't get information of changing Cassandra nodes. The possible changes are: some nodes are removed from the cluster, some added.
   a. If a node is added, then Kong will never know that and will never try to connect to the new node.
   b. If a node is removed, then Kong will try to connect to it infinitely. The number of nodes "visible" to Kong will be reduced, what in consequence will lead to performance degradation.
   c. If restart of node causes change of the IP then the result is as one node is removed and another added.
   I have to cover the following scenarios:
4. Cassandra node is down.
5. A new node is stared.
6. A node is re-started (IP changes).
7. Scaling down the Cassandra cluster (number of nodes decreased - some nodes are down).
8. Scaling up the Cassandra cluster (number of nodes increased - new nodes started).

My question is: do you plan to enhance lua-cassandra driver to keep track of Cassandra cluster nodes changes (as I described above)?

Best Regards

[thibaultcha]

Contributions to this driver aiming at supporting cluster topology events are welcome. So would be improving refresh() to support calls at runtime (see Yelp/casper#47), which would be a nice first step.

[nbisson]

Same issue. News ?

[thibaultcha]

@nbisson No news on this front. Contributions still welcome.

[suankan]

We are suffering from this problem too.

As a workaround we had to come up with a very ugly solution:

• Each Kong runs with custom nginx server block:

server {
  server_name kong-reload;
  listen 0.0.0.0:8333;

  location /kong-reload {
    content_by_lua_block {
      os.execute("/usr/local/bin/kong reload")
    }
  }

  access_log /dev/stdout;
  error_log /dev/stderr error;
}

That allows to call endpoint "/kong-reload" on the Kong node and that executes command /usr/local/bin/kong reload which under the hood causes sending SIGHUP to Kong nginx process. As a result Kong reloads its configuration and re-discovers Cassandra cluster members.

• The second part of the solution is to have what we call "cassandra-watcher" script which infinitely runs nodetool status, parces output and keeps track of IPs of your current Cassandra cluster memebers. Once it realises that the list of Cassandra IPs becomes different - it calls http://kong-node:8333/kong-reload .

Here is example of my script.

#!/bin/bash
# This script is to be used as an entrypoint for Cassandra-watcher.
# It is running infinite loop with periodically checking if
# the list of Cassandra cluster IPs has changed.
# If the list changed, then it calls Kong reload HTTP endpoint.
# If the list isn't changed it does nothing and waits for the next check.
# Cassandra container with this script as antrypoint should be running as
# a side-car container in the POD that runs Kong.

# Obtains list of Cassandra cluster IPs and:
# - stores list in provided file location
# - returns list as resul
# Params:
# 1. File to save the list.
get_cassandra_ips() {
  IPS_FILE=$1

  nodetool -h cassandra status | grep UN | awk '{print $2}' | tee $IPS_FILE
}

# Compare two files.
# Params:
# 1. file to compare
# 2. file to compare
# Returns usual return code of diff tool which is:
# - 0 if files are identical
# - 1 if files are different
if_file_lists_same() {
  file_list1=$1
  file_list2=$2
  # Use standard diff tool to check if lists are different.
  # Soft both files before diff comparision.
  diff -q <(sort $file_list1) <(sort $file_list2) > /dev/null 2>&1
}

get_time() {
  date "+%Y-%m-%d %H:%M:%S"
}

# In the beginning we initialise PREV_IPS with list of current IPs
PREV_IPS=/tmp/prev_cluster_ips
get_cassandra_ips $PREV_IPS

CURRENT_IPS=/tmp/current_cluster_ips

# Expose ability to configure Cassandra discovery period in seconds
# via K8s container env variable. If such env var is defined and not
# empty via K8s container env variable then use that value.
# Defaults to 60 seconds.
CASSANDRA_DISCOVERY_PERIOD=${CASSANDRA_DISCOVERY_PERIOD:-60}

while [ true ]
do
  echo "$(get_time): Waiting $CASSANDRA_DISCOVERY_PERIOD seconds to check Cassandra IPs again."
  sleep $CASSANDRA_DISCOVERY_PERIOD

  echo "$(get_time): Checking current Cassandra cluster ips:"
  get_cassandra_ips $CURRENT_IPS

  # Special EGDE case!
  # If the current list of Cassandra IPs is empty - do not reload Kong!
  if [ ! -s $CURRENT_IPS ]
  then
    echo "$(get_time): Unable to detect any Cassandra cluster IPs. Not going to reload Kong."
    continue
  # If current list of IPs is different from previous list
  # then at least one Cassandra nodes restarted and we should:
  # - SIGHUP Kong
  # - Save current list of IPs as previous
  elif ! if_file_lists_same $CURRENT_IPS $PREV_IPS
  then
    echo "$(get_time): Detected Cassandra IPs change!"
    echo "$(get_time): Reloading Kong"
    curl http://127.0.0.1:8333/kong-reload
    mv $CURRENT_IPS $PREV_IPS
  else
    echo "$(get_time): Cassandra IPs are the same."
  fi

done

The solution has a lot of drawbacks and requires maintenance.

Ideally Kong should be able to rediscover Cassandra cluster members via this cassandra db driver.