Part Two: Multi-cluster Cassandra deployment with Google Kubernetes Engine
Author: Jeff Carpenter
This is the second post in a series that examines patterns for using K8ssandra to create Cassandra clusters with different deployment topologies.
In the first post, we walked you through how to deploy a multi-datacenter Apache Cassandra® cluster in Kubernetes. The rest of this series will explore additional configurations that promote high availability and accessibility of your data across various different network topologies, including hybrid and multi-cloud deployments.
This second post will focus on creating a Cassandra cluster running on Kubernetes (K8s) clusters in multiple regions — all within a single cloud provider (Google Cloud). If you worked through the first blog post, many of the steps in this post will be familiar.
Note: For this exercise, you’ll create Google Kubernetes Engine (GKE) clusters in two separate regions under the same Google Cloud project. This will make it possible to use the same network.
1. Prepare the first GKE Cluster
First, you’re going to need a Kubernetes cluster in which you can create the first Cassandra datacenter. To create this first cluster, follow the instructions for K8ssandra on (GKE). These instructions reference scripts provided as part of the K8ssandra GCP Terraform example.
When building this example, I provided values for the environment variables used by the Terraform script to match my desired environment. Notice my initial GKE cluster is in the us-west4 region. You’ll want to change these values for your own environment.
After creating the GKE cluster, you can ignore further instructions on the K8ssandra GKE docs page (the “Install K8ssandra” section and beyond), since you’ll be doing a custom K8ssandra installation. The Terraform script should automatically change your kubectl context to the new cluster, but you can make sure by checking the output of kubectl config current-context.
2. Create the first Cassandra datacenter
Let’s do a bit of upfront planning. It’ll be easier to manage our K8ssandra installs in different clusters if we use the same administrator credentials in each data center. So, we’ll create a namespace for the first data center and add a secret within the namespace:
Note that I chose to create a namespace matching the GCP region in which I’m deploying K8ssandra. This is done as part of enabling DNS between the GKE clusters, which is a topic that we’ll discuss in-depth in a future post. For now, you’ll want to specify a namespace corresponding to the region you’re using.
The next step is to create a K8ssandra deployment for the first data center. You’ll need Helm installed for this step, as described on the K8ssandra GKE docs page. Create the configuration for the first datacenter in a file called dc1.yaml, making sure to change the affinity labels to match zones used in your GKE cluster:
Aside from requesting three nodes in the data center, this configuration also specifies an appropriate storage class for the GKE environment (standard-rwo), and uses affinity to specify how the racks are mapped to GCP zones. Make sure to change the referenced zones to match your configuration. For more details, check the first blog post in this series.
Now, deploy the release using this command:
This causes the K8ssandra release named k8ssandra to be installed in the namespace us-west4.
As would be the case for any Cassandra cluster deployment, you’ll want to wait for the first data center to be completely up before adding a second data center. As you’ll now be creating additional infrastructure for the second data center, you probably don’t need to wait. But, if you’re interested, one simple way to do make sure the data center is up is to watch until the Stargate pod shows as initialized since it depends on Cassandra being ready:
This is a great time to get some information you’ll need to configure the second Cassandra datacenter: seeds. In the first blog post in this series, we took advantage of a headless Kubernetes service that K8ssandra creates called the seed service, which points to a couple of the Cassandra nodes that can be used to bootstrap new nodes or datacenters into a Cassandra cluster. You can take advantage of the fact that the seed nodes are labeled to find their addresses.
Which produces output that looks like this:
10.56.6.8 10.56.5.8 10.56.4.7Record a couple of these IP addresses to use as seeds further down.
3. Prepare the second GKE cluster
Now you’ll need a second Kubernetes cluster that you’ll use to host the second Cassandra datacenter. The terraform scripts you used before to create the first GKE cluster, also create a network and service account that should be reused for the second cluster. Instead of modifying the Terraform scripts to take existing resources into account, you can create the new GKE cluster using the console or the gcloud command line.
For example, I chose the us-central1 region for my second cluster. First, create a subnet in that region as part of the same network that Terraform built for the first datacenter.
Then create the second GKE cluster using that network and the same compute specs as the first cluster:
Next, change the kubectl context to the second data center. You can find a command to do this by selecting the cluster in the GCP console and pressing the “Connect” button.
Lastly, you’ll need to create a firewall rule to allow traffic between the two clusters. I obtained the IP space of each subnet and the IP space of each GKE cluster and created a rule to allow all traffic:
If you want, you can create a more targeted rule to only allow TCP traffic between ports used by Cassandra.
4. Add a second Cassandra datacenter
Let’s start by creating a namespace for the new data center matching the GCP region name. We also need to create administrator credentials to match those created for the first data center, since the secrets aren’t automatically replicated between clusters.
Now you’ll create a configuration to deploy an additional Cassandra datacenter dc2 in the new GKE cluster. For the nodes in dc2 to join the Cassandra cluster, there are a few steps you need to do:
- Use the same Google Cloud network for both GKE clusters. This means the nodes in the new data center will be able to communicate with nodes in the original data center. Happily enough, you’ve already taken care of this one.
- Use the same Cassandra cluster name as for the first datacenter.
- Provide the seed nodes you recorded earlier so the nodes in the new data center know how to contact nodes in the first data center to join the cluster.
Now it’s time to create a configuration in a file called dc2.yaml. Below is what my file looked like. You’ll want to change the additional seeds and affinity labels to your configuration.
Similar to the configuration for dc1, this configuration also uses affinity. A similar allocation of racks can be used to make sure Cassandra nodes are evenly spread across the remaining workers. You can deploy the release using a command such as this:
If you look at the resources in this namespace using a command such as kubectl get services,pods you’ll note that there are a similar set of pods and services as for dc1, including Stargate, Prometheus, Grafana, and Reaper. Depending on how you want to manage your application, this may or may not be to your liking, but you’re free to tailor the configuration to disable any components you don’t need.
5. Configure Cassandra keyspaces
Once the second data center comes online, you’ll want to configure Cassandra keyspaces to replicate across both clusters.
Important: You’ll likely need to change your
kubectlcontext back to the first GKE cluster. For example, using thekubectl config use-contextcommand. You can list existing contexts usingkubectl config get-contexts.
To update keyspaces, connect to a node in the first datacenter and execute cqlsh :
Use the DESCRIBE KEYSPACES to list the keyspaces and DESCRIBE KEYSPACE <name> command to identify those using the NetworkTopologyStrategy. For example:
Typically, you’ll find that the system_auth, system_traces, and system_distributed keyspaces use NetworkTopologyStrategy, as well as data_endpoint_auth if you’ve enabled Stargate. You can then update the replication strategy to ensure data is replicated to the new data center.
Now you can execute something like the following for each of these keyspaces:
Important: Remember to create or alter the replication strategy for any keyspaces you need for your application, so that you have the desired number of replicas in each datacenter.
After exiting cqlsh, make sure existing data is properly replicated to the new data center with the nodetool rebuild command.
Important: Remember to change your
kubectl contextback to the second GKE cluster.
Run rebuild on each node in the new data center, for example:
Repeat for the other nodes multi-region-dc2-rack2-sts-0 and multi-region-dc2-rack3-sts-0.
6. Test the configuration
Let’s verify the second data center has joined the cluster. To do this, pick a Cassandra node to execute the nodetool status command against.
Execute the nodetool command against the node:
This will produce output similar to the following:
If everything has been configured correctly, you’ll be able to see both datacenters in the cluster output. Here’s a diagram that illustrates what you’ve just deployed, focusing on the Cassandra nodes and networking:
What’s next?
In our upcoming posts, we’ll explore additional multi-datacenter topologies across multiple Kubernetes clusters, including Cassandra clusters in hybrid cloud and multi-cloud deployments. We’ll also dive into more detail on networking and DNS configuration.
In the meantime, we’d love to hear your ideas for additional configurations you build and feel free to reach out with any questions on the K8ssandra forum or on our K8ssandra Community Discord channel.
Follow DataStax on Medium for exclusive posts on all things Cassandra, streaming, Kubernetes, and more. To join the conversation with developers from around the world, follow DataStaxDevs on Twitter.
This post was originally published on K8ssandra.
