Panic when using a HostFilter and keyspace is not replicated to every DC

[max-melentyev]

Hey, we see this panic

panic: token map different size to token ring: got 0 expected 120
goroutine 415 [running]:
github.com/apache/cassandra-gocql-driver/v2.(*networkTopology).replicaMap(0x25c6b9516810, 0x25c6b96629c0)
	/go/pkg/mod/github.com/apache/cassandra-gocql-driver/v2@v2.1.0/topology.go:391 +0x1274
github.com/apache/cassandra-gocql-driver/v2.(*tokenAwareHostPolicy).updateAllReplicas(0x25c6b8c53ac8?, 0x25c6b87a1de0, 0xd326c0?)
	/go/pkg/mod/github.com/apache/cassandra-gocql-driver/v2@v2.1.0/policies.go:559 +0x112
github.com/apache/cassandra-gocql-driver/v2.(*tokenAwareHostPolicy).schemaRefreshed(0x25c6b8b95800, 0x25c6b8782dc0)
	/go/pkg/mod/github.com/apache/cassandra-gocql-driver/v2@v2.1.0/policies.go:543 +0x90
github.com/apache/cassandra-gocql-driver/v2.refreshSchemas(0x25c6b8694f08)
	/go/pkg/mod/github.com/apache/cassandra-gocql-driver/v2@v2.1.0/metadata.go:865 +0xd71
github.com/apache/cassandra-gocql-driver/v2.NewSession.func1()
	/go/pkg/mod/github.com/apache/cassandra-gocql-driver/v2@v2.1.0/session.go:174 +0x17
github.com/apache/cassandra-gocql-driver/v2.(*refreshDebouncer).flusher(0x25c6b85b3590)
	/go/pkg/mod/github.com/apache/cassandra-gocql-driver/v2@v2.1.0/host_source.go:1009 +0x17d
created by github.com/apache/cassandra-gocql-driver/v2.newRefreshDebouncer in goroutine 414
	/go/pkg/mod/github.com/apache/cassandra-gocql-driver/v2@v2.1.0/host_source.go:945 +0x145

It happens when session is configured with host filter to connect only to hosts in a local dc:

cluster.HostFilter = gocql.HostFilterFunc(func(host *gocql.HostInfo) bool {
	return host.DataCenter() == cfg.LocalDC
})

And there is a keyspace that is not replicated to all DCs:

CREATE KEYSPACE ks1 WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3'}  AND durable_writes = true;
CREATE KEYSPACE ks2 WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3', 'dc2': '3'}  AND durable_writes = true;

UPD. It happens for clients with LocalDC: dc2

[joao-r-reis]

Please provide the version of gocql that you're using. Nevermind I can see in the panic message it's 2.1.0

[worryg0d]

I just did a bit of testing and identified that this is a regression introduced in this commit 3c46f52.

I can reproduce it locally with ccm: ccm create -v 5.0.8 gocql_test3 -n 3:3.

func main() {
	// CREATE KEYSPACE IF NOT EXISTS ks1 WITH replication = { 'class': 'NetworkTopologyStrategy', 'datacenter1': 3 };
	// CREATE KEYSPACE IF NOT EXISTS ks2 WITH replication = { 'class': 'NetworkTopologyStrategy', 'datacenter1': 3, 'datacenter2': 3 };

	const localDC = "dc1"
	const remoteDC = "dc2"

	cfg := gocql.NewCluster("127.0.0.4")
	cfg.PoolConfig.HostSelectionPolicy = gocql.TokenAwareHostPolicy(gocql.RoundRobinHostPolicy())
	cfg.Timeout = 1 * time.Hour
	cfg.ConnectTimeout = 1 * time.Hour
	cfg.HostFilter = gocql.HostFilterFunc(func(host *gocql.HostInfo) bool {
		return host.DataCenter() == remoteDC
	})
	session, err := cfg.CreateSession()
	if err != nil {
		panic(err)
	}
	defer session.Close()

	metadata, err := session.KeyspaceMetadata("ks1")
	if err != nil {
		panic(err)
	}
	fmt.Println(toJSON(metadata))
}

func toJSON(data any) string {
	json, err := json.Marshal(data)
	if err != nil {
		panic(err)
	}
	return string(json)
}

networkTopology.replicaMap method wasn't changed in that commit, so there is something the passed tokenRing networkTopology object. I didn't manage to identify it yet, but I will try to figure out what is going on

[worryg0d]

Before that commit, the driver was creating a replicaMap only for a session-level keyspace, so it didn't panic. With the change, it now computes a replica map for all available keyspaces in the cluster.
The problem is that networkTopology.replicaMap doesn't take into account that tokenRing might not have hosts of DC on which the keyspace is not replicated.

For example, we have a 2-DCS C* cluster: dc1 and dc2.
Here, it counts the number of DCs with replicas of the keyspace. In keyspace metadata, we have only a single replicated dc1, but the driver established connections to dc2 so the tokenRing contains hosts of dc2.
Later, the amount of dcs with replicas dcsWithReplicas == len(dcRacks) with the amount of dcs in tokenRing. Keyspace is only replicated to a single dc1, and tokenRing has hosts of dc2, so it passes that check.

cassandra-gocql-driver/topology.go

Lines 383 to 392 in c65c762

	dcsWithReplicas := 0
	for _, dc := range n.dcs {
	if dc > 0 {
	dcsWithReplicas++
	}
	}
	if dcsWithReplicas == len(dcRacks) && len(replicaRing) != len(tokens) {
	panic(fmt.Sprintf("token map different size to token ring: got %d expected %d", len(replicaRing), len(tokens)))
	}

However, replicaRing is empty because it skipped all hosts in tokenRing as none of them is a replica

cassandra-gocql-driver/topology.go

Lines 292 to 296 in c65c762

	for i, th := range tokenRing.tokens {
	if rf := n.dcs[th.host.DataCenter()]; rf == 0 {
	// skip this token since no replica in this datacenter.
	continue
	}

So it passes the check len(replicaRing) != len(tokens) and we see a panic.

To fix this, we can make it count DCS with replicas only for those DCS the driver is aware of:

	dcsWithReplicas := 0
	for dc, rf := range n.dcs {
		// We should count only DCs that driver is aware of and have a replication factor > 0
		if _, knownDc := dcRacks[dc]; knownDc && rf > 0 {
			dcsWithReplicas++
		}
	}

So it will return an empty replicaRing which should be safe:

cassandra-gocql-driver/policies.go

Lines 710 to 714 in c65c762

	var replicas []*HostInfo
	if ht == nil {
	host, _ := meta.tokenRing.GetHostForToken(token)
	replicas = []*HostInfo{host}
	} else {

[worryg0d]

I submitted a PR #1948 with the proposed solution