Stability of driver in ipv6 (where Scylla listens) with dns having both ipv4 and ipv6

[tarzanek]

We use IPv6 in our STG/PROD environments, but our DNS records return both A ipv4 and AAAA ipv6 records. Scylla does not listen on ipv4. Our client binary gets stuck and does not log any error; Devs are saying that neither error is returned from the driver.
When a configuration has only healthy nodes then it works well. When there is a node down in the cluster, binary freezes can not connect and no error returned.

Can we somehow check / improve the session builder?

How can this be debugged further so we at least get some error message?
(we will provide version soon, assume 1.1.0 so far)

[Lorak-mmk]

Our client binary gets stuck

Gets stuck on what? Session creation? Or is session created with no error, and it gets stuck when you try to execute queries?

Do you have a code sample that you use?

How can this be debugged further so we at least get some error message?

Hard to tell when you didn't provide any code sample, driver configuration, and didn't even say what operation hangs.

We use IPv6 in our STG/PROD environments, but our DNS records return both A ipv4 and AAAA ipv6 records. Scylla does not listen on ipv4.

I looked at relevant Rust Driver code. When resolving contact point hostname, it will prefer IPv4, only returning IPv6 record iv v4 is not found. Its not obvious to me why this is the case, I'll try to find out.
I also see that it maps one hostname to one ip. Perhaps a more reliable, but slower, way to connect is to use all IPs that hostname resolves to, in some order? cc @wprzytula

[mykaul]

I looked at relevant Rust Driver code. When resolving contact point hostname, it will prefer IPv4, only returning IPv6 record iv v4 is not found. Its not obvious to me why this is the case, I'll try to find out.

Ref. https://en.wikipedia.org/wiki/Happy_Eyeballs ?

[Lorak-mmk]

Ref. https://en.wikipedia.org/wiki/Happy_Eyeballs ?

It could be a nice approach. It is tied to a larger question of: should we try contact points in parallel? Right now we try them serially.

I'll try to find out.

AI is great at such investigations:

When and where it was introduced
The IPv4-preference behavior was originally introduced on October 26, 2020 by Kamil Braun in commit b91b7734f ("session: bring back ToSocketAddrs"), merged via PR #90 (psarna/kbr/resolution).
At that time the code lived in scylla/src/transport/session.rs (not node.rs), in a function simply called resolve. I confirmed no IPv4-selecting logic existed before this commit. The original form was an explicit loop:

// It prefers to return IPv4s first, and only if there are none, IPv6s.
async fn resolve(addr: impl ToSocketAddrs + Display) -> Result<SocketAddr> {
    let failed_err = anyhow!("failed to resolve {}", addr);
    let mut ret = None;
    for a in lookup_host(addr).await? {
        match a {
            SocketAddr::V4(_) => return Ok(a),  // prefer first V4
            _ => { ret = Some(a); }              // otherwise keep last V6
        }
    }
    ret.ok_or(failed_err)
}

How it evolved to today's code
The logic was preserved unchanged through several refactors:

• 52e5294a0 / 94bcf331c — moved/extracted the resolution function into node.rs as resolve_hostname.
• 113f50033 (Oct 25 2024, Mikołaj Uzarski) — "adjust resolve_hostname to functional programming": replaced the explicit loop with addrs.find_or_last(|addr| matches!(addr, SocketAddr::V4(_))). The commit message explicitly notes "using find_or_last is equivalent to previous version" — find_or_last returns the first V4 if any exists, otherwise the last element (a V6), exactly matching the old loop.
• bdac53a2 (Oct 2024) — added the DNS lookup timeout wrapper.
• Later commits moved the file to scylla/src/cluster/node.rs and changed the error type.
  So the behavior in the current code (node.rs:354) is functionally identical to what Kamil Braun wrote in 2020.
  Why it was introduced
  This is the important caveat: the commit does not explicitly justify why IPv4 is preferred. The stated purpose of the commit was different. The commit message says:

We now use lookup_host to perform a DNS resolution, and attempt to find an IPv4 among the returned addresses.
Connection, however, was modified to take a concrete SocketAddr. This will be useful later when we implement shard awareness.
What this tells us:

• The driver's real goal was changing Connection::new to take a concrete SocketAddr (rather than impl ToSocketAddrs), which was groundwork for shard awareness.
• Because resolution can return multiple addresses but a connection now needs exactly one concrete address, the code had to pick a single one. The author chose to deterministically prefer IPv4.
• No design rationale (e.g. dual-stack issues, Scylla server behavior, test environments being IPv4) is recorded in the commit message, PR, or surrounding commits — PR Fix setup step in Build and Cassandra workflows #90 contained only this single commit.
  Bottom line: The IPv4 preference dates back to the driver's earliest DNS-resolution code (Oct 2020, commit b91b7734f). It was introduced incidentally as part of making connections use a single concrete SocketAddr for upcoming shard-awareness work, and the codebase has carried it forward ever since as a behavior-preserving choice. The git history does not document an explicit technical reason for choosing IPv4 over IPv6 — it appears to have been an arbitrary-but-deterministic default rather than a deliberately justified decision.
  ▣ Plan · Claude Opus 4.8 · 1m 19s

[wprzytula]

@Lorak-mmk Will you open a relevant issue in Rust Driver's repo?

[tarzanek]

I am still gathering the data from affected clients, so please stay tuned.

And I was assuming the same as you - that we have a preference to ipv4 which is timing out before it tries ipv6 (if ever)
(and yes, I can try to ask if they can instrument the driver - fwiw with old cpp driver they didn't see the behaviour, so we might be missing some of old patches logic - they provided patches to old driver (also for ipv6))

[tarzanek]

ok so what is also known is - with old cpp driver this didn't happen (same setup, same dns and when node went down no strange "blocking" or "freezing" happened), so not sure if it's feasible to compare connection logic re such setup there and in new wrapper

[Lorak-mmk]

And I was assuming the same as you - that we have a preference to ipv4 which is timing out before it tries ipv6

Now how it works right now: v6 will only be used if there are no v4 DNS records at all. We'll have to address that.

(and yes, I can try to ask if they can instrument the driver

Before we get to more advanced steps, let's first ask them what operation is timing out. I would expect session creation to error after some time in such setup, not hang indefinitely.

[tarzanek]

ref https://scylladb.atlassian.net/browse/CUSTOMER-467

[MaikeR-TV]

At the moment, we're using cpp-rs-driver 1.0.1.

We're simply calling cass_cluster_set_contact_points() for each DNS to be added.
Using a DNS to the server, which returns IPv4 and IPv6 (in this order), we noticed that connection was not working.
This is the error we got from the driver (on Linux):

Could not fetch metadata, error: Control connection pool error: The pool is broken; Last connection failed with: Failed to perform a connection setup request. Request: OPTIONS, reason: Connection broken, reason: Failed to deserialize frame: Failed to read the frame header: Connection reset by peer (os error 104), target: {our IPv4 address with port}   @scylla::cluster::metadata::reader:269
Failed to fetch metadata using current control connection, control_connection_address: {our IPv4 address with port}, error: Control connection pool error: The pool is broken; Last connection failed with: Failed to perform a connection setup request. Request: OPTIONS, reason: Connection broken, reason: Failed to deserialize frame: Failed to read the frame header: Connection reset by peer (os error 104)   @scylla::cluster::metadata::reader:293

The "connection reset by peer" turned out to be a timeout due to Scylla service not listening on the IPv4 but only on IPv6.
This was no issue with the old driver, which fell back to the secondary IPv6 address.

We fixed our DNS configuration now, but it would be great if the driver would support dual stack and handle this more gracefully, as the old driver did.