CASSGO-118 Encode nil map value for UDTs as NULL by worryg0d · Pull Request #1942 · apache/cassandra-gocql-driver

worryg0d · 2026-04-07T14:39:59Z

Previously, the driver was encoding nil map values for UDT-typed columns as an initialized map with zero values for each UDT field. This patch changes its behaviour to encode nil maps as NULL values in Cassandra.

Patch by Bohdan Siryk; reviewed by TBD for CASSGO-118

worryg0d · 2026-04-07T14:41:05Z

I'm unsure whether this should be fixed in the current major because people could rely on that behavior already, but anyway submitting a PR

worryg0d · 2026-04-15T09:25:41Z

+	EncodeNilMapAsInitilizedUDT bool
+	// Supresses the warning that is emitted when EncodeNilMapAsInitilizedUDT is enabled.
+	SuppressEncodeNilMapAsInitilizedUDTWarning bool


I don't really like this approach of warning people, but this is the best I ended up with...

Why do we need to warn anyone? Is there a problem that the values are non-NULL? If there isn't any issue then I assume we don't need to warn.

To me, this is at least not something I would expect when dealing with nil-values. For example, std json lib encodes nil values as null and vice versa. But, I found out one more thing: driver behavior is inconsistent between struct-based and map-based UDT encoding.

For example, the following code will have two different behaviors that depend on what is wrapped the data var:

func insertPerson(session *gocql.Session, id int, name string, data any) error { const insertQuery = ` INSERT INTO gocqltest.persons (id, name, address) VALUES (?, ?, ?); ` err := session.Query(insertQuery, id, name, addressMapData).Exec() if err != nil { return fmt.Errorf("failed to insert person: %w", err) } return nil }

Encode as non-null when data holds map[string]any(nil)

Encode as null when data holds nil ptr struct. I guess this actually related to all ptr types: https://github.com/apache/cassandra-gocql-driver/blob/trunk/marshal.go#L75-L77

But when the driver reads that data from the db how will that be decoded for both cases? And has that inconsistency been in the driver for a long time too or is it more recent?

Decoding depends on what is written in the DB:

Null value - gocql decodes as nil map

something like this {street: null, zip: null} - gocql decodes as udt with null values

The behavior is consistent between v1.7 and v2

worryg0d · 2026-04-15T09:52:50Z

 	copy := &RegisteredTypes{}
 	copy.init()
+	// Adding default types to the copy so collection type codecs will have a pointer to the copy instead of the original.
+	copy.addDefaultTypes()


As marshaling/unmarshaling is implemented on TypeInfo types, we should somehow propagate its behavior from the cluster config. Collection types such as udtCqlType hold a pointer to RegisteredTypes object to know how to marshal/unmarshal different types. The problem was that in NewSession, where we create a copy of the provided RegisteredTypes by calling RegisteredTypes.Copy(), which just initializes a new object and populates maps with values from the original map, which includes a pointer to the original RegisteredTypes.

Fixed it by calling addDefaultTypes() on a copy and adding missing types/names/aliases to the copy.

worryg0d · 2026-04-16T07:52:57Z

+		scanned := insertNilMapAndScan(t, session, 1)
+		expected := map[string]interface{}{"field_a": "", "field_b": 0}


Actually, it is supposed to return a scan of a map with nil fields instead of zero-valued fields, isn't it?

So we should expect having nil pointers of *string and *int types:

var fieldA *string = nil var fieldB *int = nil expected := map[string]interface{}{"field_a": fieldA, "field_b": fieldB}

It looks like it makes a pointer and then unmarshals into that pointer and then dereferences it. So I think map[string]interface{}{"field_a": "", "field_b": 0} is right.

jameshartig · 2026-04-16T16:38:49Z

+	// ---+-------------------------------
+	//
+	// Default: true
+	EncodeNilMapAsInitilizedUDT bool


Shouldn't the default be the zero value in Go? In other words, this should probably be EncodeNilMapAsNULL instead.

I like the name, but what do you mean by the first part? The change doesn't affect the driver's udt decoding behavior

jameshartig · 2026-04-16T16:53:16Z

 		return nil, fmt.Errorf("the default SerialConsistency level is not allowed to be anything else but SERIAL or LOCAL_SERIAL. Recived value: %v", cfg.SerialConsistency)
 	}

+	logger := cfg.newLogger()


This will warn for people even if they're not using UDTs which seems overly annoying and unnecessary.

hmm yeah you're right. I still want to warn people about that behavior for the reasons I explained in another comment, but I'd prefer it not be too noisy...

@jameshartig I would not warn anybody at all but that is just my preference, not going to "die on this hill". If they were affected, then they would have to be in a situation when they are 1) aware of this bug 2) their "nullity check" on UDT means that they are iterating over all fields in UDT and check if it is null themselves. I do not think there is anybody out there who is doing this. We would be having this bug reported already. Us switching to just fixing the bug and having this in notes is just OK imho.

encode nil maps as NULL values in Cassandra. - this IS the correct behavior.

But take this as just a feedback from somebody external, I do not know what you guys really prefer. Ultimately up to you.

Us switching to just fixing the bug and having this in notes is just OK imho.

I think the issue is that switching to the correct behavior might lead to user apps breaking because their data is already written as empty instead of NULL on their databases so after the user upgrades and starts writing the new data as NULL then they will end up in a situation where the old data is not NULL and the new data is NULL but in reality both data values were encoded from the same input.

I do find it odd that we've not seen this bug reported before or maybe it was and I didn't see it (I haven't been involved with the project for that long). Maybe it's because what ultimately matters is that the data that comes back from a read is the same as the data that was written even if the underlying value in the database is not the correct one...

What is returned by the driver is basically what has been written to the DB:

UDT with null fields in DB - gocql decodes as a map with key pairs field_1 = nil, field_n => nil

If UDT is null in DB - gocql decodes it as a nil map

Previously, the driver was encoding nil map values for UDT-typed columns as an initialized map with zero values for each UDT field. This patch changes its behaviour to encode nil maps as NULL values in Cassandra. Patch by Bohdan Siryk; reviewed by TBD for CASSGO-118

…guration

…d bool)

… udts

worryg0d · 2026-06-29T14:56:58Z

Alright, I have updated the PR with what we discussed with @jameshartig

I added a public method RegisteredTypes.WithNullableUDTs(enabled bool) that changes encoding behavior. This approach enables users to override the encoding behavior globally, so any new session would inherit the desired behavior:

var GlobalTypes =  GlobalTypes.WithNullableUDTs(true)

Also, I have added a limitedLogger wrapper around StructuredLogger that limits the number of logs to each n-th, including the first one. Currently, I chose each 100th log to be logged. However, probably just logging once is sufficient?

worryg0d · 2026-06-29T14:57:46Z

 	return UDTTypeInfo{
-		Keyspace: keyspace,
-		Name:     name,
-		Elements: elements,
+		Keyspace:           keyspace,
+		Name:               name,
+		Elements:           elements,
+		encodeNilMapAsNull: u.types.encodeNilMapAsNull,
+		warnOnNilMap:       u.types.warnOnNilMap,
+		logger:             u.types.logger,
 	}, nil


I'm not comfortable passing configurations and the logger deep into the corresponding marshaler in such a way because it complicates things and makes it more error-prone. For example, in unit tests we're using GlobalTypes which defaults warnOnNilMap to true, so any created UDTTypeInfo object would inherit its value as well, but our unit tests don't cover it.

Right now I just intentionally called WithNullableUDTs so the warning is disabled, but this is anyway far from ideal...

worryg0d force-pushed the cassgo-118 branch from 4c1827e to 126aecf Compare April 7, 2026 14:51

worryg0d commented Apr 15, 2026

View reviewed changes

worryg0d commented Apr 16, 2026

View reviewed changes

jameshartig reviewed Apr 16, 2026

View reviewed changes

worryg0d and others added 12 commits June 29, 2026 17:11

added correct behavior option, warning log, and tests

748c610

test fix

381f913

some tricks around RegistedTypes.Copy to allow passing encoding confi…

bc6ab02

…guration

simplify code

c9d5291

added marshal unit test

cc10f71

use encoding config copy instead of ptr

4d669ac

clarify warn log

1b541ae

renamed option and remove warning

e26b6e6

Removed EncodingConfig, added RegisteredTypes.WithNullableMaps(enable…

42470ed

…d bool)

introduced limitedLogger and warning on nil map passed to marshal for…

2e4e423

… udts

shallow copy on WithNullableUDTs

3603ab9

worryg0d force-pushed the cassgo-118 branch from 6462ee0 to b812968 Compare June 29, 2026 14:36

fix tests

f98a9c1

worryg0d force-pushed the cassgo-118 branch from b812968 to f98a9c1 Compare June 29, 2026 14:41

worryg0d commented Jun 29, 2026

View reviewed changes

		scanned := insertNilMapAndScan(t, session, 1)
		expected := map[string]interface{}{"field_a": "", "field_b": 0}

Uh oh!

Conversation

worryg0d commented Apr 7, 2026

Uh oh!

worryg0d commented Apr 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smiklosovic Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

worryg0d commented Jun 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smiklosovic Apr 16, 2026 •

edited

Loading