Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Assigning reviewers:

R: @Abacn for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@Abacn gentle ping could you please take a look at this.
Thank you.

Hi folks

First, @Suvrat1629 , huge thanks for the contribution.

I needed to be more specific and detailed in #33692, my apologies.

As of today, this is the existing behavior:

• We only take care of DEFAULT type (we don't take care of default types that are of the form of Materialization and Alias)
• Even for Default, we have a small set of supported types
• We don't use the defaults as of this moment - the RowBinary format doesn't use them, and they are just being ignored.
• This functionality is not supposed to be handled on the client side (as it is impossible to materialize all ClickHouse expressions in Java), and ClickHouse created the RowBinaryWithDefaults format to move the defaults handling into the database itself.

Is there any other place we use this?

Yes.
When we implemented the BigQuery to ClickHouse Dataflow template, we received from BigQuery TableRow - a schemaless object. In order to convert it to schema schema-aware type, we fetched the schema from ClickHouse and used this function to convert each string value to the proper Java representation.

The original idea behind #33692 was not only to add these types, but also to solve/decouple the tension between the different needs (the first need is to evaluate defaults, the second is to evaluate primitive values).

As I commented here #33692 (comment), we plan to upgrade the ClickHouse client, support the use of RowBinaryWithDefaults, and refactor the parseDefaultExpression to work with primitive types only.

Suggested implementation in this PR

The main issue with the existing implementation (that is being extended with this PR) is that in case users use none primitive default values (which is usually the case), the entire pipe will collapse.

For instance, given the following target table:

CREATE OR REPLACE TABLE test
(
    id UInt64,
    updated_at DateTime DEFAULT now(),
    updated_at_date Date DEFAULT toDate(updated_at)
)
ENGINE = MergeTree
ORDER BY id;

The DESRIBE command output will be:

and the entire pipe will get broken (the code will try to create a DateTime object of the string now()).

So to conclude, the right solution for this problem would be to implement support for the RowBinaryWithDefaults format and refactor+rename the parseDefaultExpression to support only primitive objects.

@Suvrat1629 I really appreciate the time and effort you put into this, and my apologies for not being more detailed abou this in the original issue.

CC @Abacn

@BentsiLeviav Thanks for the response
That is a lot of information to register but I have gone through the code and come up with an implementation. I want to verify the approach first as I do not have a lot of experience with Clickhouse and this would save me time also.
My approach:

In ClickHouseIO.java:

• Change the format specification from RowBinary to RowBinaryWithDefaults
• Update the method call to use the new primitive-only parsing method

In TableSchema.java:

• Add a new method called parsePrimitiveValue that only handles simple literals
• This method will detect complex expressions anything with parentheses, function names like "now", "toDate", etc. and return null for them
• Mark the existing parseDefaultExpression method as deprecated
• The deprecated method can delegate to the new one for backward compatibility

In ClickHouseWriter.java:

• Completely rewrite the writeRow method to support the new format
• Before writing each column value, write a flag byte
• If we have no value and no client-side default, write the "use server default" flag
• If we have a value either from input or client-side default, write the "use provided value" flag followed by the actual data

Does this fit right? Provide me comments to improve this @BentsiLeviav @Abacn

Hi @Suvrat1629,

After discussing internally, I need to share some important context:
The ClickHouse Java client V2 already includes:

• Built-in schema implementation (replacing our custom one)
• Native RowBinaryWithDefaults handling

This means implementing the current approach would create code we'd immediately need to remove.
Additionally, using the new Java client will require breaking the API. IIRC (@Abacn, please advise), this will require updating the Dataflow template (which shares a global Beam version), adding significant testing overhead and coordination complexity.

Given that we plan to start the V2 migration refactor in the near future, it would be more efficient for us to address this issue as part of that comprehensive update rather than implementing a temporary solution now.

I really appreciate your enthusiasm and the time you've invested in understanding this issue. Would you be interested in picking up a different issue that won't be affected by the upcoming refactor?

In addition, did you implement this due to limitations you have with ClickHouse and Beam related to default values? If so, can you share more about your use case and about the issues you encounter in your pipeline?

@BentsiLeviav Ok sure I just picked up the issue because I thought I had found a fix for this but if you plan on implementing a fix for it yourself I am ok closing this pr :)