Having "Unable to receive result for write task: channel closed" error on v1.6.23

[Mhs-220]

Describe the bug
After upgrading cube-js from 1.5.15 to 1.6.23 we're seeing a random failure in our self-hosted CubeJS instance. Suddenly nothing works and we see the following log on the main Cube instances:

QueryError: Internal: Unable to receive result for write task: channel closed
    at WebSocket.<anonymous> (/cube/node_modules/@cubejs-backend/cubestore-driver/src/WebSocketConnection.ts:163:32)
    at WebSocket.emit (node:events:519:28)
    at Receiver.receiverOnMessage (/cube/node_modules/ws/lib/websocket.js:1070:20)
    at Receiver.emit (node:events:519:28)
    at Receiver.dataMessage (/cube/node_modules/ws/lib/receiver.js:502:14)
    at Receiver.getData (/cube/node_modules/ws/lib/receiver.js:435:17)
    at Receiver.startLoop (/cube/node_modules/ws/lib/receiver.js:143:22)
    at Receiver._write (/cube/node_modules/ws/lib/receiver.js:78:10)
    at writeOrBuffer (node:internal/streams/writable:572:12)
    at _write (node:internal/streams/writable:501:10)

And on the API side, we also see this error:

{
    "errors": [
        {
            "message": "QueryError: Internal: Unable to receive result for write task: channel closed",
            "locations": [
                {
                    "line": 231,
                    "column": 5
                }
            ],
            "path": [
                "cube"
            ]
        }
    ],
    "data": null,
    "extensions": {}
}

The issue persist until we restart the workers and the router.

To Reproduce
It seems to be random or at least we don't know what is causing it.

Expected behavior
Have a self-healing mechanism for cases like this.

Version:
1.6.23

Additional context
We're self hosting on ECS + Fargate and cube instances are connected via AWS Service Discover.

[Mhs-220]

Might be related to this one:
#10541

[Mhs-220]

OK found the problem but I'm not quite sure if this is the solution or I'm just moving the problem.

• Our cube router instance had 50GB of disk space and we are randomly facing this issue because suddenly we have this problem on some days (probably based on the number of requests)
• This is an issue probably caused by migrating to DataFusion 46 (?) because so far this was working before the upgrading to the new version 🤔

Here are all different kind of error messages that we're receiving at that time:

Error during read write loop execution: Internal: IO error: No space left on device: While appending to file: /cube/.cubestore/data/metastore/001195.log: No space left on device

Error in processing loop: Internal: Unable to receive result for write task: channel closed

Error during Cachestore ttl persist: CubeError { message: "Unable to receive result for write task: channel closed", backtrace: "", cause: Internal }

Error while removing olds queue results: Internal: Unable to receive result for write task: channel closed

I've increased the disk size to 150GB to see if that will "solve" the issue in the long term or not. But it feels like this should be handled in cube (or maybe it's already handled and we're just doing something wrong)

[Mhs-220]

I've just noticed that this is a duplicate of: #9114 Closed