Plan for new app data scheme

[vkgnosis]

Related to cowprotocol/services#1465 . This is a rough plan for what code changes need to happen in the frontend for the new app data scheme. This doesn't need to be implemented immediately. I just want to plan it. I can try doing some of these changes too.

Create order

Instead of uploading full app data to IPFS, pass it to the backend in the create order request, which stores it in the backend database. In the json POST body appData used to be the hash but now it is a string of full app data (JSON.stringify(appData)).

Based on this full app data string, the contract app data field has to be set to the right hash. The hash can be calculated like this:

import { keccak_256 } from '@noble/hashes/sha3'

fullAppDataToappDataHash(appData: AnyAppDataDocVersion): Uint8Array {
  const encoded = new TextEncoder().encode(JSON.stringify(appData))
  return keccak_256(encoded)
}

In the post body JSON we can set appDataHash to the hash we calculated (as a hex string) to help debugging. If it is set, the backend can give a better error message when the hash doesn't match what the backend calculates as the hash. This is useful during development.

Show order in explorer

The frontend doesn't need to upload full app data to IPFS anymore because the backend database stores it and it is returned when fetching orders in the API. However, for orders that weren't created with the new app data scheme, the backend might not have the full app data. This can be handled in the frontend by checking if "GET order" returns the full app data or null. In the case of null, the old IPFS retrieval is tried.

Edit after 2023-06-07 meeting:
The backend is going to back fill all app data from IPFS and return it with orders. So eventually there is no need for the frontend to fetch from IPFS at all.

Uploading to IFPS

It is possible to upload the new app data scheme to IPFS too. It isn't required but we can do it. However, when I checked Pinata's API docs, I couldn't find a direct way to store a raw block instead of a file, which the new app data schema specifies. Pinata can still be told to pin an arbitrary IPFs CID but it doesn't have an easy "upload block" api. Could feature request it from them or could implement a workaround.

A potential workaround is having a backend service do IPFS uploading. Pinata can be told to pin the CID immediately and eventually it will be able to retrieve it from the backend service.

Since uploading to IPFS is no longer required and can be done any time later as long as we don't have data loss in the backend database, I'd like to postpone it.

[shoom3301]

This is done