Should Blobentry.metadata dict have CRUD?

[dehann]

We have enough need for CRUD on agent.metadata. Its a lot of work to add the same CRUD to blobentry.metadata, so likely can delay into DFG v1.X as feature expansion.

[dehann]

currently blobentry is immutable.

[dehann]

is so, there can be a breaking change on the type of the metadata field:

struct Blobentry
  ...
  metadata::Dict{Symbol, Jsonable/String/Smalldatatypes}
end

[Affie]

I'm leaning towards no curd on Blobentry.metadata. We can consider renaming the field for clarity with naming options in the line of attributes or blobmetadata to help avoid breaking changes as in #1172 (comment) (if it does not match other node metadata fields)
A few options for the type:
Immutable

• ::JSONText (from JSON.jl) Basically String, but insert raw JSON in JSON output (No checks)
• ::String (currently with JSON, maybe base64-encoded)
• ::NamedTuple type not know as with Metadatatypes, but dynamic dispatch might not matter as we are dealing with blobs that are likely way slower than the dynamic dispatch.

Mutable

• ::Dict{Symbol, String} hard type but loses flexability
• ::Dict{Symbol, Metadatatypes} as with other small data fields, but no crud and serialized as before (or maybe not base64)
• ::JSON.Object{String, Any} Any has no type enforcement and not type-stable.

*with Dict approaches we have to be careful as modification will not be persisted (note NamedTuple is immutable). You should always call mergeBlobentry and know that .metadata will be overwritten.

[dehann]

just to confirm from the metadata design call,

• metadata field is purposefully kept as informal scratch space where user can dump any weird and wonderful data fragments they need to make their lives easier, with basically two conditions. (See Bloblet for more formal counterpart to metadata),
  • don't supplant the use of blobs and blobentries for large chunks of data if you intend production use (obviously the development cycle of a few Mb landing in metadata is par for the course / it happens.
  • Whatever you store in any metadata field has one basic JSONable requirement which is enforced where the metadata field content has to be key=>String pairs (implemented via either vec or dict).
• the type for any node's struct field metadata::Dict{Symbol,String}
  • These key values can maybe go as far as atomic add verbs in DB case,
  • Non-power user has to overwrite the whole node to delete keys
  • Manipulating key values in-memory is a power user feature, which requires power user to know that updates are unsafe (non-atomic to Remote), unless the entire node is overwritten.
• Introduction of the high level convenience phantom noun Bloblet which will use blobless Blobentry to achieve non-poweruser CRUD/ACID for "bloblet" (aka small data) -- see new issue below