Idea for smaller JSON

[hyanwong]

Instead of storing each column name in the JSON file, why don't we store the column names as accessors to the index. I.e. instead of this:

{"data":
  {"nodes": [
    {"size":150,"symbol":"d3.symbolSquare","fill":"#1eebb1","stroke":"#053e4e","stroke_width":4,"id":0,"flags":1,"time":0,"child_of":[21],"parent_of":[],"x_pos_reference":-1,"label":"0","fx":100,"fy":450,"y":450,"index":0,"x":100,"vy":0,"vx":0},
    {"size":150,"symbol":"d3.symbolSquare","fill":"#1eebb1","stroke":"#053e4e","stroke_width":4,"id":1,"flags":1,"time":0,"child_of":[10],"parent_of":[],"x_pos_reference":-1,"label":"1","fx":161,"fy":450,"y":450,"index":1,"x":161,"vy":0,"vx":0},
    ...
    ], ...
}

we could use

{"colnames":{
    "nodes":{"size":0,"symbol":1,"fill":2,"stroke":3,"stroke_width":4,"id":5,"flags":6,"time":7,"child_of":8,"parent_of":9,"x_pos_reference":10,"label":11,"fx":12,"fy":13,"y":15,"index":16,"x":17,"vy":18,"vx":19}
},
"data":
    "nodes": [
    [150,"d3.symbolSquare","#1eebb1","#053e4e",4,0,1,0,[21],[],-1,"0",100,450,450,0,100,0,0},
    [150,"d3.symbolSquare","#1eebb1","#053e4e",4,1,1,0,[10],[],-1,"1",161,450,450,1,161,0,0},
    ...
    ], ...
}

Then the access would be

var nodecols=colnames.nodes
sz = data.nodes[0][nodecols.size]  // was sz = data.nodes[0].size

[kitchensjn]

Though smaller, would this not mess with standard JSON readers? pandas makes it pretty easy to covert the current JSON into a DataFrame and I think we would need some custom conversion solutions for new format.

[hyanwong]

Good point. I didn't look how you had done the conversion.

However, if you are simply using pandas to convert, I think this is the same as orient="split": see https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html. So perhaps we could just switch to that instead?

[hyanwong]

Oh, or if you are reading directly from the python objects, I guess DataFrame.from_dict(data, orient='tight') is the one we could use: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.from_dict.html

[hyanwong]

Oh, I think none of this may work well in javascript, though. The JS code accesses the data using the standard JSON arrays etc, and outputs them again using JSON.stringify()

[hyanwong]

Aha, I've just been digging into this, and I think it should be quite easy, at least for the JSON versions that are directly converted from data frames (I think that's "links", "breakpoints", "mutations", but not "nodes" (because that doesn't have consistent column names: it sometimes has fx as the column, and sometimes x.

For the ones that are dataframes, we output with orient="split", which creates a JSON like this:

{
  "columns": ["id","source","source_time","target","target_time","bounds","alt_parent","alt_child","region_fraction","stroke"],
  "index": [0,1,2,3,4,5,6,7,8,9],
  "data": [
    [0, 1436683, 4.0, 1436731, 3.0, "0.0-29904.0", -1, -1, 1.0, "#053e4e"],
    [1, 1436472, 5.0, 2688969, 4.0, "0.0-29904.0", -1, -1, 1.0, "#053e4e"],
    ...
  ]
}

Then when importing in the javascript, we make a links_col that maps

{'id': 0, "source": 1, ...}

Which we can do using

const links_cols = graph.links.columns.reduce((acc, str, index) => {
            acc[str] = index;
            return acc;
        }, {});

Then we pass graph.links.data rather than graph.links and wherever we see something like .attr("bounds", d => d.bounds);, we instead do .attr("bounds", d => d[links_cols.bounds]);

[hyanwong]

So I mocked something like this up in #194. The easiest was to take the "split" json format and convert that at the start of the javascript into a format that the d3 code understood. It seems to work. The big saving will be in doing the same for the mutations and nodes, I guess, but I think they might be more difficult because node data positions are saved back into the javascript (I think?), and mutations are not always a standard tabular format, are they?

It might help, however, to make the mutation table standard between the different mutation views, somehow?