Download graph data from report as JSON

[stuartjamesmcnicholas]

Allow the addition of a button to bottom of graph which selects the same data as shown when "popping out" pimple and writes it to a JSON file.

New keyword downloadable to addFLlotGraph (and addFlotGraphGroup). e.g.:

progressGraph = parent.addFlotGraph(title="Running refmac",select=tag+"/Cycle",style="height:250px; width:400px;float:left;border:0px;",outputXml=self.outputXml,internalId=internalId,downloadable=True)

or as part of FlotGraphGroup:

graph = graphgroup.addFlotGraph( xmlnode=thisgraph, title=thisgraph.attrib["title"],downloadable=True )

In the case of a FlotGraphGroup the button will be added either if the Group is declared downloadable or any of the component graphs. This is necessary as the whole lot of data are one list when the widget is rendered.

The format is JSON as doing this with CSV would require multiple data sets per file which is perhaps somewhat ugly (and possibly non-standard).

[MartinMalyMM]

Hi Stuart! Thank you for dealing with this! I set downloadable=True in all FlotGraphs in servalcat.

• Shouldn't this be the default option for addFlotGraph() ?
• JSON becomes invalid when there are NaN values
• Could we add some indentation to JSON or use pprint? It don't like that it's one very long line.
• Values are misleading when using plot.append('xscale', 'oneoversqrt') or similar.
• I think that JSON is not a user friendly format (although it's useful for us developers). CSV would be better. Users will want to insert the values in Excel, etc. What about the old format from ccp4i for loggraph? It can handle multiple data sets and can be read by humans as well as loggraph.

Example: From loggraph servalcat.log

$TABLE: Data stats in cycle 5 :
$GRAPHS
: Mean I/F vs. Resolution :A:1,4,5:
: R :A:1,16,17:
: CC :A:1,13,14,15:
: Data completeness :A:1,12:
$$
 1/resol^2   d_max   d_min       Mn(Io)       Mn(Ic)  n_obs  n_all  n_work  n_free  n_R1work  n_R1free     Cmpl    CC*  CCIwork  CCIfree  R1work  R1free
$$
$$
    0.0098 56.3195 10.1266 1645509.8750 1482878.7154    578    579     549      29       545        29  99.8273 1.0000   0.8316   0.9633  0.2429  0.1905
    0.0192 10.1266  7.2192  840558.6875  855590.1195    938    938     891      47       882        46 100.0000 1.0000   0.9353   0.9250  0.1749  0.2144
    0.0286  7.2192  5.9106  457615.4062  513379.7522   1163   1163    1105      58      1091        57 100.0000 0.9998   0.9353   0.8101  0.1968  0.3180
    0.0381  5.9106  5.1258  462771.7500  500329.1142   1352   1352    1284      68      1259        68 100.0000 0.9999   0.9419   0.8847  0.1721  0.1916
    0.0475  5.1258  4.5885  591487.8125  582365.7848   1515   1515    1439      76      1426        76 100.0000 0.9999   0.9498   0.9407  0.1455  0.1672
    0.0569  4.5885  4.1910  483371.8125  439307.7071   1656   1656    1573      83      1553        83 100.0000 0.9999   0.9312   0.8565  0.1637  0.1993
    0.0664  4.1910  3.8817  334468.4688  294217.1378   1778   1778    1690      88      1656        88 100.0000 0.9999   0.9375   0.8466  0.1755  0.2170
    0.0758  3.8817  3.6320  261400.1719  217080.5591   1920   1920    1824      96      1768        94 100.0000 0.9999   0.9649   0.8646  0.1899  0.2510
    0.0852  3.6320  3.4251  179154.8594  142335.4004   2033   2033    1931     102      1850        99 100.0000 0.9998   0.9113   0.8056  0.2046  0.2341
    0.0947  3.4251  3.2499  117387.6406   99312.4579   2133   2133    2026     107      1881        97 100.0000 0.9997   0.9264   0.8674  0.1956  0.2411
    0.1041  3.2499  3.0992   75043.5703   66320.6218   2242   2242    2130     112      1914        97 100.0000 0.9993   0.9078   0.7967  0.1944  0.2457
    0.1136  3.0992  2.9676   49283.5820   46589.6099   2325   2325    2209     116      1863        98 100.0000 0.9985   0.9060   0.7826  0.1824  0.2179
    0.1230  2.9676  2.8515   32340.8008   33018.2074   2445   2445    2323     122      1792        89 100.0000 0.9971   0.8998   0.7929  0.1924  0.2889
    0.1324  2.8515  2.7480   21191.4473   24045.9256   2507   2507    2381     126      1637        88 100.0000 0.9933   0.9089   0.8420  0.1966  0.2342
    0.1419  2.7480  2.6550   16244.9473   18319.1010   2605   2605    2475     130      1520        84 100.0000 0.9915   0.9001   0.8061  0.1949  0.2253
    0.1513  2.6550  2.5709   11430.1055   13632.2889   2662   2662    2529     133      1297        67 100.0000 0.9800   0.8966   0.8646  0.1799  0.2325
    0.1607  2.5709  2.4943    8258.3604    9784.7534   2794   2794    2654     140      1166        65 100.0000 0.9614   0.8746   0.8316  0.1815  0.2284
    0.1702  2.4943  2.4242    6686.4956    7883.8419   2840   2840    2698     142      1016        52 100.0000 0.9617   0.8927   0.8424  0.1894  0.2328
    0.1796  2.4242  2.3596    4776.0601    5882.5348   2933   2933    2787     146       807        44 100.0000 0.9247   0.8507   0.8002  0.1901  0.2545
    0.1890  2.3596  2.3000    3309.1782    4353.5977   3008   3009    2857     151       629        29  99.9668 0.8450   0.7810   0.7324  0.2046  0.2009
$$

I like it much more than JSON:

[stuartjamesmcnicholas]

• NaN is an issue, I will deal with that.
• CSV is tricky (basically not doable) for the reason I stated in the description.
• The data is actually extracted from $TABLE in the report, downloading this (as an option) would also be possible. Or perhaps even viewing in report; that would be another PR I think.
• If pretty print JSON is possible, I will do it.

JSON makes the data (not withstanding NaN) usable in other programs. Apologies that I hadn't considered human readability to be the main issue. I will address.

[stuartjamesmcnicholas]

Multiple CSV files might be an option - one per "dataset". So, I will add this along with $TABLE, but this is only useful for inspection.

[MartinMalyMM]

Thank you for the improvements.

I quite like the output as a "plain table". Could the file format be a CSV, separated by commas? Or very nicely fixed-width text document? My wish would be both, a CSV table (machine readable) and a fixed width table (human readable). Now it's half way there, it's a strange hybrid.

Resolution(Å) CCIwork CCIfree CC* 
0.014832608871504087 0.8527368599488833 0.9595853504986023 0.9999575863833935 
0.029349948414071306 0.9347489563722172 0.8494564159040771 0.9998670473931726 
0.0438672879566385 0.9472242298294594 0.9259426607027729 0.9999384802235719 
0.058384627499205734 0.9403379211424301 0.8643187026108289 0.9999357144700334 
0.07290196704177293 0.9580398430137091 0.8893699003129526 0.9999098423692158 
0.08741930658434015 0.9137300045490496 0.8424516233671794 0.999803927831949 
0.10193664612690737 0.9137033255923551 0.7800324927476737 0.9995388173115614 
0.1164539856694746 0.8948697476280674 0.7960357521248406 0.9984549172505703 
0.13097132521204174 0.896305219709682 0.7728328133634415 0.9958817346831715 
0.14548866475460898 0.8949418213996776 0.8166767785335513 0.9902327405472963 
0.16000600429717624 0.878990345128909 0.8211139221654025 0.9685083217727338 
0.17452334383974338 0.8901582144318089 0.8276629716481041 0.9557850532075609 
0.18904068338231061 0.8031458793684162 0.7786848022086225 0.8759357539969911 

I don't like the multiple individual CSV files for each data series. I like more if they are together in a table. If we have nice table(s), then we don't need a CSV file for each data series.

Values for x are still misleading plot.append('xscale', 'oneoversqrt') was used. Could we please at least add a comment oneoversqrt in the header of the CSV and fixed width file?