Handling expensive data sources

[Waitak]

Thank you for making this wonderful package available. As I mentioned in a Slashdot question I have a use case that I wonder if it handles. I'd like to use pandoc-plot to generate a number of charts based on a Pandas data frame that is computationally costly to construct. I thought initially that I could use the preamble parameter to invoke a script that constructs the data frame, but that isn't working. Do you have any suggestions?

[LaurentRDC]

Hi Scott,

Thank you for the kind words.

Out-of-the-box there's no handling of your use-case in the pandoc-plot filter. Each code block that gets turned into a plot is intended to be independent from all others. This has many benefits, most importantly performance -- I wrote pandoc-plot for book-sized workloads, with close to 100 figures.

The reason using preamble isn't working is because the preamble script gets copy-pasted into every code block before pandoc-plot renders a figure. Therefore, the creation of your dataframe will still be duplicated.

I would recommend you proceed with a script to wrap your usage of pandoc. For example (assuming you use bash):

# Run a script that goes through your expensive computation,
# storing the results as a CSV i
python create-data.py

# Render the document, where plots can reference the file created by 
# your python script instead of re-creating the pandas dataframe for every plot
pandoc -f pandoc-plot ...

# Clean up temporary data file if you know where it is

You can communicate between the bash script above and your document plots using environment variables.