Create graphs for the DietPi survey page

[JappeHallunken]

I was thinking about creating some nice graphs for the survey page, but it's not part of this repo.
I assume there is some script running on the web server creating the page?

Is there a chance to make this public or handle it in some way, so users can also contribute to this part of the website?

[MichaIng]

Yes here the script: https://github.com/MichaIng/DietPi/blob/dev/.meta/dietpi-survey_report

But makes sense to shift this over here. It was added as minimal data visualisation before our website got even added to GitHub. I also have a JavaScript contributed by someone with a search/filter function for the tables. Pretty nice. Adding such all all to a shell script doesn't seem reasonable. I guess we should keep a template incl. separate CSS and JavaScript here on the GitHub repo, and then somehow find a neat way to add the tables/data on our server. I do not want to use a database, but keep it simple for now, so it would need to be something like generating an HTML from a header template + CSS + JavaScript, keeping the latter here on the GitHub repo that we can pull on our server 🤔.

[JappeHallunken]

Hmm, instead of having the script generate a static HTML file, maybe it could just create a JSON file, also available at /var/www/survey/. That way, users could even fiddle with the JSON data if they want.

Then we could add a survey page to the website repo with some JS that reads the JSON and builds the graphs, charts, etc.

[MichaIng]

Seems a good idea, as long as all works with vanilla JS. It would mean another request in the chain: HTML > JS > JSON, but should not be noticable. The JSON could be prefetched via header to make things asynchronous in modern browsers.

If it leads to jumping content or slow first draw time, we could still inline the JavaScript and even the JSON at the server.

[JappeHallunken]

Some time has passed, and I had another look at this idea.
What data should be visualized?
IMO the most relevant data would be:

• top used devices (e.g. only device types with more than 1000 users in the survey)
• survey opt-in vs. opt-out ratio
• DietPi versions in use (capped at a threshold)
• CPu architecture distribution
• Linux distro versions

I also think we can keep it more “diet” and avoid going the JS route. There is a CSS library, Charts.css, which would be sufficient for our purpose. The obvious chart type for the data mentioned above would be a pie chart, and it would be easy to integrate into the current shell script. The minified file of the current release is 59 KB in size.

As a JavaScript alternative, there is Chart.js. It renders onto an HTML canvas element and is significantly more powerful — we could also visualize benchmark data if needed. The current release is 66 KB in size, and while it is tree-shakeable, that would require introducing a bundler. Even without that, it is not as straightforward to integrate into a shell-script-based setup.

Performance-wise, given the small data sets involved, there would likely be no noticeable difference.

What do you think?

[MichaIng]

DietPi versions in use (capped at a threshold)

Those within the last 6 months makes sense, since any data older than 6 months is dropped from our stats. Since the age of a release is not part of the data, it could take just the latest 7, which usually means current dev cycle + last 6 releases which are usually not older than 6 months.

The obvious chart type for the data mentioned above would be a pie chart

A bar chart could do as well in some cases. Did I not mention our Grafana page in this context, which has the data as time series?

We always thought about making this public, after polishing the layout a bit. The system count evolution shows the effect of data being sorted out after untouched since 6 months, which causes a negative peak whenever a DietPi release becomes 6 months old, as of the update peak the first day(s) after the release.

Also the typical summer hole in long term stats is interesting 🌞:

And the daily fluctuation of new systems on the one hand, such which did not send a survey (or empty ping) within 6 months considered inactive, and those which were considered inactive before, but then did send a new survey (i.e. did a DietPi update or dietpi-software installation):

Gives a better idea about how to interpret "active" DietPi systems, and someone with knowledge and time for statistics calculation could derive whether we in average consider systems too long as active, or too early as inactive, with the 6 months threshold 😄.