statistics module + matomo

[mathieuLU]

configure the matomo app for the client

• a statistics module on the API side

[mathieuLU]

Matomo is installed on the client side.
I have a simple handmade way of storing all the requests on the API level. https://github.com/biodiversitydata-se/biologging-sensor-API/commit/14fd0b16d43af5506ff3811324339cd7a4be361b

Are there specific statistics that we would need to have ? Because depending of the needs i may have to change a bit the structure of the basic stats module

right now Matomo will register all the visits.
on the API side i'm counting every requests sent. But do we want to track the origin ?
for instance do we want to calculate which requests are coming from the client, and which are coming from a user using the API through scripts ?

If at some point we want to implement a role based behaviour of the API, we will need anyway the source of the requests

[DeboraArlt]

I can check the reporting forms to VR next week. They used to want to know who i sending requests bit not sure de still do. Personally i do not need to know. But we can think a little bit of what mat be useful. May be e.g. separating rewquests from just viewing and those of actual downloads.

…

--- Debora Arlt SLU Artdatabanken Dept. of Ecology 018-67229, ***@***.***

________________________________ From: mathieuLU ***@***.***> Sent: 25 October 2024 15:56:17 To: biodiversitydata-se/biologging-sensor-client Cc: Debora Arlt; Assign Subject: Re: [biodiversitydata-se/biologging-sensor-client] statistics module + matomo (Issue #111) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Matomo is installed on the client side. I have a simple handmade way of storing all the requests on the API level. ***@***.***<biodiversitydata-se/biologging-sensor-API@14fd0b1> Are there specific statistics that we would need to have ? Because depending of the needs i may have to change a bit the structure of the basic stats module right now Matomo will register all the visits. on the API side i'm counting every requests sent. But do we want to track the origin ? for instance do we want to calculate which requests are coming from the client, and which are coming from a user using the API through scripts ? If at some point we want to implement a role based behaviour of the API, we will need anyway the source of the requests — Reply to this email directly, view it on GitHub<#111 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALSQMNYZYVQNGJGNUVDBMD3Z5JEYDAVCNFSM6AAAAABQFPGWUWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZXHA2DSNBTGY>. You are receiving this because you were assigned.Message ID: ***@***.***>

--- När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <https://www.slu.se/om-slu/kontakta-slu/personuppgifter/> E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <https://www.slu.se/en/about-slu/contact-slu/personal-data/>

[DeboraArlt]

For me personally it does not matter whether requests are made through the client or through scripts. But if distinguishing this is fairly easy and as you say we may need this anyway, then why not. Could be interesting to see the proportion of script requests, and whether this changes over time.

Otherwise I think it would be useful if we could separate different requests: for example requests for downloads of data, and all other requests (viewing data).

I checked what we report for SOS to VR: number of unique domains, number of unique users, number of requests

so a suggestion could be:
client: number of all visits (Matomo)
API: number of all requests (+ all requests via client)
API: number of request for download (+ all download requests via client)
API: number of unique domains (can get info via IP adderss?)
API: number of unique users (can get info via IP adderss?)

[mathieuLU]

Looks good to me.
Everything is covered except one thing that i will try to implement : the source of the request

Right now a request can be someone using the API with the script, the web client displaying the list of datasets, or the API itself getting for example the events details to use a specific filter for the records.

If I had a "request origin" field, then we can understand what it what

[mathieuLU]

After some research I think that's the best way to solve that is to start implementing some authentication protocol.
Because at some point we may need to be able to retrieve specific data to specific user (for instance the owner that will be granted the right to see event the restricted data).

So I can start using an api-key system, in order to detect when the client-api is requesting some data to the server

[mathieuLU]

ok i managed to implement it !
so now we have an API-key system on the API (which will allow us later on to deal with authentication) and an app-id system management, which allows us to know which app/website is accessing the API.
Of course these security compenents are optional now, which means that you can fetch data from the api without apiKey and without appId.
https://github.com/biodiversitydata-se/biologging-sensor-API/commit/ee84cdb93fdf21982e811d8692730b124fb2413b
80f7498

So right now the statistics on the API side are storing the origin (appId) of the request. We'll be able to know which calls are coming from users and which are coming from the API