Dataset with embargo (isPublic=false) : download button, data available, ...

[mathieuLU]

2 cases when the download button should not be displayed :

• no archive to be downloaded (how to test ? curl ?)
• dataset is saying isPublic=false

display the embargo date instead ?

in the snippet, the embargo should be mentionned

Make sure that the API cannot provide to public when isPublic=false ? how to make it available in the client then ??

maybe a subset should be provided ?

in any case, the "total records" in the dataset listing is supposed to show the real number of records of the dataset. Not what's in the database

[mathieuLU]

ok so for a public dataset (no emargo, isPublic=true), we have this now :
snippet =>

detail =>

and so when dataset with embargo :
snippet (no button + embargo date) =>

detail =>

does that look better @DeboraArlt @blacksparrowhawk

[DeboraArlt]

looks ok for me. perhaps would be better with an explicit statement about the data set not being currently public, but we can go for this easy solution now. and see what users say.

[mathieuLU]

ok so the question is still open. How do we deal with the datasets with embargo ? (isPublic = false)
Will the data be in the database anyway with some authentication/filter needed ? How do we use the visualisation tools ? Do we only let a sample data available to download ? etc...

several options :

1- all the data is in the database, but not available to download/retrieve from the API.
Which means that the web client (which is using the API) can't use the data for the visualisation tool. SO we need to specify a way for the client to access the data anyway. And do we want the client to access the first data available as usual ? (maybe a risk to display protected data ?)

2- only a sample available in the database. This way, the database remains 100% public. No need to authenticate/filter or anything. But it means that a sample has to be created when we publish the dataset. And then it has to be clearly specified somewhre that only a sample is available. Which means that we may display the "total number of records of the dataset" (for real) and the "total of records AVAILABLE in the database"

3- maybe a mix of these 2 ? hidden data in the database + public sample available for download/API ?

@blacksparrowhawk what are you views on that ?

[mathieuLU]

i add to the discussion the comment that Debora wrote :

`
I don’t know exactly how it works, but for SOS they talk about the public index, end a sensitive/protected index. I thinks it’s a bit like two separate databases. Normally one get’s only data from the public index. But if you have permission you also get data from the protected index (according what your permissions allows access to).

For me option 2 works too. The private/embargo dataset provides a subset that is available public. But it seemed like Johan did not quite like this option. But let’s say the subset is only one animal. One could also select just part of the total time series, e.g. just half a year instead of a full annual cycle, i.e. a simple subset for illustrative purpose. I cannot see that anyone would/could misuse that data.
`

[DeboraArlt]

From the discussion today 24 Sept (Mathieu and Debora) it sounds like the easier option is having only public data in our database. That's the data we dipslay and that one can access via the API. No permissions etc. needed. It would be good though to have a public sample data from the non-public (embargo) datasets. This sample data could be explored (graphs) and also downloaded. The data owner decides what he/she wants to release as a sample - could be data for one animal only, and for a selected time period and/or selected location only, i.e. data that the owner deems as unsensitive to show in graphs and have downloadable via the API.
I suggest to add to the metadata - so metadata include information about total no. of records in the dataset, and some information about what the selected public sample is. I will have a look at our metadata form and see where this could be added.

[mathieuLU]

has been decided :

• we keep the field isPublic for the dataset. It tells if the data can be downloaded or not, and accessible or not through the API. For the whole dataset
• we add a field isPublic for each animal (default = true). when animalIsPublic =false, then the data linked to that animal can't be downloaded (excluded from the archive !!) and can't be fetched through the API.
• the field totalRecords keeps telling how many records are in the dataset, including the non-public ones
• we add a field dataAvailability at the dataset level, text field, that explains if/why there are hidden data in the dataset, and why there could be a difference between the totalRecords and the realiti in the downloadable file.

[mathieuLU]

ok so :

• field isPublic (datasetlevel) is now used as a filter on records/events in the API => https://github.com/biodiversitydata-se/biologging-sensor-API/commit/350b5456eea84346551eb6746df4dcd3fb818b80
• field isPublic is added on the organism level (and as well at the event/record level, inherited from organism) => https://github.com/biodiversitydata-se/biologging-sensor-API/commit/7b1eb94ee1c04449ef2333292c886c48fa02e8ee and https://github.com/biodiversitydata-se/biologging-sensor-API/commit/61e991902d79e28e60ed6f5ca8b0833ab44007cc
• field dataAvailability is added as well, and displayed in the snippet and the details of a dataset => 2457a8e
  

[mathieuLU]

From Debora's email :

1 accessRights

(3 options, can be extended later if needed)

data freely accessible: All data contained in the dataset are freely accessible by direct download from the website.

data partly accessible: Access to the full dataset is restricted (embargo until YYYY-MM-DD). A sample of records (individual records, or selected events or organisms) is made public by the dataset owner and can be viewed and downloaded from the website. For more information about the full dataset and access please contact the dataset owner.

restricted access: Access to the full dataset is restricted (embargo until YYYY-MM-DD). No data can be viewed or downloaded from the website. For more information about the dataset and access please contact the dataset owner.

2 intellectualRights? Used by the EML schema, is part of the GBIF metadata profile.

I added this to the data model github pages.

Not sure though what the values should be.

In principle this could be inherited – by license that is mandatory to provide. In this case the chosen license triggers the value here, would be for example:

for CC-BY:

Data are free to use, share (i.e. copy and redistribute in any medium or format for any purpose) and adept (i.e. remix, transform, and build upon the material for any purpose) provided that appropriate credit is given, a link to the license and it is indicated if changes were made.

https://creativecommons.org/licenses/by/4.0/

for CC0:

Data are free to use, users can copy, modify, and distribute for any purpose.

https://creativecommons.org/publicdomain/zero/1.0/

But what of a dataowner want’s to write something additional?

3 Funding – we talked about fundingSource as an array of string.

Looking around there is no DwC and nothing for GBIF metadata.

DataCite has an object/part fundingReference: https://datacite-metadata-schema.readthedocs.io/en/4.5/properties/fundingreference/

Definition: Information about financial support (funding) for the resource being registered.

They have funder name etc including award number. I thing we should keep it simple (at least for now) and just use

funderName for the name of the funding provider, e.g. Swedish Research Council VR.

The EML has something very similar/same:

So let’s go for funderName.

I added this to the data model github pages. But can be modified of course J

[mathieuLU]

updated in the datamodel, the API, and the client 8547783

[mathieuLU]

there's still one open question here
With the example of the movebank dataset : we'll have a public dataset, no restricted access, a number of records > 0 (the total number of real records for the dataset) , but not data to be downloaded.
So the download button will appear, even if nothing can be downloaded...

Maybe i should specify that a dataset with a relatedIdentifier = "Movebank" should not have a download button. I guess it's the easiest way

[DeboraArlt]

Perhaps easiest is Good enough :) Or: Is it easy too to do something like: Add båutton that looks like e link to an external site and actually link to the movebank page? Ideal would be linking to the dataset description at movebank but i do not think they provide such urls - but user has to search for a dataset. But then a link to movebank searchpage would be good. Is that easy?

…

--- Debora Arlt SLU Artdatabanken Dept. of Ecology 018-67229, ***@***.***

________________________________ From: mathieuLU ***@***.***> Sent: 31 January 2025 09:41:37 To: biodiversitydata-se/biologging-sensor-client Cc: Debora Arlt; Mention Subject: Re: [biodiversitydata-se/biologging-sensor-client] Dataset with embargo (isPublic=false) : download button, data available, ... (Issue #108) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. there's still one open question here With the example of the movebank dataset : we'll have a public dataset, no restricted access, a number of records > 0 (the total number of real records for the dataset) , but not data to be downloaded. So the download button will appear, even if nothing can be downloaded... Maybe i should specify that a dataset with a relatedIdentifier = "Movebank" should not have a download button. I guess it's the easiest way — Reply to this email directly, view it on GitHub<#108 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALSQMNZOO7SVX5GNXSAC2YL2NMZMDAVCNFSM6AAAAABOFVSC52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRWGY2TQMJTGY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

--- När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <https://www.slu.se/om-slu/kontakta-slu/personuppgifter/> E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <https://www.slu.se/en/about-slu/contact-slu/personal-data/>