feat: import datasets from scicat - #1390
Draft
olevski wants to merge 1 commit into
Draft
Conversation
olevski
marked this pull request as draft
July 22, 2026 12:41
sgaist
requested changes
Jul 22, 2026
| prefix=prefix, | ||
| ) | ||
| ) | ||
| return output[0] |
Collaborator
There was a problem hiding this comment.
Why not return early ? This would avoid building a whole list since only the first entry is used.
| if not isinstance(doi, str): | ||
| if doi is None: | ||
| raise errors.ValidationError( | ||
| message="Cannot get configuration for Envidat data connector because " |
Collaborator
There was a problem hiding this comment.
Suggested change
| message="Cannot get configuration for Envidat data connector because " | |
| message="Cannot get configuration for SciCat data connector because " |
| "the doi is missing from the payload." | ||
| ) | ||
| raise errors.ValidationError( | ||
| message=f"Cannot get configuration for Envidat data connector because the doi '{doi}' " |
Collaborator
There was a problem hiding this comment.
Suggested change
| message=f"Cannot get configuration for Envidat data connector because the doi '{doi}' " | |
| message=f"Cannot get configuration for SciCat data connector because the doi '{doi}' " |
| ) | ||
| if len(doi) == 0: | ||
| raise errors.ValidationError( | ||
| message="Cannot get configuration for Envidat data connector because the doi is a string with zero length." |
Collaborator
There was a problem hiding this comment.
Suggested change
| message="Cannot get configuration for Envidat data connector because the doi is a string with zero length." | |
| message="Cannot get configuration for SciCat data connector because the doi is a string with zero length." |
| res = await clnt.get(envidat_url, headers=headers) | ||
| if res.status_code != 200: | ||
| raise errors.ValidationError( | ||
| message="Cannot get configuration for Envidat data connector because Envidat responded " |
Collaborator
There was a problem hiding this comment.
Suggested change
| message="Cannot get configuration for Envidat data connector because Envidat responded " | |
| message="Cannot get configuration for Scicat data connector because SciCat responded " |
| raise errors.ValidationError( | ||
| message="Cannot get configuration for Envidat data connector because Envidat responded " | ||
| f"with an unexpected {res.status_code} status code at {res.url}.", | ||
| detail=f"Response from envidat: {res.text}", |
Collaborator
There was a problem hiding this comment.
Suggested change
| detail=f"Response from envidat: {res.text}", | |
| detail=f"Response from SciCat: {res.text}", |
|
|
||
|
|
||
| def create_scicat_metadata_url(doi: models.DOI) -> str: | ||
| """Create the metadata url for envidat from a DOI.""" |
Collaborator
There was a problem hiding this comment.
Suggested change
| """Create the metadata url for envidat from a DOI.""" | |
| """Create the metadata url for SciCat from a DOI.""" |
|
|
||
|
|
||
| def __get_rclone_s3_config_scicat(dataset: SchemaOrgDataset) -> S3Config: | ||
| """Get the S3 rclone configuration and source path from a dataset returned by scicat. |
Collaborator
There was a problem hiding this comment.
Suggested change
| """Get the S3 rclone configuration and source path from a dataset returned by scicat. | |
| """Get the S3 rclone configuration and source path from a dataset returned by SciCat. |
Member
Author
|
@sgaist thanks for the review. This is very much still WIP. I will address your comments though. But I will remove the draft on the PR when it is ready. I apologize if I was late when marking this as draft. I think I did it right away, but maybe I forgot. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the ability to import envidat datasets into Renku.