- -[**Renku Legacy:**](/docs/users/migrate-v1-v2/) Guidelines on how to migrate your code repositories from Legacy Renkulab Gitlab to your own code repository provider, and keep -
- -## Use cases +## Where do you want to start? -| Researchers | Educators | Event organizers | -| :--------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | -|  |  |  | -| **Unified Research**Connect your entire research workflow in one place, and collaborate across specialties without technical barriers.
| **Computing Courses made easy**Help your students focus on the material, not getting lost during setup. Ideal for project-based coursework and time-sensitive workshops.
| **Seamless Events**Focus on innovation, not setup and infrastructure. Provide a consistent environment for all teams, and get participants coding and collaborating right away.
| +| Researchers & Data Scientists | Educators | Event & Hackathon Organizers | +| :----------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------: | +|  |  |  | +| Connect your entire research workflow in one place, and collaborate across specialties without technical barriers. | Help your students focus on the material, not getting lost during setup. Ideal for project-based coursework and time-sensitive workshops. | Focus on innovation, not setup. Provide a consistent environment for all teams and get participants coding and collaborating right away. | +| [**Start here**](/docs/users/getting-started/tutorial-start) | [**Start here**](/docs/users/use-cases/teaching/) | [**Start here**](/docs/users/use-cases/events) | ## What's new - Check out our latest blog posts: diff --git a/docs/docs/10-users/20-getting-started/10-tutorial-start.md b/docs/docs/10-users/20-getting-started/10-tutorial-start.md index 926baad947..e90df93bd6 100644 --- a/docs/docs/10-users/20-getting-started/10-tutorial-start.md +++ b/docs/docs/10-users/20-getting-started/10-tutorial-start.md @@ -1,5 +1,15 @@ # Create a RenkuLab Account +:::info Before you begin + +**Estimated time:** ~20 minutes + +**You will need:** + +- A **GitHub or GitLab account**: required in step 4 (Connect Code) to link a code repository. + +::: + To begin using Renku, you need to create an account. Follow these steps: 1. Visit [RenkuLab.io](https://renkulab.io/) diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/15-connect-to-azure-blob-storage.md b/docs/docs/10-users/40-data/guides/00-connect-data/15-connect-to-azure-blob-storage.md new file mode 100644 index 0000000000..38d8accd45 --- /dev/null +++ b/docs/docs/10-users/40-data/guides/00-connect-data/15-connect-to-azure-blob-storage.md @@ -0,0 +1,137 @@ +--- +title: Azure Blob Storage +--- + +# How to connect to Azure Blob Storage + +:::info + +This guide assumes Storage Account-level permissions. If you had Azure IAM (Identity) rights, you could use "Service Principals" or "Entra ID" logins, which remains outside the scope of this documentation. + +::: + +## Public access + +This set-up allows you to provide access to data publicly without requiring authentication with **read-only** access. For setting this up, you just need your Azure storage account name and the container name. Bear in mind that access is controlled entirely at container level. + +### Azure portal configuration + +1. **Account level:** Go to **Settings > Configuration** and set **Allow Blob anonymous access** to **Enabled**. + ++  +
+ +2. **Container level:** Go to **Data storage > Containers**, select your container, and click **Change access level**. + - Select **Container**, which allows anonymously listing files and downloading. + ++  +
+ +### Set up in Renku + +1. Under **Data** section click on **+** button +2. Go to the tab **+ Create a data connector** and select **azureblob**. +3. Click on **Next** + ++ +
+ +4. Set the following parameters in **Connection information**: + 1. **Source path**: the `container_name` as specified in the storage account you are using (e.g. `my-container`). You can also mount a sub-folder by appending it to the bucket name with a slash, e.g. `my-container/sub-folder`. + 2. **Account Name**: the storage account name, e.g. `elisabettestsa` + +5. Click on **Test connection** and if succeeds, click **Next** + ++ +
+ +6. On the last page, fill in the final details for your data connector, namely: + 1. **Name**: pick any name for the data connector (e.g. `data`) + 2. **Owner:** select the namespace of the data connector (e.g. the user's, project's or a group's). + 3. **Visibility:** decide whether the data connector should be Public or Private. + 4. **Read-only**: keep it as read-only access. + +7. Click on **+ Add connector**. + ++ +
+ +## Restricted access + +Use this option to share data securely with your collaborators and stakeholders, controlling their access rights (e.g. read-only, write and delete files). + +### Azure portal configuration + +1. Go to **Data storage > Containers > [Your container] > Settings > Shared access tokens**. Select the desired permissions (e.g. **Read** and **List** for read-only access, select on top **Add**, **Create** and **Write** for read and write access, avoid **Delete** if you want an append/upload-only environment). + ++  +
+ +2. Click on the button **Generate SAS token and URL**. +3. Copy the generated **SAS URL** field. + ++  +
+ +### Set up in Renku + +Proceed to steps 1 to 3 from the [Public access set-up](#set-up-in-renku). + +4. In **Connection information**, click on **Show full options list** and set the following parameters: + 1. **Source path**: the storage account name, e.g. `elisabettestsa`. You can also mount a sub-folder by appending it to the bucket name with a slash, e.g. `my-container/sub-folder`. + 2. **sas_url**: the `blob_sas_url` as generated in your storage account. + +Continue with steps 5 to 7 as presented in the [public access set-up](#set-up-in-renku). + +:::warning + +You will need to share the sas_url value with your collaborators since they will be requested to enter the credentials. This variable will be stored as a user secret. + +::: + +:::info + +The most advanced way to manage external access in Azure is through controlled access with revocation, which allows you to revoke access immediately without changing account keys. You need to create the policy under **Containers > [Your Container] > Settings > Access policy** and click **+ Add policy**. Define a name and the permissions. When you are creating the shared access token, select the policy you just created under **Stored access policy**. If you delete the policy in Azure, access to the container will be instantly revoked. + +::: + +## Full access to your storage account + +This set-up allows you to have access to your full Azure storage account, with the access rights that you decide to set-up. + +### Azure portal configuration + +1. Go to **Security + networking > Shared access signature**. Select **Blob** service, **Service/Container/Object** types, and check **ONLY Read** and **List**. + ++  +
+ +### Set up in Renku + +Proceed to steps 1 to 3 from the [Public access set-up](#set-up-in-renku). + +4. Set the following parameters in **Connection information**: + 1. **Source path**: the `container_name` as specified in the storage account you are using (e.g. `my-container`). You can also mount a sub-folder by appending it to the bucket name with a slash, e.g. `my-container/sub-folder`. + 2. **Account Name**: the storage account name, e.g. `elisabettestsa` + +5. Click on **Test connection** and if succeeds, click **Next** + ++ +
+ +6. On the last page, fill in the final details for your data connector, namely: + 1. **Name**: pick any name for the data connector (e.g. `data`) + 2. **Owner:** select the namespace of the data connector (e.g. the user's, project's or a group's). + 3. **Visibility:** decide whether the data connector should be Public or Private. + 4. **Read-only**: do not uncheck this box, or the data connector will not work properly. + +7. Click on **+ Add connector**. diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/60-add-dataset-from-eth-research-collection.md b/docs/docs/10-users/40-data/guides/00-connect-data/60-add-dataset-from-eth-research-collection.md index 5749cbeb37..29dafd3310 100755 --- a/docs/docs/10-users/40-data/guides/00-connect-data/60-add-dataset-from-eth-research-collection.md +++ b/docs/docs/10-users/40-data/guides/00-connect-data/60-add-dataset-from-eth-research-collection.md @@ -36,6 +36,6 @@ In your project’s dashboard: :::note -For examples of data from the ETH Research Collection to add to your project, see [Data & Code for Reproducibility](/docs/users/use-cases/Demo-Workshop-Material). +For examples of data from the ETH Research Collection to add to your project, see [Data & Code for Reproducibility](/docs/users/use-cases/demo-workshop-material). ::: diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/add-azureblob-10.png b/docs/docs/10-users/40-data/guides/00-connect-data/add-azureblob-10.png new file mode 100644 index 0000000000..a1edc57d43 Binary files /dev/null and b/docs/docs/10-users/40-data/guides/00-connect-data/add-azureblob-10.png differ diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-full-access-sas.png b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-full-access-sas.png new file mode 100644 index 0000000000..e52db65794 Binary files /dev/null and b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-full-access-sas.png differ diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-public-account-config.png b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-public-account-config.png new file mode 100644 index 0000000000..cc374aba99 Binary files /dev/null and b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-public-account-config.png differ diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-public-container-access.png b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-public-container-access.png new file mode 100644 index 0000000000..926813981a Binary files /dev/null and b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-public-container-access.png differ diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-restricted-sas-permissions.png b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-restricted-sas-permissions.png new file mode 100644 index 0000000000..e1b6b1a217 Binary files /dev/null and b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-restricted-sas-permissions.png differ diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-restricted-sas-url-public.png b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-restricted-sas-url-public.png new file mode 100644 index 0000000000..86c46a5197 Binary files /dev/null and b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-restricted-sas-url-public.png differ diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-restricted-sas-url.png b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-restricted-sas-url.png new file mode 100644 index 0000000000..64e861a093 Binary files /dev/null and b/docs/docs/10-users/40-data/guides/00-connect-data/azure-portal-restricted-sas-url.png differ diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/azureblob-20.png b/docs/docs/10-users/40-data/guides/00-connect-data/azureblob-20.png new file mode 100644 index 0000000000..e3c53d34ca Binary files /dev/null and b/docs/docs/10-users/40-data/guides/00-connect-data/azureblob-20.png differ diff --git a/docs/docs/10-users/40-data/guides/00-connect-data/azureblob-30.png b/docs/docs/10-users/40-data/guides/00-connect-data/azureblob-30.png new file mode 100644 index 0000000000..fc62f4f3e9 Binary files /dev/null and b/docs/docs/10-users/40-data/guides/00-connect-data/azureblob-30.png differ diff --git a/docs/docs/10-users/50-code/guides/10-add-code-repository-to-project.md b/docs/docs/10-users/50-code/guides/10-add-code-repository-to-project.md index 64b50f947d..9120f47995 100755 --- a/docs/docs/10-users/50-code/guides/10-add-code-repository-to-project.md +++ b/docs/docs/10-users/50-code/guides/10-add-code-repository-to-project.md @@ -10,6 +10,7 @@ with GitHub/GitLab](connect-renku-account-to-github-or-gitlab-account). In the project page: 1. Click on ➕ in the Code Repositories section. +
diff --git a/docs/docs/10-users/60-sessions/guides/00-environments/45-use-your-own-docker-image-for-renku-session.md b/docs/docs/10-users/60-sessions/guides/00-environments/45-use-your-own-docker-image-for-renku-session.md index b5301adde7..577ed2a95b 100755 --- a/docs/docs/10-users/60-sessions/guides/00-environments/45-use-your-own-docker-image-for-renku-session.md +++ b/docs/docs/10-users/60-sessions/guides/00-environments/45-use-your-own-docker-image-for-renku-session.md @@ -32,51 +32,21 @@ In the project page: - if the image is hosted on DockerHub: - `renku/renkulab-py:3.10-0.24.0` - `continuumio/anaconda3:2024.06-1` - - if the image is hosted on gitlab.renkulab.io: - - `registry.renkulab.io/laura.kinkead1/n2o-pathway-analysis:980f4a3` + - if the image is hosted on GitHub: + - `ghcr.io/swissdatasciencecenter/renku-frontend-buildpacks/base-image:0.0.8` - The image identifier should be in the format that works with `docker pull` -4. Depending on the image you’re using, you’ll need to fill in the **Advanced settings**. See the information below for how to fill it in: - - :::danger - - This part is important! Please read carefully. - - ::: - - I’m using an image created by **Renku** and that is **newer** than version 0.24.0 (the version number is in the image tag). - - The only additional parameter you have to provide in the session launcher creation dialog is the `Default URL` and this should be set to `/lab`. - -  - - - I’m using an image created by **Renku** and that is **older** than version 0.24.0 (the version number is in the image tag). - - :::note - - If you are working with an image in a launcher where the **launcher was created before November 27, 2024**, the launcher was migrated automatically with the new Renku release to include the necessary advanced settings. The instructions below apply only to new session launchers you are creating for the first time. - - ::: - - For Renku base images of version 0.24.0 or older (or images that are based on these images), you have 2 options: - 1. **Option 1:** Upgrade your base image to 0.25.0 or newer. This can be done by going into the settings of the Renku 1.0 project that builds the image, and accepting the updates. Or, directly update your Dockerfile to refer to the newer base image. - 2. **Option 2:** Provide additional configuration in the session launcher. Here is an example configuration needed to run a Renku base image of version 0.24.0 or older: - - **Container Image**: `renku/renkulab-py:3.10-0.24.0` or whatever image you are trying to use - - **Default URL**: `/lab` (or `/rstudio` if you are using `renku/renkulab-r` or `renku/renkulab-bioc`). - - **Mount Directory**: `/home/jovyan/work` - - **Working Directory**: `/home/jovyan/work` - - **UID**: `1000` - - **GID**: `100` - - **Command ENTRYPOINT**: `["sh", "-c"]` - - **Command Arguments**: - - ```json - [ - "/entrypoint.sh jupyter server --ServerApp.ip=0.0.0.0 --ServerApp.port=8888 --ServerApp.base_url=$RENKU_BASE_URL_PATH --ServerApp.token=\"\" --ServerApp.password=\"\" --ServerApp.allow_remote_access=true --ContentsManager.allow_hidden=true --ServerApp.allow_origin=* --ServerApp.root_dir=\"/home/jovyan/work\"" - ] - ``` - - - I’m using an image created **somewhere else** (not by Renku). - - You need to fill in the **Advanced Settings** for your image to work on RenkuLab. See [Example image configurations for common front ends](#example-image-configurations) below. +4. If needed, fill in the **Advanced settings** to tell Renku how to run your image. The available fields are: + + - **Container Image**: the image identifier to use (e.g. `renku/renkulab-py:3.10-0.24.0`) + - **Default URL**: the path appended to the session URL to open the front end (e.g. `/lab` for JupyterLab, `/rstudio` for RStudio) + - **Mount Directory**: the directory inside the container where Renku mounts your project files (e.g. `/home/jovyan/work`) + - **Working Directory**: the directory the session opens in (e.g. `/home/jovyan/work`) + - **UID**: the user ID the container process runs as (e.g. `1000`) + - **GID**: the group ID the container process runs as (e.g. `100`) + - **Command ENTRYPOINT**: the shell entrypoint (e.g. `["sh", "-c"]`) + - **Command Arguments**: the command passed to the entrypoint to start the front end server + + See [Example image configurations for common front ends](#example-image-configurations) below for ready-made values per front end. 5. Select the **Resource class** that best fits your expected computational needs. diff --git a/docs/docs/10-users/60-sessions/guides/00-environments/use-your-own-docker-image-for-renku-session-30.png b/docs/docs/10-users/60-sessions/guides/00-environments/use-your-own-docker-image-for-renku-session-30.png deleted file mode 100755 index 6a886e939e..0000000000 Binary files a/docs/docs/10-users/60-sessions/guides/00-environments/use-your-own-docker-image-for-renku-session-30.png and /dev/null differ diff --git a/docs/docs/10-users/60-sessions/guides/98-create-streamlit-app.md b/docs/docs/10-users/60-sessions/guides/98-create-streamlit-app.md deleted file mode 100644 index 20d7d22371..0000000000 --- a/docs/docs/10-users/60-sessions/guides/98-create-streamlit-app.md +++ /dev/null @@ -1,50 +0,0 @@ -# How to create a Streamlit app on top of your project - -:::info - -This documentation is out-of-date and will be updated soon! - -::: - -As of now, it is still possible to launch your Streamlit app on top of your Renku 2.0 project as follows. - -1. Create a fork of this repository: https://gitlab.renkulab.io/rok.roskar/streamlit-v2-example, we will refer to the fork as project-fork. - -- -
- -2. In _project-fork_, modify line 8 in the file `jupyter_noteboox_config.py`to the path to your python entry file for Streamlit within your Renku 2.0 project ( e.g. `work/name-fork-project/helloworld.py` or `work/main-project-repository/streamlit/app.py` or `work/repository-2/app.py` ) - - ⚠️ Do not forget to add `work/` at the beginning of the path ! - -- -
- -3. Add all the package dependencies you will need to run your Renku 2.0 project in the requirements.txt file or in the environment.yml , as explained [here](./environments/install-packages-on-the-fly-in-your-session). - -- -
- -4. Commit your changes for _project-fork_. -5. Select **Deploy(5a)/Container Registry(5b)** on the left menu of _project-fork_. -6. Click on top of your image registry to access the tags **(6)**. - -- -
- -7. Copy the latest tagged image (registry_link). - -- -
- -8. Add a session launcher to your Renku 2.0 project the following **Custom Environment** set-up \*\*\*\*as described in [How to use your own docker image for a Renku session](./environments/use-your-own-docker-image-for-renku-session). Consider the following two parameters: - 1. **Container Image:** `*registry_link*` - 2. **Default URL:** `/streamlit` -9. [Optional] If the entry point for your streamlit app will be hosted in _project-fork,_ add _project-fork_ code repository to your Renku 2.0 project. - -You can now start a session with your new streamlit launcher to access the streamlit app directly in your browser. diff --git a/docs/docs/10-users/60-sessions/guides/90-create-custom-environment-and-launcher-for-shiny-app.md b/docs/docs/10-users/60-sessions/guides/_90-create-custom-environment-and-launcher-for-shiny-app.md similarity index 100% rename from docs/docs/10-users/60-sessions/guides/90-create-custom-environment-and-launcher-for-shiny-app.md rename to docs/docs/10-users/60-sessions/guides/_90-create-custom-environment-and-launcher-for-shiny-app.md diff --git a/docs/docs/10-users/60-sessions/guides/container-register.png b/docs/docs/10-users/60-sessions/guides/container-register.png deleted file mode 100644 index 9dafe794f2..0000000000 Binary files a/docs/docs/10-users/60-sessions/guides/container-register.png and /dev/null differ diff --git a/docs/docs/10-users/60-sessions/guides/dashboard.png b/docs/docs/10-users/60-sessions/guides/dashboard.png deleted file mode 100644 index 2aea852229..0000000000 Binary files a/docs/docs/10-users/60-sessions/guides/dashboard.png and /dev/null differ diff --git a/docs/docs/10-users/60-sessions/guides/jupiter-notebook.png b/docs/docs/10-users/60-sessions/guides/jupiter-notebook.png deleted file mode 100644 index d1bbba98fe..0000000000 Binary files a/docs/docs/10-users/60-sessions/guides/jupiter-notebook.png and /dev/null differ diff --git a/docs/docs/10-users/60-sessions/guides/required-files.png b/docs/docs/10-users/60-sessions/guides/required-files.png deleted file mode 100644 index cc917bfe14..0000000000 Binary files a/docs/docs/10-users/60-sessions/guides/required-files.png and /dev/null differ diff --git a/docs/docs/10-users/60-sessions/guides/streamlit-v2-example.png b/docs/docs/10-users/60-sessions/guides/streamlit-v2-example.png deleted file mode 100644 index 941bea377a..0000000000 Binary files a/docs/docs/10-users/60-sessions/guides/streamlit-v2-example.png and /dev/null differ diff --git a/docs/docs/10-users/70-collaboration/index.md b/docs/docs/10-users/70-collaboration/index.md index b07e6d7fd5..6c8aef6503 100644 --- a/docs/docs/10-users/70-collaboration/index.md +++ b/docs/docs/10-users/70-collaboration/index.md @@ -1,5 +1,7 @@ # Collaboration +In Renku, a **project** is where you do your work: it holds data, code, and sessions, while a **group** is about sharing that work with others, giving members controlled access to all projects within it. You can add individual members directly to a project, or organise multiple projects under a group to manage access and visibility in one place. + ```mdx-code-block import DocCardList from '@theme/DocCardList'; diff --git a/docs/docs/10-users/80-use-cases/10-research/01-data-exploration.md b/docs/docs/10-users/80-use-cases/10-research/01-data-exploration.md new file mode 100644 index 0000000000..3557bbeb9e --- /dev/null +++ b/docs/docs/10-users/80-use-cases/10-research/01-data-exploration.md @@ -0,0 +1,63 @@ +--- +title: Data Exploration & Exploratory Data Analysis +--- + +# Your data ready in seconds + +## Connect any cloud data source directly to your session: Renku mounts it for you + +Every researcher and data scientist faces the same ritual: you have a new dataset to explore, so you download the data you have received and open a terminal to start a transfer that will take hours: you may copy files to a remote server, update hardcoded paths across multiple notebooks/scripts, and hope your colleague has the same folder structure. By the time the data is "ready," you have already lost about half a day to logistics rather than science. + +This problem turns much worse when you have multi-terabyte datasets in fields like climate science, genomics, or medical imaging. You cannot simply `scp` a 5 TB satellite imagery archive to a laptop, and even cloud compute instances may run out of local disk mid-analysis. Hence, data scientists and researchers spend a significant amount of time managing data movement. + +Renku eliminates this entirely. Instead of downloading data to wherever your analysis runs, Renku connects your analysis to wherever your data lives, so data is kept at source, permanently for every collaborator on your project. + +### Set up and reuse data connectors in your Renku projects + +Setting up the project [Exploring a 50-Year European Climate Archive Without a Single Download](https://renkulab.io/p/renku-team/exploring-a-50-year-european-climate-archive-without-a-single-download) demonstrates Renku capabilities for exploratory data analysis. We will set up three data connectors: a **public S3 bucket in AWS** hosting the [ERA5 reanalysis dataset](https://registry.opendata.aws/nsf-ncar-era5/), a **DOI-linked Zenodo** dataset 4-years subset for [daily total precipitation](https://zenodo.org/records/17098120) and a **private Polybox** read-write data connector protected with credentials with some station observations (syntehtically create for demonstration purposes, which you can recreate by uploading the original [dataset](https://drive.google.com/drive/folders/1mdpfrlShIGksz0Ent_POihU1kqMggD7o?usp=sharing) in your favorite data storage provider. At this initial stage, we usually do not have a code repository for version control, and we usually create notebooks on-the-fly. In Renku you can either store these notebooks directly in a data connector with read and write access, or you can download them directly to your computer from the session, and upload them after session restart to your local disk. + + + +**Create the project from scratch** + +1. From a new project page, go to the **Data** section and click **+** to add a connector. +2. For the **[ERAS reanalysis dataset](https://registry.opendata.aws/nsf-ncar-era5/)**: Go to the **+ Create a data connector** tab, and select **s3** as the storage type and enter the bucket name `nsf-ncar-era5` and region `us-west-2`. +3. Click **Test connection** and a green confirmation appears immediately. +4. Name the connector as `ERA5 Climate Archive` and set the mount point to `era5` (under Advanced Settings), then click **+ Add connector**. +5. Repeat steps 1 to 4, for the other data sources: + - **[ERA5 daily total precipitation](https://zenodo.org/records/17098120)**: In the **Link a data connector** tab paste the DOI of the Zenodo dataset, `https://doi.org/10.5281/zenodo.17098120`, and click on the **Link** button. + - **Station observations**: Go to the **+ Create a data connector** tab, and select your storage provider (e.g. Polybox/Shared as explained in the [how-to guides](../../data/guides/connect-data)). Click on **Next** and provide connection details. After that, name the connector as `Field Observations 2020-2023` and set the mount point to `observations`. +6. Create a global session launcher (e.g. Python Datascience - Jupyter), and click on **Launch**. The selected UI (e.g. JupyterLab) file browser shows all mount points populated. +7. Create a notebook and begin analysis without any download, path configuration or `os.environ` juggling steps. You can find [here](https://drive.google.com/drive/folders/1mdpfrlShIGksz0Ent_POihU1kqMggD7o?usp=sharing) an example Notebook that you can upload in your session (e.g. by right-clicking on the UI file browser). + +### One-time configuration, available in every session + +Renku supports access to a wide range of cloud and institutional storage systems through [data connectors](../../data/data), including Amazon S3 buckets, Azure Blob Storage, SwitchDrive and PolyBox (via WebDAV), OpenBIS, GoogleDrive, Dropbox, SFTP servers such as the EPFL NAS, and any publicly archived dataset accessible by DOI from repositories such as Zenodo, Dataverse, or EnviDat. You configure a connector once by providing the endpoint, container or bucket name, and any required credentials, and that connector is permanently linked to your project. Every subsequent session, whether launched by you or a collaborator, finds the data already mounted and ready to read with no additional setup, except for entering the credentials when accessing protected data connectors for the first time. + +For **DOI-referenced datasets**, the process is even simpler: paste the DOI into **Link a data connector** tab, and Renku resolves it to the specific dataset version on the upstream repository. Your project now references that exact dataset, pinned by DOI, rather than a URL that may change or disappear without warning. See [How to connect data from a data repository](../../../data/guides/connect-data/connect-data-from-data-repositories) for the full walkthrough. + +### Mounted, not transferred + +When Renku connects a session to a data source, it does not copy files to local disk. It mounts the remote storage as a directory in the session filesystem, making the data appear local to your analysis code while transparently reading it on demand. + +The practical consequence is immediate: a 10 TB raw data archive is accessible in your session within seconds, and your first `pd.read_csv()` or `xr.open_dataset()` call reads only the bytes it needs. You do not have to wait for a full transfer and you can browse the directory structure of massive archives at interactive speed. Lazy-loading libraries such as `xarray`, `dask`, and `arrow` pair naturally with this approach, giving you efficient access to petabyte-scale datasets. + + + +### Your data never leaves your infrastructure + +Renku stores only the _connection configuration_ of a data connector, including the endpoint address, container name, and optionally a reference to your credential secret, but never the data itself. When a session reads a file, the read request travels directly from the session environment to your storage provider. Sensitive institutional datasets, patient records under IRB approval, and proprietary data under NDA remain entirely within your storage infrastructure. Only a user with valid credentials can read them through a Renku session. + +For public datasets accessed by DOI, no credentials are involved at all and Renku reads directly from the upstream repository with no intermediate storage. + +### Stable paths across environments and collaborators + +One of the most misleading causes of broken analysis pipelines is hardcoded data paths. A notebook that reads `/home/username/Downloads/climate/era5_2024.nc` on one machine fails silently on another. Renku solves this by giving each data connector a fixed, configurable mount point inside the session, for example `/work/era5/`, regardless of who launched it. Your code references that path, and Renku guarantees the data is there. + + + +:::tip Remember + +The data connector stores only the _connection configuration_, not the data itself. Your raw files never leave your storage provider, instead Renku requests them on demand, exactly as a mounted NFS share works on an HPC cluster. Revoke the access credentials and Renku immediately loses access. There is no copy on Renku to worry about. + +::: diff --git a/docs/docs/10-users/80-use-cases/10-research/02-model-development.md b/docs/docs/10-users/80-use-cases/10-research/02-model-development.md new file mode 100644 index 0000000000..c42656c1aa --- /dev/null +++ b/docs/docs/10-users/80-use-cases/10-research/02-model-development.md @@ -0,0 +1,44 @@ +--- +title: Model & Algorithm Development +--- + +# Reproducible environments without the Docker detour + +## Point at your dependencies file and Renku builds a session image automatically for you and your collaborators + +The "works on my machine" issue is a constant burden that slows down scientific collaboration. Traditional solutions are either fragile (a long `README.md` listing exact package versions) or steep (learning Docker well enough to write and maintain a `Dockerfile` for a non-trivial scientific stack). Neither is a reasonable expectation for a researcher whose primary expertise is hydrology, structural biology or econometrics rather than software engineering. + +Renku has a smooth solution: the environment specification you already write in a `requirements.txt`, `environment.yml`, `renv.lock` or similar is enough. Renku reads it, builds the container automatically, and every subsequent session runs inside that container. You do not have to worry about the Dockerfile, nor `conda env create` nor exchange "did you install the right CUDA version?" emails. + +### Environment builds from files you already maintain** + +When you create a session launcher in Renku and point it to a code repository, Renku scans the repository for a recognised environment definition file and builds a container image from it. Supported file types include: + +- **Python**: `requirements.txt`, `environment.yml` (conda), `pyproject.toml` (Poetry/PEP 517) +- **R**: `renv.lock` (renv package manager) +- **Julia**: `Project.toml` / `Manifest.toml` + +The build happens once; subsequent session launches use the built image unless you push a change to your dependency file and click on **Rebuild** to your session launcher. You do not write a Dockerfile, manage a container registry, or deal with base image selection: Renku handles the container plumbing so you can stay in the science layer. For full details, see [How to create an environment with custom packages installed](../../../sessions/guides/environments/create-environment-with-custom-packages-installed). + +### Direct integration with GitHub and GitLab: every push is immediately available** + +Renku connects directly to your existing code repositories on GitHub or GitLab. Once you [link your Renku account to your GitHub or GitLab account](../../../code/guides/connect-renku-account-to-github-or-gitlab-account), you can point a session launcher at any repository you have access to. Every time you push a commit, after fixing a bug in your analysis script, updating a dependency, or adding a new notebook, it reflects on the code repository. + +This means your development cycle is a standard git workflow: edit locally, push to GitHub, launch a Renku session with the latest version of your code already present. No manual file uploads, no `rsync` to a remote server, no "remember to pull before you run" instructions to collaborators. + +### Private repositories, without exposing credentials to collaborators** + +If your code repository is private, either because it contains unpublished methodology, proprietary algorithms, or pre-publication data processing scripts, Renku handles authentication transparently. You authorise Renku to access the repository using your GitHub or GitLab credentials. Collaborators who launch sessions from the same project use their own GitHub/GitLab credentials to access. In either case, no one's personal access token is exposed through the Renku interface. No one without access to your code repository can access the code. Even more, if you create a session launcher from the dependencies in your private code repository, only those who have access to the code repository can launch a session. See [How to create an environment from a private code repository](../../../sessions/guides/environments/create-environment-with-custom-packages-private-code-repository) for the step-by-step setup. + +### Global environments for rapid prototyping before you commit to a dependency set** + +Not every session needs a fully specified custom environment. When you are in an early exploratory phase, either trying out a new library, sketching a new analysis approach, or running a quick sanity check, Renku's pre-built global environments give you a working Python, R, or Julia session in seconds, without configuring anything. Global environments come with a standard scientific stack pre-installed, and you can install additional packages on the fly with `pip install` or `install.packages()` for the duration of the session. + +When your prototype hardens into a real analysis, you lock the dependencies into a `requirements.txt` or `renv.lock`, commit them to your repository, and Renku can build a reproducible custom environment from that point forward with session launchers created from code. The transition from "exploratory sketch" to "locked reproducible pipeline" is a single git commit. + + +:::tip Remember + +Environment definitions live in your code repository alongside your code. That means your git history is also your environment history, roll back a broken dependency change the same way you roll back a buggy function. + +::: \ No newline at end of file diff --git a/docs/docs/10-users/80-use-cases/10-research/03-training-experimentation.md b/docs/docs/10-users/80-use-cases/10-research/03-training-experimentation.md new file mode 100644 index 0000000000..74f2aca0a6 --- /dev/null +++ b/docs/docs/10-users/80-use-cases/10-research/03-training-experimentation.md @@ -0,0 +1,5 @@ +--- +title: Training & Experimentation +--- + +# Run long experiments diff --git a/docs/docs/10-users/80-use-cases/10-research/04-showcasing-sharing.md b/docs/docs/10-users/80-use-cases/10-research/04-showcasing-sharing.md new file mode 100644 index 0000000000..fda230b682 --- /dev/null +++ b/docs/docs/10-users/80-use-cases/10-research/04-showcasing-sharing.md @@ -0,0 +1,5 @@ +--- +title: Showcasing & Sharing +--- + +# Share your results diff --git a/docs/docs/10-users/80-use-cases/10-research/05-publication-reproducibility.md b/docs/docs/10-users/80-use-cases/10-research/05-publication-reproducibility.md new file mode 100644 index 0000000000..c52003cb64 --- /dev/null +++ b/docs/docs/10-users/80-use-cases/10-research/05-publication-reproducibility.md @@ -0,0 +1,5 @@ +--- +title: Publication & Reproducibility +--- + +# A version your reviewers will actually be able to run diff --git a/docs/docs/10-users/80-use-cases/10-research/_category_.yml b/docs/docs/10-users/80-use-cases/10-research/_category_.yml new file mode 100644 index 0000000000..34c36842b6 --- /dev/null +++ b/docs/docs/10-users/80-use-cases/10-research/_category_.yml @@ -0,0 +1 @@ +label: Renku for Research diff --git a/docs/docs/10-users/80-use-cases/10-research/dataexploration-data-flow-diagram.png b/docs/docs/10-users/80-use-cases/10-research/dataexploration-data-flow-diagram.png new file mode 100644 index 0000000000..1f449d0a0a Binary files /dev/null and b/docs/docs/10-users/80-use-cases/10-research/dataexploration-data-flow-diagram.png differ diff --git a/docs/docs/10-users/80-use-cases/10-research/dataexploration-path-comparison.png b/docs/docs/10-users/80-use-cases/10-research/dataexploration-path-comparison.png new file mode 100644 index 0000000000..46140ce65a Binary files /dev/null and b/docs/docs/10-users/80-use-cases/10-research/dataexploration-path-comparison.png differ diff --git a/docs/docs/10-users/80-use-cases/10-research/dataexploration-project-setup.png b/docs/docs/10-users/80-use-cases/10-research/dataexploration-project-setup.png new file mode 100644 index 0000000000..3cf0f34c55 Binary files /dev/null and b/docs/docs/10-users/80-use-cases/10-research/dataexploration-project-setup.png differ diff --git a/docs/docs/10-users/80-use-cases/10-research/index.md b/docs/docs/10-users/80-use-cases/10-research/index.md new file mode 100644 index 0000000000..f8b0c4647f --- /dev/null +++ b/docs/docs/10-users/80-use-cases/10-research/index.md @@ -0,0 +1,42 @@ +--- +title: Renku for Research & Data Science +--- + +# Renku for Research & Data Science + +Whether you analyse field observations, build interactive dashboards, run large-scale simulations, or iterate on your model architecture, you usually spend too much valuable time on setting the infrastructure instead of on the work itself. + +Renku addresses the friction that cuts across all of these situations: data that takes hours to transfer before you can touch it, environments that work on one machine and nowhere else, long-running experiments that die when a laptop lid closes, and results that are difficult to share or reproduce later. Renku connects your data, your code and your environments into a single project, and keeps all three in sync for every collaborator and across every session. + +# Who this is for + +Renku's research and data science features are built for anyone who: + +- **Connects to large or remote datasets**: cloud storage (S3, Azure Blob), institutional mounts (SwitchDrive, SFTP), or archived datasets from Zenodo, Dataverse, or EnviDat. +- **Works across languages, such as Python, R or Julia** and needs a reproducible environment built automatically from a `requirements.txt`, `environment.yml`, or `renv.lock`, with no Dockerfile required. +- **Runs experiments or training jobs** that exceed what a laptop can handle or that need to run overnight, and be resumed from any machine. +- **Collaborates in a team or lab** sharing analysis workflows or interactive dashboards with colleagues, reviewers or the public without sending files and setup instructions via email. +- **Needs to cite or reproduce results** referencing an exact dataset version by DOI and an exact code state by git commit, for a paper submission or a technical report. + +If you work with data and need your analysis to be reproducible and shareable, Renku is designed for you. + +# What Renku gives you + +| Capability | What it means in practice | +| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Data connectors** | Mount S3, Azure Blob, SwitchDrive, SFTP, or DOI-based files directly into your session. No download, no hardcoded paths. | +| **Automatic environments** | Create a session launcher from your GitHub or GitLab repository. Renku builds the image from your dependency file and every collaborator gets the same environment. | +| **Persistent sessions** | Sessions run independently of your browser. Close your laptop, and come back tomorrow. Reconnect from any machine. | +| **Access to supercompute** | Select a GPU resource class, including CSCS Alps, from the same dropdown as a standard session. No job scripts, no queue systems. | +| **Shareable launch links** | Share a URL that opens your exact environment in a browser. | +| **DOI-based inputs and Zenodo export** | Reference the exact dataset your analysis used, and export your results to Zenodo for a citable DOI, all from the same project page. | + +# The research and data science lifecycle + +The guides below follow the natural lifecycle of data intensive projects, from first contact with a new dataset to a published, reproducible result. Each stage is independent, so feel free to jump to the one that matches your current challenge. + +```mdx-code-block +import DocCardList from '@theme/DocCardList'; + +✨ Understand Renku's core features for educational purposes
🧑🏼🏫 Create and manage teaching materials in Renku
diff --git a/docs/docs/10-users/80-use-cases/80_Demo-Workshop-Material.md b/docs/docs/10-users/80-use-cases/80-demo-workshop-material.md similarity index 100% rename from docs/docs/10-users/80-use-cases/80_Demo-Workshop-Material.md rename to docs/docs/10-users/80-use-cases/80-demo-workshop-material.md diff --git a/docs/docs/10-users/85-migrate-v1-v2/10-migrate-renku-1-project-to-renku-2.md b/docs/docs/10-users/85-migrate-v1-v2/10-migrate-renku-1-project-to-renku-2.md deleted file mode 100755 index 2589e9b11f..0000000000 --- a/docs/docs/10-users/85-migrate-v1-v2/10-migrate-renku-1-project-to-renku-2.md +++ /dev/null @@ -1,83 +0,0 @@ -# Migrate a Renku Legacy project to Renku 2.0 - -Time is running short! Renku Legacy was discontinued in October 2025, and you still have access to your Renkulab Gitlab projects until end of January 2026. Hence, move your code repositories as explained in our [detailed migration guides](https://blog.renkulab.io/sunsetting-legacy/#how-to-move-a-code-repository-to-an-external-provider) and in our dedicated videos. - -- -*Bring your RenkuLab GitLab project to the public GitLab instance with the import feature.* -
- -## 1. Move a code repository to an external provider - -1. **Clone your repository** from the RenkuLab GitLab to your local machine - - ```bash - git clone- -
- -3. For the container image, enter your **image identifier**. - - Some examples of image identifiers: - - `renku/renkulab-py:3.10-0.24.0` - - `continuumio/anaconda3:2024.06-1` - - The image identifier should be in the format that works with `docker pull` -4. Depending on the image you’re using, you’ll need to fill in the **Advanced settings**. See the information below for how to fill it in: - --  -
- - :::warning - - This part is important! Please read carefully. - - ::: - -- -
- -To determine what resources in your project need to be migrated, look for the **Migration needed** -badge on code repositories and session launchers in your project: - -- -
- -Then, follow the instructions below to migrate each resource. - -```mdx-code-block -import DocCardList from '@theme/DocCardList'; - -- -
- -```mdx-code-block -import DocCardList from '@theme/DocCardList'; - -