Merge branch 'develop' into events-schema-0.1.1

Author: infotroph

CHANGELOG.md

@@ -25,6 +25,9 @@ For more information about this file see also [Keep a Changelog](http://keepacha
 - Added `PEcAn.data.land::to_co2e()` for converting SOC change, CH4, and N2O to CO2-equivalent emissions using IPCC Global Warming Potential values.
 
 ### Fixed
+- Fixed broken pecanproject.github.io, pecan.gitbooks.io, and other outdated documentation links across book_source, tutorials, models, modules, web, and shiny files (#3710).
+- Added note to DEV-INTRO.md documenting Traefik workaround for Apple Silicon (ARM64) Macs: use `traefik:v2.11` with `platform: linux/arm64` to fix 404 errors (#3910)
+- Fixed `web/08-finished.php`: show database info instead of "Still running" when workflow folder doesn't exist locally (#3501).
 
 ### Changed
 - Updated Docker architecture documentation to match current docker-compose.yml: removed portainer/minio/thredds, added rstudio/api sections, updated service lists and volumes (#3268).
@@ -39,6 +42,7 @@ For more information about this file see also [Keep a Changelog](http://keepacha
   `MCMCpack`, `mvtnorm`, `neonUtilities`, `neonstore`, `PEcAn.benchmark`,
   `PEcAn.visualization`, `rjags`, `sirt`, and `sp` from `Imports` to
   `Suggests` (@omkarrr2533, #3599).
+- Management events specified via `events.json` are now required to specify a crop code for each planting event, so that models can know when to restart with a different PFT (#3828, #3836).
 
 
 

DEV-INTRO.md

@@ -65,6 +65,13 @@ cp docker-compose.dev.yml docker-compose.override.yml
 
 You can now use the command `docker compose` to work with the containers setup for development. **The rest of this document assumes you have done this step.**
 
+_Note for Apple Silicon (M1/M2/M3/M4) macOS users:_ The default Traefik image (`traefik:v2.9`) is built only for `linux/amd64` and cannot reliably communicate with the Docker socket when running under emulation on ARM64. This causes all web routes to return 404 errors. To fix this, add the following to your `docker-compose.override.yml` under `services:` and above `pecan:`:
+```sh
+  traefik:
+    image: traefik:v2.11
+    platform: linux/arm64
+```
+
 ### First time setup
 
 The steps in this section only need to be done the first time you start working with the stack in docker. After this is done you can skip these steps. You can find more detail about the docker commands in the [pecan documentation](https://pecanproject.github.io/pecan-documentation/latest/docker-index.html).
@@ -246,7 +253,7 @@ A better way of doing this is developed as part of GSOC, in which case you can l
 
 ## PEcAn URLs
 
-You can check the RabbitMQ server used by pecan using <https://rabbitmq.pecan.localhost> on the same server that the docker stack is running on. You can use rstudio either with <http://server/rstudio> or at <http://rstudio.pecan.localhost>. To check the traefik dashboard you can use <http://traefik.pecan.localhost>.
+You can check the RabbitMQ server used by pecan using <https://rabbitmq.pecan.localhost> on the same server that the docker stack is running on. You can use rstudio either with `http://server/rstudio` or at <http://rstudio.pecan.localhost>. To check the traefik dashboard you can use <http://traefik.pecan.localhost>.
 
 If the stack is running on a remote machine, you can use ssh and port forwarding to connect to the server. For example `ssh -L 8000:localhost:80` will allow you to use <http://rabbitmq.pecan.localhost:8000/> in your browser to connect to the remote PEcAn server RabbitMQ.
 

apps/api/R/posteriors.R

@@ -104,7 +104,7 @@ searchPosteriors <- function(req, pft_id = NA, host_id = NA,
       result$prev_page <- paste0(
         req$rook.url_scheme, "://",
         req$HTTP_HOST,
-        "/api/workflows",
+        "/api/posteriors",
         req$PATH_INFO,
         substr(req$QUERY_STRING,
                0,

apps/api/R/workflows.R

@@ -312,7 +312,9 @@ getWorkflowFilesAsZip <- function(req, id, filenames, res){
 
       full_files[i] <- filepath
     }
-    zip_file <- zip::zipr("output.zip", full_files)
-    return(zip_file)
+    zip_path <- zip::zipr("output.zip", full_files)
+    bin <- readBin(zip_path, "raw", n = file.info(zip_path)$size)
+    file.remove(zip_path)
+    return(bin)
   }
 }

base/db/R/dbfiles.R

@@ -766,7 +766,7 @@ dbfile.move <- function(old.dir, new.dir, file.type, siteid = NULL, register = F
   files.indb <- 0
 
   # check for file type and update to make it *.file type
-  if (file.type != "clim" | file.type != "nc") {
+  if (file.type != "clim" && file.type != "nc") {
     PEcAn.logger::logger.error("File type not supported by move at this time. Currently only supports NC and CLIM files")
     error <- 1
   }

base/db/R/get.trait.data.pft.R

@@ -1,15 +1,54 @@
 ##' Get trait data from the database for a single PFT
 ##'
-##' @details `pft` should be a list containing at least `name` and `outdir`, and optionally `posteriorid` and `constants`. BEWARE: All existing files in `outir` will be deleted!
-##' @param pft list of settings for the pft whose traits to retrieve. See details
-##' @param modeltype type of model that is used, this is used to distinguish between different pfts with the same name.
+##' @md
+##' Queries BETYdb for trait observations and prior distributions for a single
+##' plant functional type (PFT). Results are saved to files
+##' in the PFT output directory (`pft$outdir`), and also registered in the
+##' database as posterior records when `write = TRUE`.
+##'
+##' @details
+##' `pft` should be a list containing at least `name` and `outdir`, and
+##' optionally `posteriorid` and `constants`.
+##'
+##' **File-based side effects (saved to `pft$outdir`):**
+##' \describe{
+##'   \item{`trait.data.Rdata`}{Contains a single object `trait.data`: a named
+##'     list of data frames, one per trait. Each data frame has columns from
+##'     BETYdb's traits/yields views (e.g., `mean`, `stat`, `n`, `site_id`,
+##'     `treatment_id`). Names correspond to trait variable names
+##'     (e.g., `"SLA"`, `"Vcmax"`).}
+##'   \item{`prior.distns.Rdata`}{Contains a single object `prior.distns`: a
+##'     data frame with one row per trait and columns `distn`, `parama`,
+##'     `paramb`, and `n`. Row names are trait variable names. Traits listed
+##'     in `pft$constants` are excluded.}
+##'   \item{`trait.data.csv`}{CSV export of `trait.data` (all traits
+##'     row-bound).}
+##'   \item{`prior.distns.csv`}{CSV export of `prior.distns`.}
+##'   \item{`species.csv` or `cultivars.csv`}{PFT membership list used to
+##'     detect changes between runs.}
+##' }
+##'
+##' **Downstream contract:** The files `trait.data.Rdata` and
+##' `prior.distns.Rdata` are expected by \code{run.meta.analysis.pft}, which
+##' loads them from `pft$outdir`. This implicit file-based coupling means
+##' the two functions must agree on directory path and object names. A future
+##' refactoring goal is to pass these objects directly via function arguments
+##' instead.
+##'
+##' @param pft list of settings for the pft whose traits to retrieve. See details.
+##' @param modeltype type of model that is used, this is used to distinguish
+##'   between different pfts with the same name.
 ##' @param dbfiles location where previous results are found
 ##' @param dbcon database connection
-##' @param forceupdate set this to true to force an update, auto will check to see if an update is needed.
+##' @param forceupdate set this to true to force an update, auto will check to
+##'   see if an update is needed.
 ##' @param write (Logical) If `TRUE` updated posteriors will be written to
-##'   BETYdb.  Defaults to FALSE.
+##'   BETYdb.  Defaults to `FALSE`.
 ##' @param trait.names list of trait names to retrieve
-##' @return updated pft with posteriorid
+##' @return The `pft` input list, updated with `pft$posteriorid` set to the
+##'   ID of the (possibly new) posterior record in BETYdb. The posterior ID can
+##'   be used to locate the output files (`trait.data.Rdata`, `prior.distns.Rdata`,
+##'   etc.) via BETYdb's `dbfiles` table.
 ##' @author David LeBauer, Shawn Serbin, Rob Kooper
 ##' @export
 get.trait.data.pft <-
@@ -27,10 +66,6 @@ get.trait.data.pft <-
     PEcAn.logger::logger.error(paste0("Couldn't create PFT output directory: ", pft$outdir))
   }
 
-  ## Remove old files.  Clean up.
-  old.files <- list.files(path = pft$outdir, full.names = TRUE, include.dirs = FALSE)
-  file.remove(old.files)
-
   # find appropriate pft
   pftres <- query_pfts(dbcon, pft[["name"]], modeltype)
   pfttype <- pftres[["pft_type"]]

base/db/man/get.trait.data.pft.Rd

@@ -10,9 +10,9 @@ check_model_run <- function(out, stop.on.error = TRUE) {
     success <- FALSE
     msg <- paste0("Model run aborted with the following error:\n", out)
     if (stop.on.error) {
-      PEcAn.logger::logger.severe(msg)
+      PEcAn.logger::logger.severe(msg, wrap = FALSE)
     } else {
-      PEcAn.logger::logger.error(msg)
+      PEcAn.logger::logger.error(msg, wrap = FALSE)
     }
   } else {
     success <- TRUE

base/remote/R/check_model_run.R

@@ -2,7 +2,7 @@
 #' Standardized variable names and units for PEcAn
 #'
 #' A lookup table giving standard names, units and descriptions for variables in PEcAn input/output files.
-#' Originally based on the \href{https://nacp.ornl.gov/MsTMIP_variables.shtml}{MsTMIP} standards,
+#' Originally based on the \href{https://nacp-prod.ornl.gov/MsTMIP_variables.shtml}{MsTMIP} standards,
 #' with additions to accomodate a wider range of model inputs and outputs.
 #' The standard_vars table replaces both `mstmip_vars` and `mstmip_local`,
 #' both of which are now deprecated.
@@ -13,7 +13,7 @@
 #' @format data frame, all columns character
 #' \describe{
 #'  \item{Variable.Name}{Short name suitable for programming with}
-#'  \item{standard_name}{Name used in the NetCDF \href{http://cfconventions.org/Data/cf-standard-names/current/build/cf-standard-name-table.html}{CF metadata conventions} }
+#'  \item{standard_name}{Name used in the NetCDF \href{https://cf-convention.github.io/Data/cf-standard-names/current/build/cf-standard-name-table.html}{CF metadata conventions} }
 #'  \item{Units}{Standard units for this variable. Do not call variables by these names if they are in different units.
 #'    See `ud_convert` for conversions to and from non-standard units}
 #'  \item{Long.Name}{Human-readable variable name, suitable for e.g. axis labels}

base/utils/R/datasets.R

@@ -11,7 +11,7 @@
 #' surface runoff (Qs), subsurface runoff (Qsb), and rainfall (Rainf).
 #'
 #' For more details, see the [MsTMIP
-#' variables](http://nacp.ornl.gov/MsTMIP_variables.shtml)
+#' variables](https://nacp-prod.ornl.gov/MsTMIP_variables.shtml)
 #' documentation.
 #'
 #' @param runid the ID distinguishing the model run. Can be omitted