Merge pull request #74 from gchq/release/1.1.0

Release 1.1.0

Adds Batch Baking endpoint to more efficiently process large amounts of data in a single request

Author: GCHQDeveloper1138

CHANGELOG.md

@@ -13,7 +13,12 @@ All major and minor version changes will be documented in this file. Details of
 
 ## Details
 
-### [1.0.0-beta] - 2024-10-04
+### [1.1.0] - 2025-02-26
+- Added Batch Baking functionality
+- Fixed some typos in Readme
+- Improved Swagger spec
+
+### [1.0.0] - 2024-10-04
 - CyberChef functionality brought in line with the mainline release
 - Upgraded to use ESM modules throughout
 - Works with recent versions of NodeJS

Dockerfile

@@ -1,8 +1,8 @@
 FROM node:18.20.4-alpine
-LABEL author "Wes Lambert, wlambertts@gmail.com"
+LABEL author="Wes Lambert, wlambertts@gmail.com"
 LABEL description="Dockerised version of Cyberchef server (https://github.com/gchq/CyberChef-server)"
-LABEL copyright "Crown Copyright 2020"
-LABEL license "Apache-2.0"
+LABEL copyright="Crown Copyright 2020"
+LABEL license="Apache-2.0"
 COPY . /CyberChef-server
 WORKDIR /CyberChef-server
 RUN npm cache clean --force && \

README.md

@@ -30,7 +30,7 @@ After using [CyberChef](https://gchq.github.io/CyberChef/) to experiment and fin
 
 
 ## Installing
-While the instructions below are straightforward, it's worth understanding what happens under the hood as quirks in the installation process can cause issues with upgrades etc. The upstream CyberChef library uses a postinstall script to manipulate some dependenciesafter installation. That postinstal script doesn't work when install CC as a dependency into another project, so we now have our own postinstall process to install the CyberChef library (hence it does not appear in the package.json dependency list).
+While the instructions below are straightforward, it's worth understanding what happens under the hood as quirks in the installation process can cause issues with upgrades etc. The upstream CyberChef library uses a postinstall script to manipulate some dependencies after installation. That postinstall script doesn't work when installing CC as a dependency into another project, so we now have our own postinstall process to install the CyberChef library (hence it does not appear in the package.json dependency list).
 
 This process can cause problems when updating libs, so the recommended approach is to remove node_modules and package-lock.json and install from scratch.
 
@@ -170,6 +170,92 @@ Response:
 }
 ```
 
+### `/batch/bake`
+
+`/batch/bake` allows a user to POST multiple input values and a configuration for a CyberChef Recipe. The application will run each elemnt of the input through the recipe and return the results as an array of output objects.
+
+This endpoint accepts a POST request with the following body:
+
+|Parameter|Type|Description|
+|---|---|---|
+input|Array|The input data for the recipe. Currently accepts an array of Strings.
+recipe|String or Object or Array|One or more operations, with optional arguments. Uses default arguments if they're not defined here.
+outputType (optional)|String|The [Data Type](https://github.com/gchq/CyberChef/wiki/Adding-a-new-operation#data-types) that you would like the result of the bakes to be returned as. This will not work with `File` or `List<File>` at the moment.
+
+#### Example: one operation, default arguments
+```javascript
+{
+    "input": ["One", "two", "three", "four"],
+    "recipe": "to decimal"
+}
+```
+
+Response:
+```javascript
+[
+  {
+    "success": true,
+    "value": "79 110 101",
+    "type": "string"
+  },
+  {
+    "success": true,
+    "value": "116 119 111",
+    "type": "string"
+  },
+  {
+    "success": true,
+    "value": "116 104 114 101 101",
+    "type": "string"
+  },
+  {
+    "success": true,
+    "value": "102 111 117 114",
+    "type": "string"
+  }
+]
+
+```
+> For more information on how operation names are handled, see the [Node API docs](https://github.com/gchq/CyberChef/wiki/Node-API#operation-names)
+
+#### Example: one operation, non-default arguments by name
+```javascript
+{
+    "input": ["One", "two", "three", "four"],
+    "recipe": {
+        "op": "to decimal",
+        "args": {
+            "delimiter": "Colon"
+        }
+    }
+}
+```
+Response:
+```javascript
+[
+  {
+    "success": true,
+    "value": "79:110:101",
+    "type": "string"
+  },
+  {
+    "success": true,
+    "value": "116:119:111",
+    "type": "string"
+  },
+  {
+    "success": true,
+    "value": "116:104:114:101:101",
+    "type": "string"
+  },
+  {
+    "success": true,
+    "value": "102:111:117:114",
+    "type": "string"
+  }
+]
+```
+
 ### `/magic`
 
 [Find more information about what the Magic operation does here](https://github.com/gchq/CyberChef/wiki/Automatic-detection-of-encoded-data-using-CyberChef-Magic)

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "cyberchef-server",
-  "version": "1.0.0-beta",
+  "version": "1.1.0",
   "description": "An application providing API access to CyberChef",
   "author": "d98762625 <d98762625@gmail.com>",
   "license": "Apache-2.0",

package.json

@@ -13,6 +13,9 @@ import helmet from "helmet";
 import bakeRouter from "./routes/bake.mjs";
 import magicRouter from "./routes/magic.mjs";
 import healthRouter from "./routes/health.mjs";
+import batchBakeRouter from "./routes/batchBake.mjs";
+
+const isProduction = process.env.NODE_ENV === "production";
 
 const app = express();
 app.disable("x-powered-by");
@@ -22,7 +25,7 @@ app.use(cors({
     origin: "*"
 }));
 
-if (process.env.NODE_ENV === "production") {
+if (isProduction) {
     app.use(helmet({ crossOriginResourcePolicy: { policy: "cross-origin" } }));
     app.use(logger({
         level: "error",
@@ -45,6 +48,8 @@ const swaggerFile = fs.readFileSync("./swagger.yml", "utf8");
 app.use("/health", healthRouter);
 app.use("/bake", bakeRouter);
 app.use("/magic", magicRouter);
+// Batch routes
+app.use("/batch/bake", batchBakeRouter);
 
 
 // Default route

src/app.mjs

@@ -24,6 +24,7 @@ export default function errorHandler(err, req, res, next) {
     ) {
         res.status(400).send(err.message).end();
     } else {
-        res.status(500).send(err.stack).end();
+        req.log.error(err.stack);
+        res.status(500).send("Internal Server Error").end();
     }
 }

src/lib/errorHandler.mjs

@@ -7,12 +7,9 @@ import { bake, Dish } from "cyberchef";
  */
 router.post("/", async function bakePost(req, res, next) {
     try {
-        if (!req.body.input) {
-            throw new TypeError("'input' property is required in request body");
-        }
-
-        if (!req.body.recipe) {
-            throw new TypeError("'recipe' property is required in request body");
+        const noRecipeOrInput = !req.body.input || !req.body.recipe;
+        if (noRecipeOrInput) {
+            throw new TypeError(`'${!req.body.input ? "input" : "recipe"}' property is required in request body`);
         }
 
         const dish = await bake(req.body.input, req.body.recipe);

src/routes/bake.mjs

@@ -0,0 +1,61 @@
+import { Router } from "express";
+const router = Router();
+import { bake, Dish } from "cyberchef";
+
+/**
+ * batchBakePost
+ */
+router.post("/", async function batchBakePost(req, res, next) {
+    try {
+        if (!req.body.input || !Array.isArray(req.body.input)) {
+            throw new TypeError("'input' property of type 'Array' is required in request body");
+        }
+
+        if (req.body.input.length === 0) {
+            throw new TypeError("'input' array must be non-empty");
+        }
+
+        if (!req.body.recipe) {
+            throw new TypeError("'recipe' property is required in request body");
+        }
+
+        // Results are objects with a result param if the result of the bake on the data
+        // item was successful and and error param if not that contains the string exception
+        // Result also contains the type, in line with the single bake endpoint
+        const retArr = req.body.input.map((input) => {
+            try {
+                const dish = bake(input, req.body.recipe);
+                let retVal = dish.value;
+                // Attempt to translate to another type. Any translation errors
+                // propagate through to the errorHandler.
+                if (req.body.outputType) {
+                    retVal = dish.get(req.body.outputType);
+                }
+                return {
+                    success: true,
+                    value: retVal,
+                    type: Dish.enumLookup(dish.type),
+                };
+            } catch (err) {
+                // Chef uses TypeError to indicate a problem with recipes. Safe to assume that a bad recipe
+                // will be a showstopper for all data so use the global error handler to handle this.
+                if (err instanceof TypeError) {
+                    req.log.error("Bad recipe");
+                    // Rethrow to activate the outer try/catch and exit the handling of this request
+                    throw err;
+                }
+                return {
+                    success: false,
+                    error: err.message,
+                };
+            }
+        });
+
+        res.send(retArr);
+
+    } catch (e) {
+        next(e);
+    }
+});
+
+export default router;