Merge pull request #79 from ReactiveBayes/fix-invertible-nn-optim-v2

Fix Invertible Neural Network Tutorial for Optim.jl v2

Author: bvdmitri

examples/Problem Specific/Invertible Neural Network Tutorial/Invertible Neural Network Tutorial.ipynb

@@ -43,20 +43,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": [
-    "using RxInfer\n",
-    "using Random\n",
-    "using StableRNGs\n",
-    "\n",
-    "using ReactiveMP        # ReactiveMP is included in RxInfer, but we explicitly use some of its functionality\n",
-    "using LinearAlgebra     # only used for some matrix specifics\n",
-    "using Plots             # only used for visualisation\n",
-    "using Distributions     # only used for sampling from multivariate distributions\n",
-    "using Optim             # only used for parameter optimisation"
-   ]
+   "source": "using RxInfer\nusing Random\nusing StableRNGs\n\nusing ReactiveMP        # ReactiveMP is included in RxInfer, but we explicitly use some of its functionality\nusing LinearAlgebra     # only used for some matrix specifics\nusing Plots             # only used for visualisation\nusing Distributions     # only used for sampling from multivariate distributions\nusing Optim             # only used for parameter optimisation\nusing ADTypes           # only used to specify the automatic differentiation backend for Optim\nusing ForwardDiff       # only used for automatic differentiation in the parameter optimisation"
   },
   {
    "attachments": {},
@@ -9351,93 +9341,14 @@
    "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
-   "source": [
-    "Optimization can be performed using the `Optim` package. Alternatively, other (custom) optimizers can be implemented, such as:\n",
-    "\n",
-    "```julia\n",
-    "res = optimize(f, randn(StableRNG(42), nr_params(model)), GradientDescent(), Optim.Options(store_trace = true, show_trace = true, show_every = 50), autodiff=:forward)\n",
-    "``` \n",
-    "\n",
-    "- uses finitediff and is slower/less accurate.\n",
-    "\n",
-    "*or*\n",
-    "\n",
-    "```julia\n",
-    "# create gradient function\n",
-    "g = (x) -> ForwardDiff.gradient(f, x);\n",
-    "\n",
-    "# specify initial params\n",
-    "params = randn(nr_params(model))\n",
-    "\n",
-    "# create custom optimizer (here Adam)\n",
-    "optimizer = Adam(params; λ=1e-1)\n",
-    "\n",
-    "# allocate space for gradient\n",
-    "∇ = zeros(nr_params(model))\n",
-    "\n",
-    "# perform optimization\n",
-    "for it = 1:10000\n",
-    "\n",
-    "    # backward pass\n",
-    "    ∇ .= ForwardDiff.gradient(f, optimizer.x)\n",
-    "\n",
-    "    # gradient update\n",
-    "    ReactiveMP.update!(optimizer, ∇)\n",
-    "\n",
-    "end\n",
-    "\n",
-    "```"
-   ]
+   "source": "Optimization can be performed using the `Optim` package. Since Optim v2 the automatic differentiation backend is specified with an `ADTypes` object (e.g. `AutoForwardDiff()`) instead of a symbol. Alternatively, other (custom) optimizers can be implemented, such as:\n\n```julia\nres = optimize(f, randn(StableRNG(42), nr_params(model)), GradientDescent(), Optim.Options(store_trace = true, show_trace = true, show_every = 50))\n``` \n\n- uses finitediff and is slower/less accurate.\n\n*or*\n\n```julia\n# create gradient function\ng = (x) -> ForwardDiff.gradient(f, x);\n\n# specify initial params\nparams = randn(nr_params(model))\n\n# create custom optimizer (here Adam)\noptimizer = Adam(params; λ=1e-1)\n\n# allocate space for gradient\n∇ = zeros(nr_params(model))\n\n# perform optimization\nfor it = 1:10000\n\n    # backward pass\n    ∇ .= ForwardDiff.gradient(f, optimizer.x)\n\n    # gradient update\n    ReactiveMP.update!(optimizer, ∇)\n\nend\n\n```"
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Iter     Function value   Gradient norm \n",
-      "     0     5.888958e+02     8.943663e+02\n",
-      " * time: 0.02565789222717285\n",
-      "   100     1.059823e+01     4.118858e+00\n",
-      " * time: 6.649883985519409\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       " * Status: success\n",
-       "\n",
-       " * Candidate solution\n",
-       "    Final objective value:     9.904775e+00\n",
-       "\n",
-       " * Found with\n",
-       "    Algorithm:     Gradient Descent\n",
-       "\n",
-       " * Convergence measures\n",
-       "    |x - x'|               = 1.22e-03 ≰ 0.0e+00\n",
-       "    |x - x'|/|x'|          = 5.79e-04 ≰ 0.0e+00\n",
-       "    |f(x) - f(x')|         = 9.55e-03 ≰ 0.0e+00\n",
-       "    |f(x) - f(x')|/|f(x')| = 9.65e-04 ≤ 1.0e-03\n",
-       "    |g(x)|                 = 2.21e+00 ≰ 1.0e-08\n",
-       "\n",
-       " * Work counters\n",
-       "    Seconds run:   8  (vs limit Inf)\n",
-       "    Iterations:    116\n",
-       "    f(x) calls:    312\n",
-       "    ∇f(x) calls:   312\n"
-      ]
-     },
-     "execution_count": 21,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "res = optimize(f, randn(StableRNG(42), nr_params(model)), GradientDescent(), Optim.Options(f_tol = 1e-3, store_trace = true, show_trace = true, show_every = 100), autodiff=:forward)"
-   ]
+   "outputs": [],
+   "source": "res = optimize(f, randn(StableRNG(42), nr_params(model)), GradientDescent(), Optim.Options(f_reltol = 1e-3, store_trace = true, show_trace = true, show_every = 100), autodiff=AutoForwardDiff())"
   },
   {
    "attachments": {},

examples/Problem Specific/Invertible Neural Network Tutorial/Project.toml

@@ -1,5 +1,7 @@
 [deps]
+ADTypes = "47edcb42-4c32-4615-8424-f2b9edc5f35b"
 Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
+ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
 LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
 Optim = "429524aa-4258-5aef-a3af-852621145aeb"
 Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"