Add higher-order functions changes to upgrade guide (#22107)

## Which issue does this PR close?

Follow up of #21679 

## Rationale for this change

<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->

Author: gstvg

docs/source/library-user-guide/upgrading/54.0.0.md

@@ -751,10 +751,10 @@ SET datafusion.runtime.file_statistics_cache_limit = '0'
 
 Use the file statistics cache provided by the `CacheManager` when initializing a new `ListingTable`:
 
-````rust,ignore
+```rust,ignore
 ListingTable::try_new(config)?
   .with_cache(ctx.runtime_env().cache_manager.get_file_statistic_cache())
-[#21075]: https://github.com/apache/datafusion/pull/21075
+```
 
 ### `UnionsToFilter` optimizer rule is now disabled by default
 
@@ -771,7 +771,7 @@ SELECT * FROM t WHERE a = 2
 
 -- After: one scan
 SELECT DISTINCT * FROM t WHERE a = 1 OR a = 2
-````
+```
 
 **Who is affected:**
 
@@ -788,7 +788,158 @@ a combined `OR` (e.g., index-backed sources).
 SET datafusion.optimizer.enable_unions_to_filter = true;
 ```
 
-### The `RegisterFunction` enum has a new `HigherOrder` variant
+See [PR #21075](https://github.com/apache/datafusion/pull/21075) for more details
+
+### Higher-order functions and lambdas
+
+The changes below are related to the added support for higher-order functions
+and lambdas. For more info, see [#14205], [PR #18921], [PR #21679] and
+[EPIC #21172].
+
+[#14205]: https://github.com/apache/datafusion/issues/14205
+[pr #18921]: https://github.com/apache/datafusion/pull/18921
+[pr #21679]: https://github.com/apache/datafusion/pull/21679
+[epic #21172]: https://github.com/apache/datafusion/issues/21172
+
+#### `FunctionRegistry` exposes two additional methods
+
+`FunctionRegistry` exposes two additional methods, `higher_order_function`
+which returns the registered higher-order function with the given name, if
+any, and `higher_order_function_names` which exposes the set of registered
+user defined higher-order function names.
+
+**Who is affected:**
+
+- Users who implement the `FunctionRegistry` trait
+
+**Migration guide:**
+
+Add `higher_order_function` and `higher_order_function_names` to your implementation.
+
+```diff
+impl FunctionRegistry for FunctionRegistryImpl {
+      fn udfs(&self) -> HashSet<String> {
+         self.scalar_functions.keys().cloned().collect()
+     }
++
++    fn higher_order_function(&self, name: &str) -> Result<Arc<dyn HigherOrderUDF>> {
++        self.higher_order_functions
++            .get(name)
++            .cloned()
++            .ok_or_else(|| plan_datafusion_err!("Higher-order function {name} not found"))
++    }
++
++    fn higher_order_function_names(&self) -> HashSet<String> {
++        self.higher_order_functions.keys().cloned().collect()
++    }
+}
+```
+
+#### `ContextProvider` exposes two additional methods
+
+`ContextProvider` exposes two additional methods, `get_higher_order_meta`
+which returns the registered higher-order function with the given name,
+if any, and `higher_order_function_names` which exposes the registered
+user defined higher-order function names.
+
+**Who is affected:**
+
+- Users who implement the `ContextProvider` trait
+
+**Migration guide:**
+
+Add `get_higher_order_meta` and `higher_order_function_names` to your implementation.
+
+```diff
+impl ContextProvider for ContextProviderImpl {
+      fn udfs(&self) -> HashSet<String> {
+         self.scalar_functions.keys().cloned().collect()
+     }
++
++    fn get_higher_order_meta(&self, name: &str) -> Option<Arc<dyn HigherOrderUDF>> {
++        self.higher_order_functions.get(name).cloned()
++    }
++
++    fn higher_order_function_names(&self) -> Vec<String> {
++        self.higher_order_functions.keys().cloned().collect()
++    }
+}
+```
+
+#### Add `higher_order_functions()` method to `Session`
+
+The `higher_order_functions` method has been added to the `Session` trait,
+which exposes the registered user defined higher-order functions.
+
+**Who is affected:**
+
+- Users who implement the `Session` trait
+
+**Migration guide:**
+
+Add `higher_order_functions` to your implementation.
+
+```diff
+impl Session for MySession {
+     ...
++    fn higher_order_functions(&self) -> &HashMap<String, Arc<dyn HigherOrderUDF>> {
++        &self.higher_order_functions
++    }
+}
+```
+
+#### New argument on `TaskContext::new`
+
+`TaskContext::new` expects a new argument, `higher_order_functions`, which is
+a map of higher-order functions keyed by name.
+
+**Who is affected:**
+
+- Users who call `TaskContext::new`
+
+**Migration guide:**
+
+Provide the new argument to the function. An empty hash map is sufficient.
+
+```diff
++let higher_order_functions = HashMap::new();
+
+TaskContext::new(
+     task_id,
+     session_id,
+     session_config,
+     scalar_functions,
++    higher_order_functions,
+     aggregate_functions,
+     window_functions,
+     runtime,
+)
+```
+
+#### The `Expr` enum has three new variants
+
+- `HigherOrderFunction`: Call a higher-order function with a set of arguments
+- `Lambda`: A Lambda expression with a set of parameters names and a body
+- `LambdaVariable`: A named reference to a lambda parameter
+
+**Who is affected:**
+
+- Users who match on an `Expr` without a default branch `_ => {}`
+
+**Migration guide:**
+
+Add the new branches to the match with the logic applicable to the context.
+
+```diff
+match expr {
+    Expr::Column(column) => ...,
+    ...,
++   Expr::HigherOrderFunction(func) => {},
++   Expr::Lambda(lambda) => {},
++   Expr::LambdaVariable(lambda_var) => {},
+```
+
+#### The `RegisterFunction` enum has a new `HigherOrder` variant
 
 `RegisterFunction` now has a `HigherOrder(Arc<dyn HigherOrderUDF>)` variant
 so user-defined higher-order functions can be registered