Sync master to feature branch#6984
Merged
liulinC merged 144 commits intofeature/ldapsfrom Mar 27, 2026
Merged
Conversation
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Add initial version
We never 'throw' exceptions between threads so we can use thread-local backtrace tables to avoid too much contention / having to resize the global table. Users must wrap all their threads in a Backtrace.with_backtraces (fun () -> ... ) Failure to do so will generate a backtrace with an error in it (therefore it should be obvious that this wasn't done) Signed-off-by: David Scott <dave.scott@eu.citrix.com>
We use Hashtbl.add and then Hashtbl.remove, but always adding a reference to the same record. We use the size of the bindings list in the hashtable as the reference count. Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Use thread-local backtrace tables
We need a way to construct backtraces from data sent from other languages. Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Add support for Interoperating with other languages
We take the hit of parsing the OCaml < 4.02 stacktraces safe in the knowledge that we can optimise this away (with optcomp) later. This also means we take full control over the rendering of stacktraces and can make the python and OCaml backtraces look the same. Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Store backtraces as lists of records rather than strings
All backtraces have been off by one... 0/9 xapi @ renoir Raised at file "db_rpc_client_v1.ml", line 33, characters 14-39 ... 8/9 xapi @ renoir Called from file "lib/backtrace.ml", line 150, characters 17-21 Signed-off-by: Si Beaumont <simon.beaumont@citrix.com>
Correct backtrace ordering by indexing from 1
Signed-off-by: Si Beaumont <simon.beaumont@citrix.com>
Signed-off-by: Si Beaumont <simon.beaumont@citrix.com>
Signed-off-by: Si Beaumont <simon.beaumont@citrix.com>
Enable Travis using ocaml-travisci-skeleton
Signed-off-by: Si Beaumont <simon.beaumont@citrix.com>
Signed-off-by: Si Beaumont <simon.beaumont@citrix.com>
Signed-off-by: David Scott <dave@recoil.org>
Signed-off-by: David Scott <dave@recoil.org>
Release 0.3
Signed-off-by: Marcello Seri <marcello.seri@citrix.com>
Merge pp-ely into master for moving to OCaml 4.02.3 and PPX
Run Travis build on all branches, but only upload docs on the master branch. Signed-off-by: Gabor Igloi <gabor.igloi@citrix.com>
Fix travis build; update opam file
Signed-off-by: Gabor Igloi <gabor.igloi@citrix.com>
Removing this device class from the USB config would let admins configure VMs for digital signing purposes. These devices are also not very useful in Dom0. Signed-off-by: Tu Dinh <ngoc-tu.dinh@vates.tech>
Quicktest was listing potentially thousands of VDIs and querying its properties, just to find out whether the resulting list was empty or not. We don't need to produce the full list to check whether it is empty or not: we can stop once we have the 1st element. Replace the List with a Seq. Before: ``` time ./quicktest.exe -run-only Quicktest_vm_lifecycle [...] real 0m13.957 ``` After: ``` real 0m12.584s ``` Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Some (read-only) SRs can contain a large number of VDIs, but these cannot "leak" VDIs if we cannot create or destroy them. Before: ``` real 0m12.584s ``` After: ``` real 0m7.853s ``` Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Before: ``` real 0m7.853s ``` After: ``` real 0m7.746s ``` Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Before: ``` real 0m7.746s ``` After: ``` real 0m6.095s ``` Signed-off-by: Edwin Török <edwin.torok@citrix.com>
The difference is not really measurable. But I found out there is another pre-existing flag that can speed it up, `-default-sr`. Then the leak checks are only done on that SR, and not all the SRs. So the best way is to run this (at least during development): ``` time ./quicktest.exe -run-only Quicktest_vm_lifecycle -skip-stress -default-sr [...] real 0m1.709s ``` Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Removing this device class from the USB config would let admins configure VMs for digital signing purposes. These devices are also not very useful in Dom0.
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Instead of parsing the string backtrace, use the record that is available since OCaml 4.02.
This is more future-proof, and performance is also improved ~10x:
Before:
```
Backtrace.is_important (ns):
{ monotonic-clock per run = 2674.017426 (confidence: 2685.374519 to 2661.996895);
r² = Some 0.999347 }
```
After:
```
Backtrace.is_important (ns):
{ monotonic-clock per run = 263.881150 (confidence: 264.748572 to 263.176838);
r² = Some 0.999726 }
```
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
With a long recursive call there could be thousands of these.
Show just one in the logs.
There is a small <5% performance impact:
Before:
```
Backtrace.is_important (ns):
{ monotonic-clock per run = 263.881150 (confidence: 264.748572 to 263.176838);
r² = Some 0.999726 }
```
After:
```
Backtrace.is_important (ns):
{ monotonic-clock per run = 275.430030 (confidence: 275.818027 to 275.087604);
r² = Some 0.999938 }
```
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
forkexecd, resources and xapi-stdext-pervasives wasn't using it. Signed-off-by: Edwin Török <edwin.torok@citrix.com>
…se_with_backtrace Signed-off-by: Edwin Török <edwin.torok@citrix.com>
…ndency The backtrace wasn't retrieved before calling the logging function, so it was potentially lost. Use Printexc module instead. Signed-off-by: Edwin Török <edwin.torok@citrix.com>
This is long overdue, but we finally got some QA resource to complete the test plan, and no major bugs were found. There is one improvement that could be made: when deleting a user/group (or otherwise changing permissions) on the XAPI side we could immediately expire any caches related to that user.
OCaml 4.02 added column location, and 4.11 added function names.
Drop any wrapper modules by dropping anything before the last `__` in
the string, if any.
For backward compatibility with python interop exceptions these new fields are optional.
Backtrace.t_of_sexp is only called in `xapi_xenopsd.ml`, and we require updating XAPI and xenopsd together already.
For everything else (the Task API) we'll accept backtraces without the additional info, but will start producing backtraces
with the additional info.
<10% performance impact:
Before:
```
Backtrace.is_important (ns):
{ monotonic-clock per run = 275.430030 (confidence: 275.818027 to 275.087604);
r² = Some 0.999938 }
```
After:
```
Backtrace.is_important (ns):
{ monotonic-clock per run = 292.372339 (confidence: 293.039727 to 291.671323);
r² = Some 0.999771 }
```
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Ensures that when make_rpc switches protocols between XMLRPC and JSONRPC, then timeboxed_rpc follows. Have to be careful to pass the correct values for subtask_of (outer task), and task_id (inner task). Signed-off-by: Edwin Török <edwin.torok@citrix.com>
When using `opam` to install all packages from XAPI some libraries failed to build because the internal `backtrace` library was only available in the `xapi-log` package. The build worked when using `dune` directly (or `koji`), because they build the entire repository instead of individual packages. To avoid conflicts with the `xapi-backtrace` installed from `xs-opam` make the backtrace library properly part of `xapi-log`, by exposing it as `xapi-log.backtrace`. Put the cram tests into the xapi-log.backtrace package too. Add rpclib as a dependency for xapi-log, it was a dependency of xapi-backtrace. Fixes: e4da294 ("[maintenance]: use internal backtrace library") Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Otherwise it'd try to build it as part of another package and fail because `rpclib.core` cannot be found. Fixes: 1316037 ("CP-311150: test code for new library") Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Fixes a build failure caused by #6957. The build worked with `dune` and with `koji`, but not when using `opam` to install all packages.
I intended to write some new VM tests, and the existing VM lifecycle tests looked like a good starting point. But it turns out that we stopped running this test completely when we dropped the `CoreOS` template. Use a memtest ISO instead of the CoreOS template. Also optimize various startup and shutdown code in quicktest, because even a no-op test that did nothing took 14s to run. Draft PR, because it needs more testing, and I'm not entirely happy with how the SR filtering works, e.g. this test still won't run if `-default-sr` is passed (and passing `-default-sr` is very useful during development because it further cuts down startup time by 5s). I should then be able to reuse this code in the new tests that I'm writing.
There is a lot of duplicated code, make most of them use a choose_rpc function. This'll enable us to choose between JSONRPC/XMLRPC globally. There are some that are left as XMLRPC for now: * system-domains (driver domains): we don't currently have any * vpx: don't know if this'd support JSONRPC * Storage_utils.rpc: AFAICT this is XMLRPC only, we should really choose based on Content-Type, not path, but for now keep this unchanged Signed-off-by: Edwin Török <edwin.torok@citrix.com>
Newer OCaml 4.x versions provide more details about a backtrace:
* the location within a line (column), this is useful when you have a
long `|>` pipeline for example
* the name of the function (useful for a quick glance at what could be
wrong)
Previously `xapi-backtrace` used to parse the string formatted
exceptions, and tried to extract file and line number information from
it. Use the newer functions in `Printexc` to do this instead.
String formatting needs to remain in the our `backtrace` module, because
we want to consistently format backtraces from other languages too.
This is an example of how the backtrace would look like after these
changes:
```
Raised Db_exn.DBCache_NotFound("missing table", "Observer", "")
1/4 /opt/xensource/bin/xapi @ xrtmia-13-01 Raised at Xapi_database__Db_cache_types.TableSet.find in file "ocaml/database/db_cache_types.ml", line 298, characters 22-73
2/4 /opt/xensource/bin/xapi @ xrtmia-13-01 Called from Xapi_database__Db_cache_impl.read_refs in file "ocaml/database/db_cache_impl.ml", line 314, characters 12-70
3/4 /opt/xensource/bin/xapi @ xrtmia-13-01 Called from Db_actions.DB_Action.Observer.get_all in file "ocaml/xapi/db_actions.ml", line 23053, characters 40-69
4/4 /opt/xensource/bin/xapi @ xrtmia-13-01 Called from Xapi_observer_components.is_component_enabled.(fun) in file "ocaml/xapi/xapi_observer_components.ml", line 83, characters 26-56
```
Also prevent long `List.map` sequences in a backtrace, by deduplicating
adjacent identical entries in a backtrace.
These changes also speed up calls to `Backtrace.is_important` ~10x,
although performance isn't the main goal here (raising exceptions
shouldn't be on the common path)
…lls in XAPI (#6979) There were a few more places that used XMLRPC that were missed the first time. In particular intra-pool communication and XAPI<->xenopsd communication. The feature flag will eventually be enabled (on `feature/perf-edvint` first), but the feature flagged code can be merged to master directly.
BengangY
approved these changes
Mar 27, 2026
changlei-li
approved these changes
Mar 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.