"rsync error 23" (Partial transfer due to errors)

[aryoda]

This issue is meant to centralize all "error 23" issues to discuss a common solution...

Background

BiT Version 1.4.0 (2023-09-14) introduced the evaluation of rsync exit codes for better error recognition:

backintime/CHANGES

Lines 32 to 42 in 726bffa

	* Fix bug: takeSnapshot() recognizes errors now by also evaluating the rsync exit code (#489)
	Fixes related problem: Killing `rsync` was not handled gracefully (by ignoring the rsync exit code)
	* Fix bug: The error user-callback is now always called if an error happened while taking a snapshot (#1491)
	* Fix bug: D-Bus serviceHelper error "LimitExceeded: Maximum length of command line reached (100)":
	Max command length is now 120 instead of 100 (#1027)
	* Feature: Introduce new error codes for the "error" user callback (as part of #1491):
	5: Error while taking a snapshot.
	6: New snapshot taken but with errors.
	* Feature: The `rsync` exit code is now contained in the snapshot log (part of #489). Example:
	[E] Error: 'rsync' ended with exit code -9 (negative values are signal numbers, see 'kill -l')
	* Fix bug: Treat rsync exit code 24 as INFO instead of ERROR (#1506)

Before this change rsync errors reported only via exit code were hidden and left users with the feeling that everything worked well (but didn't).

Collected reasons for rsync exit code 23

The final error message in the snapshot log is always

[I] Take snapshot (rsync: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7])
[E] 'rsync' ended with exit code 23: See 'man rsync' for more details

Here are typical snapshot log entries causing exit code 23 (shall be extended here to be extended step by step to get a holistic picture):

• [I] Take snapshot (rsync: symlink has no referent: "/home/user/Documents/dead-link")
• [E] Error: rsync: [sender] send_files failed to open "/home/user/Documents/root_only_file.txt": Permission denied (13)

Other known reasons: See #1587 (comment)

Current impact

Some of the reasons for exit code 23 may be out of control of the BiT user or are OK to be ignored since not considered as "severe" (eg. dead links).

Despite that BiT snapshots do always fail with an error now.

Even though rsync treats exit code 23 always as error the BiT user should have the freedom to treat some reasons for exit code 23 as INFO or WARNING only...

Possible solutions

1. Add exit code 23 to the ignore list (as before) and let the user think everything went OK (until looking into the snapshot log)
   

   backintime/common/snapshots.py

   Lines 1247 to 1252 in 726bffa

   	# dict of exit codes (as keys) that are treated as INFO only by BiT
   	# (not as ERROR). The values are message strings for the snapshot log.
   	rsync_non_error_exit_codes = {
   	0: _("Success"),
   	24: _("Partial transfer due to vanished source files (see 'man rsync')")
   	}

2. Find and use rsync options so that we can ignore minor exit code 23 reasons:
   https://unix.stackexchange.com/questions/598736/how-to-skip-copying-broken-links-with-rsync-copy-links

3. Introduce a new snapshot result state "Warning" (currently we have only "OK" and "with errors)
   

   backintime/common/snapshots.py

   Lines 1110 to 1131 in 726bffa

   	def takeSnapshot(self, sid, now, include_folders):
   	"""
   	This is the main backup routine. It will take a new snapshot and store
   	permissions of included files and folders into ``fileinfo.bz2``.
   	Args:
   	sid (SID): snapshot ID which the new snapshot
   	should get
   	now (datetime.datetime): date and time when this snapshot was
   	started
   	include_folders (list): folders to include. list of
   	tuples (item, int) where ``int`` is 0
   	if ``item`` is a folder or 1 if ``item``
   	is a file
   	Returns:
   	list: list of two bool
   	(``ret_val``, ``ret_error``)
   	where ``ret_val`` is ``True`` if a new
   	snapshot has been created and
   	``ret_error`` is ``True`` if there was
   	an error during taking the snapshot

Challenges

rsync has several reasons for exit code 23 and always treats it as an error.

Related issues

• Rsync error 23 (partial transfer due to error) undiscovered #1555
• How to automatically retake snapshot after rsync error 23 using user-callback #1573

[buhtz]

It might help or give us some more ideas if we would involve the rsync-folks and -maintainer via their mailing list.

I assume there is a reason why multiple problems are summarized with one single error code.

[aryoda]

FYI: I have just posted my questions at the rsync mailing list
and also looked into the relevant (C) source code of rsync

https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1271C1-L1282C28

but without debugging rsync (which I won't do ;-) I see no way to find out how to suppress this error (one of many that leads to exit code 23):

symlink has no referent: %s

[aryoda]

Internal note: I have examined the rsync source code and identified these possibles reasons (citing the error msg format string here) leading to exit code 23:

1. receive_sums failed
2. send_files failed to open %s
3. fstat failed
4. read errors mapping %s
5. change_dir %s failed
6. skipping overly long name: %s
7. symlink has no referent: %s
8. readlink_stat(%s) failed
9. skipping file with bogus (zero) st_mode: %s
10. skipping symlink with 0-length value: %s
11. [%s] cannot convert filename: %s (%s)
12. [%s] cannot convert symlink data for: %s (%s)
13. get_acl(fname, &sx) < 0 // with no explicit error message!
14. get_xattr(fname, &sx) < 0 // with no explicit error message!
15. link_stat %s failed
16. opendir %s failed
17. filename overflows max-path len by %u: %s/%s
18. cannot send file with empty name in %s
19. readdir(%s)
20. link_stat %s failed
21. cannot add local filter rules in long-named directory: %s

[emtiu]

Ideally, I would prefer the solution where an rsync-23 marks a snapshot as "with warnings" (in addition to "with errors"). Users could then clear the warnings (e.g. by renaming the snapshot).

I don't think it's worth the trouble of second-guessing rsync by trying to figure out why exactly a 23 was returned. Users can do that by looking at the logs.

[buhtz]

With my current but limited knowledge I also do prefer "with warnings".
But we need to add enough documentation about it. Because this will trigger user to ask questions in Issues and on the mailing list.

They have to investigate the log files etc pp

This is also related to our log-file-parsing issue. When we are able to parse the log file then we are able to give better feedback to the user which type of code 23 caused the warning. But in a first place I would say a link to a docu/faq would be enough in the first place.

[aryoda]

Ideally, I would prefer the solution where an rsync-23 marks a snapshot as "with warnings" (in addition to "with errors").
[...]
Users can do that by looking at the logs.

This would be my preferred (and perfect) solution too.

Unfortunately

• our snapshot logs do currently support only [I] and [E] (no warnings)

• the snapshot log viewer does not yet allow filtering on warnings

• the above mentioned 21 reasons for exit code 23 are contained as [I] only in the snapshot logs

  [I] Take snapshot (rsync: symlink has no referent: "/home/user/Documents/dead-link")
  

• rsync does not provide a reliable way to allow our log parser to recognize the 21 reasons for exit code 23
  except parsing the message text
  so we are not able to easily and reliably mark them as [W]
  (without false positives or missed entries due to rsync msg text changes or reasons I have overseen in the code)

  => this means users cannot even filter and check the log entries causing exit code 23 easily

  User had to scroll through the complete snapshot log or search for keywords (which are determined by the msg strings of the 21 rsync reasons)

• I am not sure if rsync's output may be translated into other languages (which would effectively render log parsing into nightmare ;-)

  Lukily in the source code I could find no indication for that but I will ask at the rsync mailing list.

All this is IMHO too risky and time consuming ATM (I prefer to prepare the next release for Jan, 2023 and the Qt6 release then).

I propose to

• DONE: Add a FAQ entry for exit code 23 (as @buhtz suggested)
• DONE: Add exit code 23 to the exclusion list for until we have introduced warnings.
  This effectively re-introduces the same behavior as before checking the rsync exit codes for errors