UnicodeDecodeError (testing files included)

[pureair]

Environment:

Python version: 3.12.2
adb version: Android Debug Bridge version 1.0.41 / Version 35.0.1-11580240
Operating System: Windows 10

Error Description:

The adbsync command fails with a UnicodeDecodeError when encountering specific folder/file names on the Android device. The error message indicates:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 62: invalid start byte
Full error log is at the end of this report.

Investigation:

I have narrowed down the problematic folder/files and included an archive (tester.tar.gz) containing their names and folder structure for further analysis. The content of the files have been emptied, only their filenames are kept.

Each of the three folders would be:

1. successfully pushed to the phone initially (as there is no existing files on the phone)
2. failed to be pulled from the phone
3. failed to be pushed to the phone again (as existing files on the phone would be examined first)
4. a) the folder with only English characters will be pulled with "--adb-encoding latin1", but the defect remains (ie. do step 1, 2, 3 to the newly pulled files and the result is the same). I think this means the filenames are perfectly utf-8 encoded so nothing changes.
   b) the folder with some Chinese characters will not be pulled with "--adb-encoding latin1" potentially because of bad filenames (error log below)

[INFO] SYNCING
[INFO]
[INFO] Empty delete tree
[INFO]
[INFO] Copying copy tree
[INFO] .\
[INFO] ./éè´é¸-HOYO-MiX - åç¥-éªèç群æ The Stellar Moments\
[INFO] ./éè´é¸-HOYO-MiX - åç¥-éªèç群æ The Stellar Moments/01. Bard's Adventure è¯äººçå·¥ä½.m4a
[CRITICAL] Non-zero exit code from adb pull
[CRITICAL] Exiting

they will not be pulled with "--adb-encoding gb2312" or gbk, gb18080, utf-8, utf-16, etc. because of 'utf-8' codec can't decode byte 0x0b in position xx (0xe9, 0xb8, etc.).

I actually look at the hex of the folder and files names of the folder with only English characters, there is actually no 0xa0 in either the folder name or the file names.

---

Full error log:

PS G:\> adbsync -n pull /sdcard/Music ./
* daemon not running; starting now at tcp:5037
* daemon started successfully
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\USERX\AppData\Local\Programs\Python\Python312\Scripts\adbsync.exe\__main__.py", line 7, in <module>
  File "C:\Users\USERX\AppData\Local\Programs\Python\Python312\Lib\site-packages\BetterADBSync\__init__.py", line 374, in main
    files_tree_source = fs_source.get_files_tree(path_source, follow_links = args.copy_links)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERX\AppData\Local\Programs\Python\Python312\Lib\site-packages\BetterADBSync\FileSystems\Base.py", line 45, in get_files_tree
    return self._get_files_tree(tree_path, statObject, follow_links = follow_links)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERX\AppData\Local\Programs\Python\Python312\Lib\site-packages\BetterADBSync\FileSystems\Base.py", line 33, in _get_files_tree
    tree[filename] = self._get_files_tree(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USERX\AppData\Local\Programs\Python\Python312\Lib\site-packages\BetterADBSync\FileSystems\Base.py", line 30, in _get_files_tree
    for filename, stat_object_child, in self.lstat_in_dir(tree_path):
  File "C:\Users\USERX\AppData\Local\Programs\Python\Python312\Lib\site-packages\BetterADBSync\FileSystems\Android.py", line 176, in lstat_in_dir
    for line in self.adb_shell(["ls", "-la", path]):
  File "C:\Users\USERX\AppData\Local\Programs\Python\Python312\Lib\site-packages\BetterADBSync\FileSystems\Android.py", line 87, in adb_shell
    adb_line = adb_line.decode(self.adb_encoding).rstrip("\r\n")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 62: invalid start byte

[LazyWizard]

Encountered this error myself and decided to take a whack at it. Thanks for the test data, it makes things a million times easier!

I think with byte strings you need to use the dummy encoder unicode_escape when decoding strings containing escaped multi-byte unicode characters, but when I tried this it returned the right filenames but ADB couldn't find the files. Never mind, looks like it fixes the non-breaking space that's causing sync issues but breaks on actual multi-byte characters.

I'm guessing it only has to be decoded this way on one side (wherever it's actually broken), but I'm unfamiliar with the codebase. I'll tinker with the code a bit and see if I can get things working.

[gAtrium]

I have also tried to tackle this problem, there are indeed nbsp characters present in the stdout.
What I have done to fix the nbsp problem is to get the output as raw bytes, iterate over each byte and fix any nbsp charcodes that are not part of a multi-byte utf-8 character.
This is sort of google's issue as well for not parsing wildcard (*) characters to match these problematic filenames during pull, though I could be wrong.

[pureair]

OK, it seems replacing non-breaking space 0xa0 with regular space solves the problem.
I wrote a powershell script to batch rename files:

# Get all files in the current directory and sub-directory
$files = Get-ChildItem -File -Recurse

foreach ($file in $files) {
    # Create the new file name by replacing spaces with "$~#"
    $newFileName = $file.Name -replace [char]0x00A0 , '_'
	
    # Define the full path for the new file name
    $newFilePath = Join-Path -Path $file.DirectoryName -ChildPath $newFileName
    
    # Rename the file if the new name is different
    if ($file.FullName -ne $newFilePath) {
	echo Renaming "$file.FullName"
        Rename-Item -Path $file.FullName -NewName $newFilePath
    }
}

pause

Not sure how to run bash terminal in android with file access to /sdcard/, so not able to write a bash equivalent and test on android.

[KirllosAtef]

I'm using this script to backup my android phone, but I encountered this bug here?
Can someone explain a working workaround, that can be used for my phone? "I'm not that good at scripting"
Hope the dev fix this bug!

[DimitarSamarov07]

Any updates on this? I am getting the same error trying to adb pull my Download folder.

[gAtrium]

Would anyone be interested in a method of creating a tar file as the backup similar to how TWRP does?

[DimitarSamarov07]

Would anyone be interested in a method of creating a tar file as the backup similar to how TWRP does?

I honestly prefer the rsync-like approach, as long as it works properly. This way, data is easily viewable from the PC, and it isn't bound to a single file corrupting. Could be just me, but the current approach feels better.

I really hope that this project isn't orphaned as it is way faster than rsync through mtp, despite the bugs with certain files. If this gets fixed, I will be very pleased.

[gAtrium]

Would anyone be interested in a method of creating a tar file as the backup similar to how TWRP does?

I honestly prefer the rsync-like approach, as long as it works properly. This way, data is easily viewable from the PC, and it isn't bound to a single file corrupting. Could be just me, but the current approach feels better.

I really hope that this project isn't orphaned as it is way faster than rsync through mtp, despite the bugs with certain files. If this gets fixed, I will be very pleased.

Due to how different filesystems handle names and paths differently there will always be a case where files would be unable to created/pulled, most obvious example is "con" in windows
Android mainly uses f2fs (iirc), host FS are either NTFS, ext4/btrfs, APFS.
I have used different back up solutions to date, most proprietary solutions resort to installing a third party android application to the device and creating a reverse tunnel (adb reverse) and communicating from there. However this falls short when the device is in recovery mode (Is networking enabled in recovery mode? Enabled through init?).
It seems the proper way to combat is to create binaries for each target architecture to listen on a predefined port (which can fail for the reasons specified before), send the file list, if there are any characters or long paths there is no choice but to resort to some ugly hacks, which might cause issues if the file is application data located in Android/data/ directory.