Skip to content

git-annex-basics.md

Latest commit

 

History

History
223 lines (219 loc) · 7.15 KB

git-annex-basics.md

File metadata and controls

223 lines (219 loc) · 7.15 KB
Raw

Git-Annex

Git-Annex is a data management tool designed for handling large files with Git. Under the hood, git-annex creates a "git-annex" branch and uses hashes, symlinks, and checksums to manage these files (metadata and file locations). This allows the content of the files to be version-controlled in normal git repositories while the actual files can be stored elsewhere.

Installation

  • Linux (Debian) 
    apt update
apt install git git-annex

Usage

  • Initialization 
    git init
git annex init ["REPO_NAME"] [-c annex.commitmessage="MESSAGE"]
    • git annex init (or git-annex init) creates a "git-annex" branch.
  • Configuration
    • For git 
      git config --local user.name "NAME"
git config --local user.email "EMAIL"
    • For git-annex (optional)
      • Define rules for file tracking (e.g., only track files larger than 100KB in data/ folder) 
        git config --local annex.largefiles 'largerthan=100kb and include=data/*'
      • Not to push local metadata to remote 
        git config annex.private true
      • Protect remote from modification (e.g., git-annex sync) 
        git config remote.origin.annex-readonly true
      • Not to store annexed files from remote 
        git config remote.origin.annex-ignore true
      • Set minimum number of copies 
        git annex numcopies N # default: 1
    • Check configuration 
      git config --list
  • Special remotes
    • Add special remotes 
      git-annex initremote REMOTE_NAME type=TYPE PARAM=VALUE [OPTIONS] [-c annex.commitmessage="MESSAGE"]
git-annex describe REMOTE_NAME "DESCRIPTION"
      • [OPTIONS]
        • exporttree=yes: use readable names and paths
      • Examples
        • type=rsync rsyncurl=URL encryption=none
        • type=webdav url=URL encryption=shared
        • type=directory directory=PATH encryption=none
        • type=httpalso url=URL
        • type=rclone ...
    • Enable remote (before getting data) 
      git-annex enableremote # check available remotes
git-annex enableremote REMOTE_NAME
    • Remove special remotes 
      git remote rm REMOTE_NAME
git-annex dead REMOTE_NAME
git-annex forget --drop-dead --force
    • Configure sync-branch 
      git config --local annex.synconlyannex true
    • Preferred content
  • Data management
    • Add data 
      git-annex add [OPTIONS] DATA [-c annex.commitmessage="MESSAGE"]
git commit -m "MESSAGE"
      • Replace files with symbolic links pointing to the contents stored in the .git/annex/objects/.
      • [OPTIONS]
        • --force-small: add to git as all files were small
        • --force-large: add to git-annex as all files were large
    • Move data 
      git-annex move DATA [--from=LOCATION|--to=LOCATION]
    • Sync data 
      git-annex sync [OPTIONS]
      • [OPTIONS]
        • --only-annex: only sync "git-annex" branches
        • --cleanup: clean up git-annex-created branches
    • Copy data 
      git-annex copy DATA [--from=LOCATION|--to=LOCATION]
    • Export data 
      git-annex export BRANCH --to LOCATION

      [!Note]

      • Use when initremote with exporttree=yes
      • Less flexible than copy
    • Drop data 
      git-annex drop DATA [OPTIONS]
      • [OPTIONS]
        • --from=LOCATION
        • --force
    • Reinject known data 
      git-annex reinject SRC DEST
    • Get data
      • Download from URLs (e.g., web or bittorrent) 
        git-annex addurl [OPTIONS] URL
        • [OPTIONS]
          • --file=NAME: download file as NAME
          • --preserve-filename --pathdepth=N: download with original filename with URL path depth of N (e.g., -1, 1, 3)
          • --fast/--relaxed: skip verfication
      • Remove URLs 
        git-annex rmurl DATA URL
      • Register URLs 
        KEY=$(basename `readlink DATA`)
#KEY=$(git-annex lookupkey DATA)
git-annex registerurl $KEY URL
      • Get annexed files 
        git-annex get [OPTIONS] DATA
        • [OPTIONS]
          • --debug (git-annex get --debug DATA 2>&1 | grep -v process)
    • Switch backend (e.g. from URL to MD5) 
      git-annex migrate [--backend=BACKEND] DATA
      • BACKEND: SHA256E, MD5, etc.
    • Unlocak files 
      git-annex unlock DATA
# chmod -R +w

      [!Note]

      • Since annexed files are write-protected, unlock them before editing.
    • Unannex data (e.g., for archiving) 
      git-annex unannex DATA
  • Data inspection
    • Show data information 
      git-annex info [OPTIONS] [DATA]
      • [OPTIONS]
        • --bytes: display file sizes in bytes

      [!Note]

      • For keys starting with URL-, content cannot be verified (untracked).
      • For key starting with SHA256E-/MD5E-, data integrity can be checked.
    • List repositories that contain data 
      git-annex whereis [DATA]
    • Display a table of remotes that contain data 
      git-annex list [DATA]
      • Output example 
        here
|REMOTE
||DIR
|||web
||||bittorrent
|||||
X_X__ DATA
    • File system check 
      git fsck # for git database
git-annex fsck # for annexed files
  • Remote Git repository
    • Add remote repositories 
      git remote add REMOTE_REPO URL
    • Push data 
      git push REMOTE_REPO main
git push REMOTE_REPO git-annex
  • Miscellaneous
    • Initialize a shared bare repository 
      git init --share=group --bare BARE_REPO
    • Show git history 
      git log --all --graph --decorate --oneline
    • Archive git repository 
      git archive --format=tar.gz --prefix REPO_NAME/ -o ../archive.tgz HEAD

References