Tex2Sem

Tex2Sem: Learning from Textures to Semantics for Robust Semantic Correspondence

This paper was accepted for publication in IEEE Transactions on Circuits and Systems for Video Technology (https://ieeexplore.ieee.org/document/10311392/).

Abstract

Recent advances in semantic correspondence have witnessed growing interest in SD and DINO. However, existing methods underutilize the matching potential of SD and DINOv2 features and show similar background interference patterns. They lack texture-to-semantic learning and intra- and inter-image feature interaction. This study proposes Tex2Sem, a framework learning from textures to semantics, to address the two problems. For the first problem, we propose a texture-to-semantic learning paradigm that achieves texture-semantic trade-offs on features and correlation maps. SD and DINOv2 features are aggregated from textures to semantics to produce multi-stage progressive fusion features. For the second problem, MamFormer, a hybrid architecture of Mamba-2 and Transformer, is proposed to improve intra- and inter-image feature aggregation and interaction. The terminal-stage aggregation and interaction mechanism (TAIM) is proposed to enhance feature learning efficiency.

Application

• Tex2Sem-based Video Swap

• Tex2Sem-based Human Pose Estimation

Environment Setup

conda create -n tex2sem python=3.9
conda activate tex2sem
conda install pytorch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 pytorch-cuda=11.8 -c pytorch -c nvidia
conda install -c "nvidia/label/cuda-11.8.0" libcusolver-dev
git clone git@github.com:wzhlearning/Tex2Sem.git 
cd Tex2Sem
pip install -e .

You should install xformers for efficient transformer (which can significantly reduce the VRAM consumpution):

pip install xformers==0.0.23

You should install mamba:

pip install causal-conv1d>=1.4.0
pip install mamba-ssm==2.2.1

You should install SAM to extract the instance masks for adaptive pose alignment:

pip install git+https://github.com/facebookresearch/segment-anything.git
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

Start

1. Prepare datasets

We provide the scripts to download the datasets in the data folder. To download specific datasets, use the following commands:

• SPair-71k:

bash data/prepare_spair.sh

• PF-Pascal:

bash data/prepare_pfpascal.sh

• AP-10k:

First, download the original dataset with the following command:

bash data/prepare_ap10k.sh

Then, you could preprocess AP-10k to get the correspondence benchmark by running the notebook prepare_ap10k.ipynb.

2. Pre-extract features

To enable efficient training and evaluation, we pre-extract the feature maps of the datasets. To do so, run the following commands:

• SPair-71k:

python preprocess_map.py ./data/SPair-71k/JPEGImages

Substitute the path ./data/SPair-71k/JPEGImages with ./data/PF-Pascal/JPEGImages or ./data/AP-10k/JPEGImages for PF-Pascal and AP-10k, respectively.

3. Training

Training on AP-10k

python pck_train.py --config configs/train_ap10k.yaml

Some important parameters in train_ap10k.yaml include:

• --AP10K_EVAL_SUBSET to evaluate on the different test settings (e.g., cross-species, cross-family).
• --COMPUTE_GEOAWARE_METRICS: set to True to evaluate on the geometry-aware semantic correspondence subset.
• --PAIR_AUGMENT: set to True to enable the pose-variant pair augmentation.
• --LOAD: the path to the pretrained weight. Noth that train_ap10k.yaml does not require this option.

Pre-Training on AP-10k

AP-10k pre-training is actually performed on the test set. The parameter setting on line 327 of pck_train.py needs to be manually modified: change "split" from "val" to "test". Except for the AP-10 pretraining, all other experiments must be set to "val". See issue of GeoAware-SC

def eval(args, aggre_net, save_path, split='val'): #  pretrain ap10k: split='test'; other: split='val'

python pck_train.py --config configs/train_ap10k.yaml

Training on SPair-71k (with pretrained weight)

python pck_train.py --config configs/train_spair.yaml

Note that the TEST_SAMPLE is the number of sampled pairs for each category for evaluation, which is set to 20 by default. Set to 0 to use all the samples in the evaluation set.

Training on PF-Pascal (with pretrained weight)

python pck_train.py --config configs/train_pascal.yaml

4. Evaluation

python pck_train.py --config configs/eval_spair.yaml

You could also replace the --config with configs/eval_pascal.yaml or configs/eval_ap10k.yaml to evaluate on PF-Pascal or AP-10k.

Citation

@ARTICLE{Wang2025Tex2Sem,
  author={Wang, Zenghui and Du, Songlin and Yan, Yaping and Xiao, Guobao and Lu, Xiaobo},
  journal={IEEE Transactions on Circuits and Systems for Video Technology}, 
  title={Tex2Sem: Learning from Textures to Semantics for Robust Semantic Correspondence}, 
  year={2025},
  volume={35},
  number={11},
  pages={10875-10890},
  doi={10.1109/TCSVT.2025.3576772}}

Acknowledgement

Our code is largely based on the following open-source projects: A Tale of Two Features and GeoAware-SC. We are deeply grateful for their spirit of open-source sharing.