GUIDE: FIX FOR DEPLOYMENT ERRORS & TENSORRT CRASHES

[smyhlin]

GUIDE: FIX FOR DEPLOYMENT ERRORS & TENSORRT CRASHES

FOR THOSE UNABLE TO LAUNCH THE APPLICATION

Since the current installer and portable versions have issues with unpacking/installing dependencies, follow these steps to get it working manually.

1. Download the Portable Version
   Download the latest portable release here:
   https://github.com/the-database/VideoJaNai/releases/download/1.2.2/VideoJaNai-win-Portable.zip

2. Run it Once
   Extract it and attempt to run the .exe.
   NOTE: You will receive an error at this stage. This is expected. We just need it to initialize the Python environment and folder structure.

3. Download Working Dependencies
   Download the dependency pack from this link:
   https://github.com/the-database/mpv-upscale-2x_animejanai/releases/tag/3.2.2
   (Look for the portable zip in the assets)

4. Replace Files (Crucial Step)
   Extract the downloaded dependency archive. You need to move the following folders into your VideoJaNai AppData directory:

   • animejanai
   • Lib
   • Locale
   • portable_config
   • vs-coreplugins
   • vs-plugins
   • vs-scripts
   • vsgenstubs4

   Destination: C:\Users\%username%\AppData\Roaming\VideoJaNai\Python

   IMPORTANT: When asked, select REPLACE FILES (Overwrite everything).

   After this, the application should launch correctly.

---

TENSORRT SPECIFIC CRASHES (RTX 20xx/30xx vs 40xx/50xx)

If you are using the TensorRT backend (which offers ~40% performance boost), you might encounter crashes depending on your GPU architecture.

• RTX 5090 / 4090: The code works fine out of the box.
• RTX 2070 Super (and similar older generations): You may encounter bits per sample mismatch errors because the strict TensorRT engine generation requires specific FP16 (RGBH) input/output, but the code does not handle the format conversion automatically.

The Error

If you see logs like this:

vapoursynth.Error: Levels: Input clip must be constant format 8..16 bit integer or 32 bit float, passed RGBH.
...
vapoursynth.Error: operator (): bits per sample mismatch

The Fix (No performance loss)

You need to manually edit one Python file to ensure formats are converted correctly before entering the TensorRT engine and after exiting it.

File location:
C:\Users\%username%\Downloads\VideoJaNai-win-Portable Prerelease\current\backend\animejanai\core\animejanai_core.py
(Or wherever you extracted the portable version)

FIX 1: Input to TensorRT

Find the function upscale2x_trt (around line 153).

BEFORE:

def upscale2x_trt(clip, engine_name, num_streams, trt_settings):
    # ... (engine check code) ...
    return core.trt.Model(
        clip,
        engine_path=engine_path,
        num_streams=num_streams
    )

AFTER (Replace with this):

def upscale2x_trt(clip, engine_name, num_streams, trt_settings):
    engine_path = get_engine_path(engine_name, trt_settings)
    if not os.path.isfile(engine_path):
        create_custom_engine(engine_name, trt_settings)

    if not os.path.exists(engine_path):
        logger.debug("Engine failed to generate, exiting...")
        sys.exit(1)

    # === FIX: FORCE RGBH INPUT FOR TENSORRT ===
    # Older GPUs/drivers are strict about input format matching the engine (FP16)
    if clip.format.id != vs.RGBH:
        clip = vs.core.resize.Bicubic(clip, format=vs.RGBH, matrix_in_s="709")
    # ==========================================

    return core.trt.Model(
        clip,
        engine_path=engine_path,
        num_streams=num_streams
    )

FIX 2: Output before 'Levels' filter

Find the function run_animejanai_upscale (around line 260).

BEFORE:

    # ...
    # upscale 2x
    clip = upscale2x(clip, backend, model_conf['name'], num_streams, trt_settings)
    
    clip = vs.core.std.Levels(clip, min_in=0.0, max_in=1.0, min_out=0.0, max_out=1.0)
    return clip

AFTER (Replace with this):

    # ...
    # upscale 2x
    clip = upscale2x(clip, backend, model_conf['name'], num_streams, trt_settings)

    # === FIX: CONVERT BACK TO RGBS BEFORE LEVELS ===
    # 'Levels' filter crashes on RGBH (FP16) input, needs RGBS (FP32)
    if clip.format.sample_type == vs.FLOAT and clip.format.bits_per_sample == 16:
        clip = vs.core.resize.Bicubic(clip, format=vs.RGBS)
    # ===============================================

    clip = vs.core.std.Levels(clip, min_in=0.0, max_in=1.0, min_out=0.0, max_out=1.0)
    return clip

Applying these fixes resolves the format mismatch errors on RTX 20xx/30xx cards while maintaining full FP16 performance.

[Hyphonical]

Thank you ❤️

[SmaugTD]

Neat