nsys and ncu profiling on the real colabfold GPU DBs

[keiran-rowell-unsw]

The MMSeqs2 /example .fastas are too small (1k seqs query, 40k seqs DB)

Nsys shows 44% time spend on memset but that should be a fixed cost, so see what realistic CUDA occupancy looks like on the canonical DBs (setup script running evening of 2026-02-18)

Nsight tools (nsys and ncu) are installed user-land from the run file, gather meaningful profiles to present to NVIDIA reps

[tlitfin]

Can we force async loading to VRAM more aggressively to avoid costly startup?

[keiran-rowell-unsw]

Tipping point where hostAlloc doesn't dominate is the 10,000+ order-of-magnitude query. But this is only against a 10k targetDB while setting up colabfolddb indexing.

There are long cudaStreamSynchronise (approx. 70%) but during the synchronisation to CPU, GPU is doing work running kernels (see the blue trace)

So it's not like there's idling GPU that a multi-process service could take advantage but want to check with these mentors.

[keiran-rowell-unsw]

For the 10k targetDB, consistently a 1.9s start-up time per query, regardless of query size. This is with --gpu 1 assuming the end site doesn't have gpu-server loaded

Time (s) the kernel is occupied, by # query sequences [H100]:

• 100: 0.2
• 1,000: 1.6
• 10,000: 17
• 100,000:

The B200s paid a 20 seconds start up time -- I want to look into why that seems to be that much worse, did something change beyond the GPU

Complete H100/B200 equivalent factor of x10 tomorrow.

Oh also is the multi-GPU mode increasing the startup cost?

Uniref30:
1x B200 - 100 query - 18s of cudaMemcpyToSymbol, 18s of kernel execution
8x B200 - 100 query - 15s of cudaMemcpyToSymbol, 5s of kernel execution
1x B200 - 1,000 query - 13s of cudaMemcpyToSymbol, 155s kernel execution
1 x B200 - 10,000 query -

Colabfoldenvdb:
1x B200 - 100 query - 14s of cudaMemcpyToSymbol, 66s of kernel execution
8x B200 - 100 query - 14s of cudaMemcpyToSymbol, 16s of kernel execution [only one fully saturated]
1x B200 - 1,000 query - 15s of cudaMemcpyToSymbol, 651s of kernel execution
8x B200 - 1,000 query - 15s of cudaMemcpyToSymbol, 156s of kernel execution
45 GB MIG (2/4) - 100 query - 20s of cudaMemcpyToSymbol, 300s of kernel execution time
45 GB MIG (1/4) - 1,000 query - 15s of cudaMemcpyToSymbol, 3000s of kernel execution (not 4x)

The cudaMemcpyToSymbol is a context start up cost, forcing JIT compilation (CUDA_FORCE_PTX_JIT=1) and remove the cache (CUDA_CACHE_DISABLE=1) balloons it out to 200s!

[keiran-rowell-unsw]

Tomorrow set up the gpu-server and running and empirically test timing of with cuda-mps on

[keiran-rowell-unsw]

Nsys appears to demonstrate this is not a productive avenue. Closing to focus on the nextflow pipeline

[keiran-rowell-unsw]

Tom suggested trying to view asynchronous streaming where it doesn't fit into RAM but B200s have amble RAM and nvidia-smi doesn't provide memory exposure tweaking on a card level.

Workaround is using MIG to create 45 GB VRAM partition and test on that.