source._get_buffers calls opts.get_bufnrs() to get the buffers for indexing. It checks if each buffer is in buffers and adds it if it's not there. It fails to remove buffers that were not returned from opts.get_bufnrs().

I was experiencing the performance issues described in #40 and this was the cause. The number of indexed buffers only grows, regardless of what get_bufnrs() returns.