List pattern support

[juusom]

Add support for reading from list patterns such as logstash.*. Related issue: #49

The motivation in our case is that we want to be able to see and control when individual applications produce spammy logs for whatever reason. In those situations, it's also important to be able to guarantee some level of fairness when consuming the queues, as opposed to having just one queue dominated by a potentially misbehaving application.

The implementation supports consuming single items per list using LPOP, as well as batched consuming using the same EVAL based logic as the regular list data_type input.

The logic is based on using a configurable-size concurrent-ruby thread pool. The main loop keeps polling the key pattern using the KEYS command and whenever the thread pool queue isn't full, posts a new job to the thread pool for consuming a single key. It keeps track of the keys currently queued and makes sure there's only one job per key at a time.

I also did some refactoring and wanted to do more but decided to make this PR before going further in order to get some opinions. So far I extracted the Redis connection logic as a MixIn - it's all in a single commit and can be reverted if need be. (edit: scratch that, I reverted this bit already, for now) What I'd like to do is make all the 3 different use cases (list, pattern list, PubSub) independent classes implementing an interface. It's currently quite messy with so much unrelated code in a single class.

I'm not a Ruby developer (although I did gather input from colleagues who are) so let me know if there's anything silly, or if this somehow violates any logstash/elastic practices :)

[cla-checker-service]

💚 CLA has been signed

[bbmhmmad]

@juusom im unable to get this to work after installing logstash-input-redis off this branch. Logstash is not pulling redis messages.

Redis Key: new.data.2020-06-16-01-24-21

input {

    redis {
        host => "iris-redis-slave"
        port => 6379
        data_type => "list"
        password => "${TL_REDIS_PASSWORD}"
        key   => "new.data*"
        threads => 4
        batch_count => 1000
    }

[juusom]

Thanks for your interest in this @bbmhmmad! In your config snippet, the data type is wrong. It should be pattern_list. Also, you probably don't want to use threads to configure this, but instead use pattern_list_threads.