Supported IP2Location.io API

Author: ip2location-com

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2022 IP2Location.com
+Copyright (c) 2024 IP2Location.com
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

README.md

@@ -1,5 +1,5 @@
 # IP2Proxy Filter Plugin
-This is IP2Proxy filter plugin for Logstash that enables Logstash's users to reverse search of IP address to detect VPN servers, open proxies, web proxies, Tor exit nodes, search engine robots, data center ranges, residential proxies, consumer privacy networks, and enterprise private networks using IP2Proxy BIN database. Other information available includes proxy type, country, state, city, ISP, domain name, usage type, AS number, AS name, threats, last seen date and provider names. The library took the proxy IP address from **IP2Proxy BIN Data** file.
+This is IP2Proxy filter plugin for Logstash that enables Logstash's users to reverse search of IP address to detect VPN servers, open proxies, web proxies, Tor exit nodes, search engine robots, data center ranges, residential proxies, consumer privacy networks, and enterprise private networks using IP2Proxy BIN database. Other information available includes proxy type, country, state, city, ISP, domain name, usage type, AS number, AS name, threats, last seen date and provider names. The library took the proxy IP address from **IP2Proxy BIN Data** file and **IP2Location.io** data.
 
 For the methods to use IP2Proxy filter plugin with Elastic Stack (Elasticsearch, Filebeat, Logstash, and Kibana), please take a look on this [tutorial](https://blog.ip2location.com/knowledge-base/how-to-use-ip2proxy-filter-plugin-with-elastic-stack).
 
@@ -11,6 +11,9 @@ This plugin requires IP2Proxy BIN data file to function. You may download the BI
 * IP2Proxy LITE BIN Data (Free): https://lite.ip2location.com
 * IP2Proxy Commercial BIN Data (Commercial): https://www.ip2location.com
 
+## Dependencies (IP2LOCATION.IO DATA)
+This plugin requires API key to function. You may sign up for a free API key at https://www.ip2location.io/pricing.
+
 
 ## Installation
 Install this plugin by the following code:
@@ -32,7 +35,33 @@ filter {
     match => { "message" => "%{COMBINEDAPACHELOG}"}
   }
   ip2proxy {
-    source => "clientip"
+    source => "[source][address]"
+  }
+}
+
+output {
+  elasticsearch {
+    hosts => [ "localhost:9200" ]
+  }
+}
+```
+
+## Config File Example using IP2Location.io
+```
+input {
+  beats {
+    port => "5043"
+  }
+}
+
+filter {
+  grok {
+    match => { "message" => "%{COMBINEDAPACHELOG}"}
+  }
+  ip2proxy {
+    source => "[source][address]"
+    lookup_type => "ws"
+    api_key => "YOUR_API_KEY"
   }
 }
 
@@ -51,12 +80,16 @@ output {
 |database|a valid filesystem path|No|
 |use_memory_mapped|boolean|No|
 |use_cache|boolean|No|
+|lookup_type|string|No|
+|api_key|string|No|
 |hide_unsupported_fields|boolean|No|
 
 * **source** field is a required setting that containing the IP address or hostname to get the ip information.
 * **database** field is an optional setting that containing the path to the IP2Proxy BIN database file.
 * **use_memory_mapped** field is an optional setting that used to allow user to enable the use of memory mapped file. Default value is false.
 * **use_cache** field is an optional setting that used to allow user to enable the use of cache. Default value is true.
+* **lookup_type** field is an optional setting that used to allow user to decide the lookup method either using IP2Proxy BIN database file(db) or IP2Location.io data(ws). Default value is db.
+* **api_key** field is an optional setting that used to allow user to set the API Key of the IP2Location.io lookup.
 * **hide_unsupported_fields** field is an optional setting that used to allow user to hide unsupported fields. Default value is false.
 
 

lib/logstash-filter-ip2proxy_jars.rb

@@ -1,3 +1,4 @@
 require 'jar_dependencies'
 require_jar('com.ip2proxy.ip2proxy', 'ip2proxy', '3.4.0')
-require_jar('org.logstash.filters', 'logstash-filter-ip2proxy', '2.3.2')
+require_jar('com.google.gson', 'gson', '2.11.0')
+require_jar('org.logstash.filters', 'logstash-filter-ip2proxy', '2.4.0')

lib/logstash/filters/ip2proxy.rb

@@ -27,41 +27,61 @@ class LogStash::Filters::IP2Proxy < LogStash::Filters::Base
   # The field used to allow user to hide unsupported fields.
   config :hide_unsupported_fields, :validate => :boolean, :default => false
 
+  # The field used to define lookup type.
+  config :lookup_type, :validate => :string, :default => 'db'
+
+  # The field used to define the apikey of IP2location.io.
+  config :api_key, :validate => :string, :default => ''
+
   # The field used to define the size of the cache. It is not required and the default value is 10 000 
   config :cache_size, :validate => :number, :required => false, :default => 10_000
 
   public
   def register
-    if @database.nil?
-      @database = ::Dir.glob(::File.join(::File.expand_path("../../../vendor/", ::File.dirname(__FILE__)),"IP2PROXY-LITE-PX1.BIN")).first
-
-      if @database.nil? || !File.exists?(@database)
-        raise "You must specify 'database => ...' in your ip2proxy filter (I looked for '#{@database}')"
+    if @lookup_type == "ws"
+      @logger.info("Using IP2Location.io API")
+      if @api_key == ""
+        raise "An IP2Location.io API key is required. You may sign up for a free API key at https://www.ip2location.io/pricing."
+      end
+    else
+      if @database.nil?
+        @database = ::Dir.glob(::File.join(::File.expand_path("../../../vendor/", ::File.dirname(__FILE__)),"IP2PROXY-LITE-PX1.BIN")).first
+  
+        if @database.nil? || !File.exists?(@database)
+          raise "You must specify 'database => ...' in your ip2proxy filter (I looked for '#{@database}')"
+        end
       end
+      @logger.info("Using ip2proxy database", :path => @database)
     end
 
-    @logger.info("Using ip2proxy database", :path => @database)
-
-    @ip2proxyfilter = org.logstash.filters.IP2ProxyFilter.new(@source, @target, @database, @use_memory_mapped, @hide_unsupported_fields)
+    @ip2proxyfilter = org.logstash.filters.IP2ProxyFilter.new(@source, @target, @database, @use_memory_mapped, @hide_unsupported_fields, @lookup_type, @api_key)
   end
 
   public
   def filter(event)
     ip = event.get(@source)
 
     return unless filter?(event)
-    if @use_cache
-      if value = IP2ProxyCache.find(event, ip, @ip2proxyfilter, @cache_size).get('ip2proxy')
-        event.set('ip2proxy', value)
+    if @lookup_type == "ws"
+      if @ip2proxyfilter.handleEvent(event)
         filter_matched(event)
       else
         tag_iplookup_unsuccessful(event)
       end
     else
-      if @ip2proxyfilter.handleEvent(event)
-        filter_matched(event)
+      if @use_cache
+        if value = IP2ProxyCache.find(event, ip, @ip2proxyfilter, @cache_size).get('ip2proxy')
+          event.set('ip2proxy', value)
+          filter_matched(event)
+        else
+          tag_iplookup_unsuccessful(event)
+        end
       else
-        tag_iplookup_unsuccessful(event)
+        if @ip2proxyfilter.handleEvent(event)
+          filter_matched(event)
+        else
+          tag_iplookup_unsuccessful(event)
+        end
       end
     end
   end

logstash-filter-ip2proxy.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-filter-ip2proxy'
-  s.version         = '2.3.2'
+  s.version         = '2.4.0'
   s.licenses        = ['Apache-2.0']
   s.summary         = "Logstash filter IP2Proxy"
   s.description     = "IP2Proxy filter plugin for Logstash enables Logstash's users to reverse search of IP address to detect VPN servers, open proxies, web proxies, Tor exit nodes, search engine robots, data center ranges and residential proxies using IP2Proxy BIN database."