SDK failed to fetch X-Ray tcp address

[z-zhenbin]

We have a rails application and run in docker.

There is our docker-compose file content.

  web:
    image: xxx
    container_name: xxx
    command: ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]
    environment:
      AWS_XRAY_TRACING_NAME: xxxx
      AWS_XRAY_DAEMON_ADDRESS: x-ray:2000
      AWS_XRAY_CONTEXT_MISSING: LOG_ERROR
    ports:
      - 80:3000

  x-ray:
    image: amazon/aws-xray-daemon
    container_name: x-ray
    command: ['-t', '0.0.0.0:2000', '-o', '--log-level','dev']
    environment:
      AWS_REGION: ap-northeast-1
      AWS_ACCESS_KEY_ID: xxxx
      AWS_SECRET_ACCESS_KEY: xxxx
    ports:
      - 2000:2000/udp

Now, we can see the trace information at aws x-ray console. but the sdk always output warn log like this:

W, [2019-05-11T05:06:21.583099 #45]  WARN -- : No effective centralized sampling rule match. Fallback to local rules.

and this

W, [2019-05-11T05:06:21.611366 #54]  WARN -- : failed to fetch X-Ray sampling rules due to Failed to open TCP connection to 127.0.0.1:2000 (Connection refused - connect(2) for "127.0.0.1" port 2000)

Why SDK try to fetch x-ray sampling rule, but use the default address 127.0.0.1:2000? I have pass the x-ray daemon address by AWS_XRAY_DAEMON_ADDRESS, and the SDK also success to send trace information.

[z-zhenbin]

And we try to two version sdks.
The default version support by gem.

gem 'aws-xray-sdk'

and we also try to use the github master branch

gem 'aws-xray-sdk', 
  :git => 'https://github.com/aws/aws-xray-sdk-ruby.git', 
  :branch => 'master',
  require: ['aws-xray-sdk/facets/rails/railtie']

[z-zhenbin]

Update:
if I specify the daemon address in code like:

Rails.application.config.xray = {
    daemon_address: "172.27.0.2:2000",
    patch: %I[net_http aws_sdk],
    active_record: true
}

then will give me an exception like this:

/usr/local/bundle/bundler/gems/aws-xray-sdk-ruby-55291b0e94a9/lib/aws-xray-sdk/emitter/default_emitter.rb:39:in `rescue in daemon_config=': Invalid X-Ray daemon address specified: #<XRay::DaemonConfig:0x00005577dceafa48>. (XRay::InvalidDaemonAddressError)

Is this a bug?

[z-zhenbin]

Update:
I think maybe I found the reason.
When I JUST specify AWS_XRAY_DAEMON_ADDRESS, then the

aws-xray-sdk-ruby/lib/aws-xray-sdk/configuration.rb

Line 47 in 55291b0

sampler.daemon_config = config if sampler.respond_to?(:daemon_config=)

will be not called, it's mean the sampler will not update the daemon_config for poller.connector.

aws-xray-sdk-ruby/lib/aws-xray-sdk/sampling/default_sampler.rb

Line 73 in 55291b0

def daemon_config=(v)

So the poller will still use the deafult daemon address 127.0.0.1.

On the other hand. When I just specify daemon_address.
The configuration.rb will update the daemon_config twice.
First time, when the emitter be initail.

aws-xray-sdk-ruby/lib/aws-xray-sdk/configuration.rb

Line 28 in 55291b0

@emitter = DefaultEmitter.new

Second time when the daemon_address=(v) be called.
So the socket.connect() be called twice in the default_emitter.rb.

aws-xray-sdk-ruby/lib/aws-xray-sdk/emitter/default_emitter.rb

Line 37 in 55291b0

@socket.connect(v.udp_ip, v.udp_port)

So I think that casue the Exception.

[z-zhenbin]

Now I specify both AWS_XRAY_DAEMON_ADDRESS and daemon_address. Then every thing is OK.

[luluzhao]

Hi @z-zhenbin, thank you for contacting us and sorry for the late response. There are two addresses used by daemon. The first one is an udp address which daemon listens on to receive segments generated by X-Ray SDK. The second one is a tcp address which daemon fetches the centralized sampling rules from X-Ray backend. By default, both udp and tcp are using 127.0.0.1:2000. If you want to override two address, please using notation like this tcp:127.0.0.1:2000 udp:127.0.0.2:2001. Regarding of priority, first AWS_XRAY_DAEMON_ADDRESS and then recorder's configuration for daemon_address. Let me know if you have any other questions.

[z-zhenbin]

@luluzhao Thank you for your response. I tested that use the tcp:127.0.0.1:2000 udp:127.0.0.2:2001 for AWS_XRAY_DAEMON_ADDRESS, but it still have an exception. So like what I said I think that when I just use the AWS_XRAY_DAEMON_ADDRESS, I don't find the source code to update the tcp address for default_sampler.

[luluzhao]

@z-zhenbin . We have a bug fix which we haven't release it yet. Could you check the X-Ray ruby version you are using? If it is 0.11.0, that means it didn't conclude the fix. I am wondering would you like to try the fix by using master branch to see whether the issue has gone or not?

[joesiewert]

I'm getting this warning as well when using 0.11.2:

WARN -- : failed to fetch X-Ray sampling rules due to Failed to open TCP connection to 127.0.0.1:2000 (Connection refused - connect(2) for "127.0.0.1" port 2000)

[schneems]

I'm getting this warning as well when using 0.11.2:

We have a bug fix which we haven't release it yet

It sounds like you need to use master to get the fix. In your Gemfile, add:

gem 'aws-xray-sdk-ruby', github: 'aws/aws-xray-sdk-ruby'

[joesiewert]

#18 is the fix I wanted to try and from what I can tell it should be part of 0.11.2 based on the CHANGELOG.

Just to be sure I did just try installing from master as well and still see the warning.

[srprash]

I'm getting this warning as well when using 0.11.2:

WARN -- : failed to fetch X-Ray sampling rules due to Failed to open TCP connection to 127.0.0.1:2000 (Connection refused - connect(2) for "127.0.0.1" port 2000)

Hi @joesiewert ,
Can you provide more details on your application? Is it a Rails application running in docker? What is the ruby version that you're using? Could you provide your recorder and daemon configuration as well?

I'll be happy to take a look into the issue.

[rlister]

This bug appears still to be present in 0.11.4. Setting AWS_XRAY_DAEMON_ADDRESS to either form host:2000 or udp:host:2000 tcp:host:2000 is setting udp_ip and udp_port but not the tcp values. I have to set daemon_address in the config hash for tcp to be configured, otherwise it is falling back to the default:

failed to fetch X-Ray sampling rules due to Failed to open TCP connection to 127.0.0.1:2000

I must also ensure daemon_address is not set in initializers during asset:precompile or tests, as the gem tries to resolve the tcp host, and it is not available at this time (it is a kubernetes-internal service).

Running ruby:2.6.6, aws-xray-sdk:0.11.4.

[fnicola]

I never experienced this issue before while running Ruby 2.5.3 and Rails 6.0.3 with x-ray 0.11.5.
Since I upgraded to Ruby 2.7.2 and Rails 6.1 I started to experience the same issue:
WARN -- : failed to fetch X-Ray sampling rules due to Failed to open TCP connection to 127.0.0.1:2000 (Connection refused - connect(2) for "127.0.0.1" port 2000)
I tried to use master instead of the latest version, but no luck. @rlister have you managed to find a workaround?

[rlister]

Versions here are:
ruby:2.6.6
rails:6.0.3

We are successfully working around by setting daemon_address opt in addition to the env var:

opts = {  } # aws-xray-sdk options hash

if ENV['AWS_XRAY_DAEMON_ADDRESS']
  opts[:daemon_address] = ENV['AWS_XRAY_DAEMON_ADDRESS']
end

Rails.application.config.xray = opts