Consider changing the Log Cache default host

[ctlong]

Issue

CAPI currently retrieves container metric stats from Log Cache. We're in the process of moving Log Cache from the Doppler instance group to its own instance group, which necessitates a cloud_controller property change in order for CAPI to continue to pull container metrics from Log Cache.

Context

Currently, CAPI defines the default Log Cache Host as "doppler.service.cf.internal" in

• capi-release here,
• and, cloud_controller_ng here.

Our PR to move Log Cache to its own instance group also overwrites the default cc.logcache.host property with the new Log Cache host address, "log-cache.service.cf.internal", which should work fine for now.

Eventually we'd expect that the default Log Cache host should be changed to the new host address.

Steps to Reproduce

• Deploy CF from the PR referenced above
  • Do not include the bit that overwrites the cc.logcache.host property
• Push an application

Expected result

Everything works as normal

Current result

• No container metrics available and error message that includes "Stats server temporarily unavailable."
• CAPI logs on the api VM show errors related to connecting to Log Cache

/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:376:in `request_response'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:180:in `block in request_response'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:179:in `request_response'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/gem_home/ruby/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:171:in `block (3 levels) in rpc_stub_class'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/client.rb:32:in `block in container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/client.rb:49:in `with_request_error_handling'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/client.rb:31:in `container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb:49:in `get_container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb:20:in `block in container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb:19:in `loop'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/logcache/traffic_controller_decorator.rb:19:in `container_metrics'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/cloud_controller/diego/reporters/instances_stats_reporter.rb:124:in `envelopes'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/cloud_controller/diego/reporters/instances_stats_reporter.rb:23:in `stats_for_app'
/var/vcap/data/packages/cloud_controller_ng/03b9aa136a3fe19546117d3075af2e1bcd3c8878/cloud_controller_ng/lib/cloud_controller/backends/instances_reporters.rb:25:in `stats_for_app'

Possible Fix

• Change the default in new versions of capi-release and cloud_controller_ng moving forward.
• Remove the overwrite cc.logcache.host in cf-deployment

name of issue screenshot

N/A

[soroshidg]

Will this be fixed with next CAPI release? We are seeing same behavior when using cf-deployment v18+