using the auth library with grpc fails with Error: 7 PERMISSION_DENIED

[odedpeer]

Please make sure you have searched for information in the following guides.

• Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
• Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
• Check our Troubleshooting guide: https://github.com/googleapis/google-cloud-node/blob/main/docs/troubleshooting.md
• Check our FAQ: https://github.com/googleapis/google-cloud-node/blob/main/docs/faq.md
• Check our libraries HOW-TO: https://github.com/googleapis/gax-nodejs/blob/main/client-libraries.md
• Check out our authentication guide: https://github.com/googleapis/google-auth-library-nodejs
• Check out handwritten samples for many of our APIs: https://github.com/GoogleCloudPlatform/nodejs-docs-samples

A screenshot that you have tested with "Try this API".

this works in API Explorer.
this doesn't work with the service account that requires impersonation, which I can't test in API explorer.

Link to the code that reproduces this issue. A link to a public Github Repository or gist with a minimal reproduction.

https://gist.github.com/odedpeer/0aa4ca8b2efc928cb19806c5a129a5a4

A step-by-step description of how to reproduce the issue, based on the linked reproduction.

• create a node project
• yarn add @googleapis/iam@28.0.1
• yarn add @google-cloud/logging@11.2.0
• create a service account with domain wide delegation as explained in https://developers.google.com/identity/protocols/oauth2/service-account#delegatingauthority
• set the service account email address and path to credentials file in the javascript file
• run the program

A clear and concise description of what the bug is, and what you expected to happen.

I am working on upgrading our Google SDK packages using the latest packages:

   "@googleapis/iam": "28.0.1",
   "google-auth-library": "10.1.0",
   "@google-cloud/logging": "11.2.0"

when I invoke the grpc call Logging.getEntries() from the @google-cloud/logging package it results in error
Error: 7 PERMISSION_DENIED: Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.

I expect to list all the log entries.

A clear and concise description WHY you expect this behavior, i.e., was it a recent change, there is documentation that points to this behavior, etc. **

I was able to track this down to the following regression bug:

#1938 changed the getRequestMetadataAsync function from returning a JSON to returning an object
https://github.com/googleapis/google-auth-library-nodejs/pull/1938/files#diff-320fe75112224dd3e6d983b88d752cd3b6ab866ef7fcb8723cd30975b6a5f869R955

@grpc module converts the headers to a Metadata object by using Object.keys to fill the metadata object.
this used to work with JSON objects, but it does not work with Headers since the keys are not object properties.
https://github.com/grpc/grpc-node/blob/master/packages/grpc-js/src/call-credentials.ts#L130C42-L131C29

[sofisl]

I think this is because @google-cloud/logging is still on auth 9 instead of 10 (it won't work with 10). @googleapis does work with node 10. @cindy-peng, do you have an ETA of when logging will use 10?