{
"testdata_path": "my_input/qa.jsonl",
"results_dir": "my_results/experiment_refined",
"target_url": "https://app-backend-uchshrwcihlhg.azurewebsites.net/chat",
"target_parameters": {
"overrides": {
"semantic_ranker": false,
"prompt_template": "<READFILE>my_input/prompt_refined.txt"
}
}
}
diberry ➜ /workspaces/ai-rag-chat-evaluator (main) $ python3 -m scripts evaluate --config=my_config.json --numquestions=14
[22:43:56] INFO Running evaluation from config evaluate.py:207
/workspaces/ai-rag-chat-evaluator/my_config.json
INFO Replaced prompt_template in config with contents of evaluate.py:201
my_input/prompt_refined.txt
INFO Using Azure OpenAI Service with Azure Developer CLI Credential service_setup.py:27
INFO Running evaluation using data from evaluate.py:77
/workspaces/ai-rag-chat-evaluator/my_input/qa.jsonl
INFO Limiting evaluation to 14 questions evaluate.py:80
INFO Sending a test question to the target to ensure it is running... evaluate.py:83
[22:44:43] INFO Successfully received response from target for question: "What information is evaluate.py:92
in your knowledge base?"
"answer": "Our knowledge base includes information on custome..."
"context": "role_library.pdf#page=31: Responsibilities: · Mai..."
INFO Sending a test chat completion to the GPT deployment to ensure it is evaluate.py:102
running...
[22:44:46] INFO Successfully received response from GPT: "Hello! How can I assist you evaluate.py:109
today?"
INFO Starting evaluation... evaluate.py:114
[22:45:00] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:00.062907884Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: ba0a99de-13ba-4ae6-975c-b465e1195500 Correlation ID:
6906592c-47f3-4e5b-9d23-8e221afc8ea6 Timestamp: 2024-05-13 22:45:00Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[22:45:01] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:01.462735824Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: f46f2ce6-c2df-41f5-8a8b-e288cbcd5600 Correlation ID:
a866b280-de86-489b-8b32-6decb8b920a1 Timestamp: 2024-05-13 22:45:01Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[22:45:02] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:02.186514156Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 0d826a6e-4544-4706-8e05-8cd318e55c00 Correlation ID:
c68a6fef-59ca-4916-b452-012072a747e2 Timestamp: 2024-05-13 22:45:02Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:02.420259563Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 90ec62ee-fc30-447e-a097-b8883cd25800 Correlation ID:
6a064df5-461c-4dce-9354-e552ce9fa798 Timestamp: 2024-05-13 22:45:02Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:02.593558324Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 9175eb02-a11c-464e-86cb-5642c8716600 Correlation ID:
64131b49-df44-4386-8781-83bd7688020b Timestamp: 2024-05-13 22:45:02Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[22:45:05] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:05.920895603Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 9ba8e8ea-8145-4f06-b234-8ae6a7b05e00 Correlation ID:
3583cdf4-9218-47a4-aa45-af7529d9c1ed Timestamp: 2024-05-13 22:45:05Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[22:45:06] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:06.9722591Z","data":{"
message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: f3bc8a18-e5f4-47c9-b4de-4c4ad1cd5400 Correlation ID:
4ea32ae6-f7b4-464c-a01a-ce3471caa562 Timestamp: 2024-05-13 22:45:06Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[22:45:09] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:09.183899943Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 72a4f6c6-e0c3-4fa7-8e23-c7cb14444f00 Correlation ID:
0d471dd1-5f8d-4e20-8193-5181930da6b5 Timestamp: 2024-05-13 22:45:09Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[22:45:10] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:10.152382316Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: da7b4698-7ccd-4c23-a971-1fa78dbc7400 Correlation ID:
332fe3da-4f22-4244-9a08-895c734ec079 Timestamp: 2024-05-13 22:45:10Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[22:45:14] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:14.587324371Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 812f3b95-f0d4-42f6-90f3-0a6a25a45b00 Correlation ID:
82126b7c-93c2-4c8e-a82d-5d98644160e4 Timestamp: 2024-05-13 22:45:14Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:14.629979902Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 2de505a7-0a08-4b9b-a33f-01ebc94e6f00 Correlation ID:
032e19b9-352d-421e-adc5-b59f83c7ed26 Timestamp: 2024-05-13 22:45:14Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:14.68679813Z","data":{
"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 2b6d94bd-5dfa-4de5-adbf-26eef5935000 Correlation ID:
fbb61a5c-3945-4c0b-b0c7-29d2c4209b4c Timestamp: 2024-05-13 22:45:14Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[22:45:16] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:16.094273693Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 9175eb02-a11c-464e-86cb-564244746600 Correlation ID:
d516d97b-614c-4c79-b401-16ed0c2d1701 Timestamp: 2024-05-13 22:45:16Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
[22:45:20] WARNING DefaultAzureCredential failed to retrieve a token from the included chained.py:123
credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication
unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot
to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication
unavailable. The requested identity has not been assigned to this resource.
Error: Unexpected response "{'error': 'invalid_request', 'error_description':
'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication
unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential:
{"type":"consoleMessage","timestamp":"2024-05-13T22:45:20.531998191Z","data":
{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope)
AADSTS70011: The provided request must include a 'scope' input parameter. The
provided value for the input parameter 'scope' is not valid. The scope
https://cognitiveservices.azure.com/ openid offline_access profile is not
valid. The scope format is invalid. Scope must be in a valid URI form
\u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e.
Trace ID: 451a61af-cf36-4f88-9194-1536ab515800 Correlation ID:
4c65d9ed-12db-4f71-9033-3b743c28135c Timestamp: 2024-05-13 22:45:20Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here
at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/workspaces/ai-rag-chat-evaluator/scripts/__main__.py", line 6, in <module>
app()
File "/home/vscode/.local/lib/python3.11/site-packages/typer/main.py", line 326, in __call__
raise e
File "/home/vscode/.local/lib/python3.11/site-packages/typer/main.py", line 309, in __call__
return get_command(self)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/typer/core.py", line 723, in main
return _main(
^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/typer/core.py", line 193, in _main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/typer/main.py", line 692, in wrapper
return callback(**use_params)
^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/ai-rag-chat-evaluator/scripts/cli.py", line 47, in evaluate
run_evaluate_from_config(Path.cwd(), config, numquestions, targeturl)
File "/workspaces/ai-rag-chat-evaluator/scripts/evaluate.py", line 214, in run_evaluate_from_config
evaluation_run_complete = run_evaluation(
^^^^^^^^^^^^^^^
File "/workspaces/ai-rag-chat-evaluator/scripts/evaluate.py", line 153, in run_evaluation
questions_with_ratings.append(future.result())
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspaces/ai-rag-chat-evaluator/scripts/evaluate.py", line 135, in evaluate_row
result = metric.evaluator_fn(openai_config=openai_config)(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/evals/evaluators/_groundedness/_groundedness.py", line 56, in __call__
llm_output = self._flow(answer=answer, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/_sdk/entities/_flows/prompty.py", line 57, in __call__
return self._core_prompty(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/tracing/_trace.py", line 469, in wrapped
output = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/core/_flow.py", line 410, in __call__
response = send_request_to_llm(api_client, self._model.api, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/core/_prompty_utils.py", line 164, in send_request_to_llm
result = client.chat.completions.create(**parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/openai/_utils/_utils.py", line 277, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 590, in create
return self._post(
^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/openai/_base_client.py", line 1240, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/openai/_base_client.py", line 921, in request
return self._request(
^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/openai/_base_client.py", line 939, in _request
self._prepare_options(options)
File "/home/vscode/.local/lib/python3.11/site-packages/openai/lib/azure.py", line 288, in _prepare_options
azure_ad_token = self._get_azure_ad_token()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/openai/lib/azure.py", line 274, in _get_azure_ad_token
token = provider()
^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/core/_connection.py", line 247, in get_token
return self._token_provider.get_token()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/promptflow/_core/token_provider.py", line 52, in get_token
return self.credential.get_token(audience).token
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/azure/identity/_credentials/default.py", line 225, in get_token
token = super().get_token(*scopes, claims=claims, tenant_id=tenant_id, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.local/lib/python3.11/site-packages/azure/identity/_credentials/chained.py", line 124, in get_token
raise ClientAuthenticationError(message=message)
azure.core.exceptions.ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource. Error: Unexpected response "{'error': 'invalid_request', 'error_description': 'Identity not found'}"
SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
AzureDeveloperCliCredential: {"type":"consoleMessage","timestamp":"2024-05-13T22:45:02.593558324Z","data":{"message":"\nERROR: fetching token: failed to authenticate:\n(invalid_scope) AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope https://cognitiveservices.azure.com/ openid offline_access profile is not valid. The scope format is invalid. Scope must be in a valid URI form \u003chttps://example/scope\u003e or a valid Guid \u003cguid/scope\u003e. Trace ID: 9175eb02-a11c-464e-86cb-5642c8716600 Correlation ID: 64131b49-df44-4386-8781-83bd7688020b Timestamp: 2024-05-13 22:45:02Z\n\n"}}
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
This issue is for a: (mark with an
x)Minimal steps to reproduce
python3 -m scripts generate --output=my_input/qa.jsonl --numquestions=14 --persource=2python3 -m scripts evaluate --config=my_config.json --numquestions=14with the following my_config.jsonazd config set auth.useAzCliAuth "true"thenazd auth loginagain and az cli isn't in repo.Any log messages given by the failure
Expected/desired behavior
OS and Version?
Versions
Mention any other details that might be useful