tools: incomplete dataset parameter specification by user returns 404 error

[annav-gov]

Describe the bug
If the input for the dataset parameter in the fetch-dataset-geography tool is incomplete (based on the information that the user provided in the prompt), then the tool/API will return a 404 error.

To Reproduce
Steps to reproduce the behavior:

1. Input prompt: "What levels of geography are available for 2023 planning database (PDB)?"
2. Client correctly identifies that it needs to use fetch-dataset-geography tool and identifies the inputs as {"dataset": "pdb", "year": 2023}.
3. The above returns a 404 error, because dataset should be either "pdb/tract" or "pdb/blockgroup."

Expected behavior
I would like the tool to return an error that say that the dataset "pdb" doesn't exist AND suggest what the user may have meant. For example, for client to ask if the user meant "pdb/tract" or "pdb/blockgroup" and continue the interaction with the updated information.

The challenge here will be to know if the dataset parameter or the year parameter failed (the latter meaning the dataset doesn't exist for that year).

Screenshots

Desktop:

• OS: iOS Sequoia 15.5
• IDE: VSCode 1.101.0
• Client: Github Copilot
• Model Version: GPT4.1
• Latest repo commit: 5475422

Additional context
In the API, dataset can have more than one "description/identifier" (for lack of a better term). For example, /cbp/ has one, acs/acs1 has two, and cps/basic/dec has three.

[annav-gov]

This issue also will likely apply to the fetch-aggregate-data tool, because it requires the same dataset parameter.

[annav-gov]

Also, saw the same issue with the describe-dataset tool.

[annav-gov]

@luke-keller-census, is this the intended behavior if the dataset parameter isn't specified correctly? This is now what's being returned instead of the 404 error. The dataset in the input isn't correct (prompt doesn't provide enough information to specify a CPS dataset correctly).

Prompt:
Can you fetch the metadata for the Current Population Survey 2021?

Input (for mcp tool):
{
"dataset": "cps",
"year": 2021
}

Response:
Response validation failed: [
{
"code": "invalid_type",
"expected": "string",
"received": "undefined",
"path": [
"@context"
],
"message": "Required"
},
{
"code": "invalid_literal",
"expected": "DatasetMetadata",
"path": [
"@type"
],
"message": "Invalid literal value, expected "DatasetMetadata""
},
{
"code": "invalid_type",
"expected": "object",
"received": "undefined",
"path": [
"dataset"
],
"message": "Required"
}
]

[luke-keller-census]

These are different issues. I think your last issue pertains to a validation error that is unique to the describe-dataset tool and this dataset, since the API endpoint this constructs should be correct (unlike your other prompts)

[luke-keller-census]

I think the solution is to create a dataset-search-service of some kind that takes the data listed in api.data.census.gov and does the following:

• Fuzzy matches the input string to a dataset, perhaps making some assumptions about the most recent year if no year included
• Provides basic information about the table, e.g., name, survey, survey description
• Provides an example to call the correct tool to fetch a table from it with valid arguments

[annav-gov]

I'm curious why that validation is unique to describe-dataset. If other tools also have the same dataset parameter, shouldn't they also have the validation? I expected describe-dataset to also return a 404 error (like fetch-dataset-geography).

To clarify, the api endpoint shouldn't return anything because "cps" isn't a valid dataset specification (which is why I was expecting a 404 error).