add: Implement a new task for Turkish Thinking datasets

[aleynahukmet]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @aleynahukmet, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new TurkishThinking task, aimed at facilitating the evaluation of generative models using Turkish Chain of Thought datasets. It streamlines the process by combining two existing datasets into a single, shuffled source, and establishes a clear, exact-match evaluation metric for assessing model performance on these conversational examples.

Highlights

• New Task Implementation: A new task named TurkishThinking has been added, extending the base Task class for evaluating generative models.
• Dataset Integration: This task integrates and concatenates two Turkish Chain of Thought datasets, altaidevorg/gemini-turkish-thinking and altaidevorg/gemini-turkish-thinking-v2, providing a unified dataset for evaluation.
• Generative Evaluation: The task is configured for generative evaluation, where the assistant's response is directly compared against a ground truth message for correctness.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new task for Turkish Thinking datasets. The implementation correctly loads and combines two datasets from Hugging Face. The task is set up as a generative task. My review includes a minor code style fix and a point for consideration regarding the evaluation logic, which currently uses a strict exact-match that might be too brittle for a generative reasoning task. Overall, the changes are clear and well-structured.

[gemini-code-assist]

There is a trailing whitespace at the end of this line. It's good practice to remove it to maintain code cleanliness.

Suggested change

	assistant_message = row['assistant']
	assistant_message = row['assistant']

[gemini-code-assist]

The current evaluation uses a strict string comparison between the generated response and the ground truth. For a generative 'Chain of Thought' task, this can be very brittle, as a slightly different (but still correct) phrasing of the reasoning would be marked as incorrect. Have you considered a more flexible evaluation method? For example, if applicable, you could extract the final answer from the reasoning chain and compare that, similar to how the GSM8K task is evaluated. If an exact match is intended, it might be worth adding a comment to explain why.