Add eval recipe for multiturn chat

Author: dkashkin

genai-on-vertex-ai/gemini/model_upgrades/multiturn_chat/promptfoo/README.md

@@ -0,0 +1,55 @@
+# Multi-Turn Conversation (Chat)
+### _Eval Recipe for model migration_
+
+This Eval Recipe demonstrates how to evaluate a multi-turn conversation (chat) on Gemini 1.0 and Gemini 2.0 using the open source evaluation tool [Promptfoo](https://www.promptfoo.dev/).
+
+![](diagram.png "Promptfoo")
+
+- Use case: multi-turn conversation
+
+- Evaluation Dataset is based on [Multi-turn Prompts Dataset](https://www.kaggle.com/datasets/softageai/multi-turn-prompts-dataset). It includes 5 conversations: [`dataset.jsonl`](./dataset.jsonl). Each record in this file links to a JSON file with the system instruction followed by a few messages from the User and responses from the Assistant. This dataset does not include any ground truth labels.
+
+- Prompt Template located in [`prompt_template.txt`](./prompt_template.txt) injects the `chat` variable from our dataset which represents the conversation history.
+
+- [`promptfooconfig.yaml`](./promptfooconfig.yaml) contains all Promptfoo configuration:
+    - `providers`: list of models that will be evaluated
+    - `prompts`: location of the prompt template file
+    - `tests`: location of the dataset file
+    - `defaultTest`: configures the evaluation metric:
+        1. `type: select-best` auto-rater that decides which of the two models configured above generated the best response.
+        1. `providers` configures the judge model
+        1. `value` configures the custom criteria that is evaluated by the `select-best` auto-rater
+
+## How to run this Eval Recipe
+
+- Google Cloud Shell is the easiest option as it automatically clones our Github repo:
+
+    <a href="https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/applied-ai-engineering-samples&cloudshell_git_branch=main&cloudshell_workspace=genai-on-vertex-ai/gemini/model_upgrades">
+        <img alt="Open in Cloud Shell" src="http://gstatic.com/cloudssh/images/open-btn.png">
+    </a>
+
+- Alternatively, you can use the following command to clone this repo to any Linux environment with configured [Google Cloud Environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment):
+
+    ``` bash
+    git clone --filter=blob:none --sparse https://github.com/GoogleCloudPlatform/applied-ai-engineering-samples.git && \
+    cd applied-ai-engineering-samples && \
+    git sparse-checkout init && \
+    git sparse-checkout set genai-on-vertex-ai/gemini/model_upgrades && \
+    git pull origin main
+    cd genai-on-vertex-ai/gemini/model_upgrades
+    ```
+
+1. Install Promptfoo using [these instructions](https://www.promptfoo.dev/docs/installation/).
+1. Navigate to the Eval Recipe directory in terminal and run the command `promptfoo eval`.
+
+    ``` bash
+    cd genai-on-vertex-ai/gemini/model_upgrades/multiturn_chat/promptfoo
+    promptfoo eval
+    ```
+1. Run `promptfoo view` to analyze the eval results. You can switch the Display option to `Show failures only` in order to investigate any underperforming prompts.
+
+## How to customize this Eval Recipe:
+1. Copy the configuration file `promptfooconfig.yaml` to a new folder.
+1. Add your labeled dataset file with JSONL schema similar to `dataset.jsonl`. 
+1. Save your prompt template to `prompt_template.txt` and make sure that the template variables map to the variables defined in your dataset.
+1. That's it! You are ready to run `promptfoo eval`. If needed, add alternative prompt templates or additional metrics to promptfooconfig.yaml as explained [here](https://www.promptfoo.dev/docs/configuration/parameters/).

genai-on-vertex-ai/gemini/model_upgrades/multiturn_chat/promptfoo/chats/chat_01.json

@@ -0,0 +1,34 @@
+[
+  {
+    "role": "system",
+    "content": "You are a helpful AI assistant. Your goal is to provide accurate and conscise answers to user questions."
+  },
+  {
+    "role": "user",
+    "content": "write a short guide for a cooking video about making spaghetti carbonara. keep it short, around 400 words. explain how to cook pasta just right, make pancetta crispy, and mix eggs into a creamy sauce without messing up. share tips on adding salt and pepper, and topping with cheese and parsley. say why timing and skill are important for a smooth sauce. tell people to try making it at home and enjoy the tasty italian dish."
+  },
+  {
+    "role": "assistant",
+    "content": "Title: Spaghetti Carbonara - A Creamy Italian Delight\nHello, and welcome to our cooking video on Spaghetti Carbonara! Today, we'll guide you through the steps to create a smooth, creamy sauce and crispy pancetta that will make your taste buds dance.\nLet's start with the pasta. Cook your spaghetti al dente, which means it should be firm but not hard in the center. To achieve this, follow the package instructions, but reduce the cooking time by 1-2 minutes. Once cooked, drain the pasta, reserving some pasta water for later.\nNow, let's move on to the pancetta. Crispy pancetta adds a delightful crunch to the dish. Cut it into small cubes and sauté in a hot pan with a little oil until it becomes golden brown and crispy. Remove it from the pan and set it aside.\nThe key to a creamy carbonara sauce is the perfect mix of eggs and cheese. In a separate bowl, whisk together 2 eggs, 1/2 cup grated Parmesan cheese, and a pinch of black pepper. This mixture will thicken and create a silky sauce when combined with the hot pasta.\nTiming and skill are crucial for a smooth sauce. Here's how to do it:\n1. Remove the pan with the pancetta fat from the heat.\n2. Add the cooked spaghetti to the pan and toss it in the pancetta fat to coat the pasta.\n3. Slowly pour the egg and cheese mixture over the pasta, stirring constantly to prevent the eggs from scrambling.\n4. If the sauce is too thick, add reserved pasta water, a tablespoon at a time, until you reach your desired consistency.\nSeason the dish with salt and freshly ground black pepper to taste. Remember, the pancetta and Parmesan cheese are already salty, so go easy on the salt.\nTo finish, top your Spaghetti Carbonara with crispy pancetta, a sprinkle of grated Parmesan cheese, and a handful of chopped parsley.\nNow, it's your turn to try making this delicious Italian dish at home. With a little practice, you'll master the art of creating a creamy sauce and crispy pancetta. Enjoy your cooking journey and the delightful flavors of Spaghetti Carbonara!"
+  },
+  {
+    "role": "user",
+    "content": "give steps on how a person cooking for the 1st time make sure that the pasta is cooked to perfection without over or undercooking it, maintaining its ideal texture for the spaghetti carbonara"
+  },
+  {
+    "role": "assistant",
+    "content": "To ensure that the pasta is cooked perfectly for your Spaghetti Carbonara, especially if you're cooking for the first time, follow these steps:\n1. **Boil the Water**: Start by bringing a pot of water to a rolling boil. Use a generous amount of salt in the water to season the pasta as it cooks.\n2. **Add the Pasta**: Once the water is boiling, add the spaghetti to the pot. Stir it initially to prevent sticking and ensure even cooking.\n3. **Check for Doneness**: After about 6-7 minutes of boiling, start checking the pasta for doneness. Use tongs to remove a strand of spaghetti, run it under cold water, and taste it. The pasta should be al dente, meaning it should be firm to the bite but not raw or hard in the center.\n4. **Test the Texture**: Al dente pasta should be resilient and snappy, not spongy or overcooked. Keep tasting and checking until you achieve the ideal texture.\n5. **Transfer to Sauce**: Once the pasta is al dente, transfer it directly from the pot to the sauce without draining. This helps retain some pasta water, which can be used to adjust the sauce consistency later.\nBy following these steps and mastering the art of cooking pasta to perfection, you'll ensure that your Spaghetti Carbonara turns out delicious with the ideal texture and flavor. Enjoy your cooking experience and savor the delightful Italian dish!\nCitations:\n[1] https://damndelicious.net/2014/03/29/spaghetti-carbonara/\n[2] https://www.amateurgourmet.com/blog/2008/07/spaghetti_carbo.html\n[3] https://www.alisoneroman.com/recipes/spaghetti-carbonara\n[4] https://www.simplyrecipes.com/recipes/spaghetti_alla_carbonara/\n[5] https://cooking.nytimes.com/recipes/12965-spaghetti-carbonara"
+  },
+  {
+    "role": "user",
+    "content": "what considerations should be made when selecting and adding cheese and parsley toppings to complement the flavors of the dish without overpowering them with spcies"
+  },
+  {
+    "role": "assistant",
+    "content": "When selecting and adding cheese and parsley toppings for Spaghetti Carbonara, consider the following:\n1. **Cheese**: Freshly grated Parmesan or Pecorino Romano cheese is recommended for the best flavor[1][2][3][4][5]. Pecorino Romano has a slightly stronger flavor than Parmesan, so choose based on personal preference[3]. Using a mix of both can also be an option[1]. Avoid using pre-shredded cheese, as it often contains additives that can affect the texture and taste of the sauce[3].\n2. **Parsley**: Fresh parsley is essential for Spaghetti Carbonara, as dried parsley does not provide the same level of flavor and freshness[2][3]. Chop the parsley finely to distribute it evenly as a garnish[2].\n3. **Complementing Flavors**: Cheese and parsley should complement the dish without overpowering it. The cheese adds a rich, salty flavor, while the parsley adds a fresh, herbal note[1][3]. Add the cheese and parsley as a garnish just before serving to maintain their freshness and flavors[1][3].\n4. **Seasoning**: When adding cheese and parsley, be mindful of the overall seasoning of the dish. The pancetta and cheese are already salty, so you may not need to add extra salt[1][3]. Taste the dish before adding any additional seasoning and adjust accordingly.\n5. **Timing**: Add the cheese and parsley at the right time to ensure they maintain their flavors and textures. Stir in the cheese with the egg mixture, and add the parsley as a garnish just before serving[1][3].\nBy considering these factors, you can ensure that the cheese and parsley toppings complement the flavors of the Spaghetti Carbonara without overpowering them.\nCitations:\n[1] https://www.savorynothings.com/the-best-spaghetti-alla-carbonara/\n[2] https://www.cookingtothepoint.com/recipes/spaghetti-carbonara\n[3] https://oldcutkitchen.com/2020/04/06/spaghetti-carbonara/\n[4] https://damndelicious.net/2014/03/29/spaghetti-carbonara/\n[5] https://www.simplyrecipes.com/recipes/spaghetti_alla_carbonara/"
+  },
+  {
+    "role": "user",
+    "content": "what strategies can be done to effectively season the dish with salt and pepper, balancing the flavors to enhance the overall taste of the spaghetti carbonara"
+  }
+]

genai-on-vertex-ai/gemini/model_upgrades/multiturn_chat/promptfoo/chats/chat_02.json

@@ -0,0 +1,34 @@
+[
+  {
+    "role": "system",
+    "content": "You are a helpful AI assistant. Your goal is to provide accurate and conscise answers to user questions."
+  },
+  {
+    "role": "user",
+    "content": "planning a family trip to Maui, Hawaii, on July 2024 and want to stay in a nice beach villa with a great view of the ocean from where we can see sunset. We're a family of five, with two kids aged 6 and 9, and my two wife and I. We're looking for a place that's comfortable for all of us and fits our budget of about $800 per night. villa must have relaxed tropical vibe, big living spaces, and private outdoor areas like balconies and lanais where we can enjoy the ocean views and villa should have easy access to the beach and give things like beach chairs, umbrellas, and snorkeling gear while offering activities like paddleboarding, snorkeling tours, and beach bonfires that the whole family can enjoy recommend 4 luxury beachfront villas in Maui that meet these criteria, and give details on their amenities, proximity to the beach, and any unique features like private pools and outdoor barbecue areas."
+  },
+  {
+    "role": "assistant",
+    "content": "Based on the search results, here are four luxury beachfront villas in Maui that meet your criteria:\n\n1. **Hyatt Residence Club Maui**: This luxurious resort offers one, two, and three-bedroom villas with expansive private lanais, gourmet kitchens, and modern furnishings. The resort is oceanfront on Ka'anapali Beach and has a kid's pool, waterslide, fitness center, and infinity pool. The two-bedroom villa is 1,200 square feet, and the three-bedroom villa is 2,100 square feet, making them perfect for large families. The resort is adjacent to the Hyatt Regency Maui Resort & Spa, which offers great pools and a beautiful property. The average rate per night is around $1,200.\n\nAmenities:\n- Gourmet kitchens\n- Private lanais\n- Kid's pool\n- Waterslide\n- Fitness center\n- Infinity pool\n- Adjacent to the Hyatt Regency Maui Resort & Spa\n\n2. **Wailea Beach Resort**: This family paradise combines elegance with many kid-friendly amenities. The vibe is upscale yet welcoming, making it perfect for families with teens. The resort has an adult-only pool, a 325-foot waterslide, a splash pad, a game room, and a movie theater with free popcorn and lemonade. The resort is oceanfront and has three restaurants, including a food truck.\n\nAmenities:\n- 325-foot waterslide\n- Splash pad\n- Game room\n- Movie theater\n- Three restaurants\n- Oceanfront\n\n3. **Fairmont Kea Lani**: This all-suite and villa luxury resort is Hawai'i's only one, with the most generous, handsomely appointed, and well-designed accommodations on the islands. The resort is oceanfront on Polo Beach and has a kid's pool, waterslide, and fitness center. The spacious suites and luxury beach villas cater to families large or small, with a compact fridge, microwave, and in-suite pantry.\n\nAmenities:\n- Gourmet kitchens\n- Private lanais\n- Kid's pool\n- Waterslide\n- Fitness center\n- Oceanfront\n\n4. **Andaz Maui at Wailea Resort**: This beachfront resort offers luxury accommodations with private lanais, gourmet kitchens, and modern furnishings. The resort has a lagoon-style pool, a tranquility pool, and a fitness center. The resort also offers a kids' club, water sports, and cultural activities.\n\nAmenities:\n- Gourmet kitchens\n- Private lanais\n- Lagoon-style pool\n- Tranquility pool\n- Fitness center\n- Kids' club\n- Water sports\n- Cultural activities\n\nThese four luxury beachfront villas in Maui offer comfortable accommodations, great views, and plenty of amenities for families. They are all oceanfront and offer easy access to the beach, making them perfect for watching the sunset. The resorts also offer activities like paddleboarding, snorkeling tours, and beach bonfires that the whole family can enjoy.\n\nCitations:\n[1] https://www.outrigger.com/landing-pages/feature-page/maui-family-resorts\n[2] https://www.mommyneedsamaitai.com/best-places-to-stay-on-maui-for-families/\n[3] https://thehawaiivacationguide.com/maui-resorts-for-families/\n[4] https://www.melissastraveladdiction.com/travel/where-your-family-should-stay-in-maui-on-a-budget\n[5] https://www.fairmont-kea-lani.com/experience/family-travel/"
+  },
+  {
+    "role": "user",
+    "content": "What transportation options are available to reach these luxury beachfront villas from Maui's major airports"
+  },
+  {
+    "role": "assistant",
+    "content": "There are several transportation options available to reach the luxury beachfront villas from Maui's major airports.\n\n1. **Hyatt Residence Club Maui**: The Kahului Airport (OGG) is located 19 miles from the community, making it ideal for those who wish to take advantage of ROAM's convenient flight services[1]. Alternatively, you can rent a car from Avis Car Rental at the Fairmont Kea Lani, which is located 17 miles from the airport[2].\n2. **Fairmont Kea Lani**: The Kahului Airport (OGG) is located 17 miles (27 km) from the Fairmont Kea Lani, Maui hotel. Travel time is approximately 30 minutes. Transportation options include renting a car from Avis Car Rental at the hotel or arranging for airport transfer through the hotel's concierge[2].\n3. **Private Paradise Villas at Wailea Beach Villas**: The Wailea Beach Villas are situated on the beach in the heart of Wailea along Maui's sunny southwestern shoreline. You can arrange for airport transfers, yoga coaches, and fitness trainers through the Private Paradise Villas Collection[3].\n4. **Sea La Vie L510 at Wailea Beach Villas**: The Wailea Beach Villas are located in the heart of Wailea along Maui's sunny southwestern shoreline. You can arrange for airport transfers, yoga coaches, and fitness trainers through the Private Paradise Villas Collection[3].\n5. **Kaanapali Villas**: The Kahului Airport (OGG) is located approximately 27 miles from Kaanapali, and the drive takes around 45 minutes. You can rent a car from Avis Car Rental at the airport or arrange for airport transfer through the villa's management company[4].\n\nIt's important to note that rental cars are widely available and are a convenient way to explore Maui's many attractions at your own pace. However, if you prefer not to drive, you can arrange for airport transfer or private transportation through the hotel or villa's concierge.\n\nCitations:\n[1] https://www.mauieliteproperty.com/blog/how-roam-maui-is-redefining-luxury-travel-to-hawaii.html\n[2] https://www.fairmont.com/kea-lani-maui/\n[3] https://www.privateparadisevillas.com/sea-la-vie-wailea-beach-villas.aspx\n[4] https://www.vrbo.com/181046\n[5] https://www.fourseasons.com/maui/- role: user"
+  },
+  {
+    "role": "user",
+    "content": "give me info on the dining options available within and nearby these villa complexes, especially regarding family-friendly menus and dining experiences?"
+  },
+  {
+    "role": "assistant",
+    "content": "The dining options available within and nearby these villa complexes are as follows:\n\n1. **Kaanapali Villas**: The villa complex is located near several dining options, including Maui Brewing, a great coffee shop, and fun food options[2]. The villa itself offers a tropical cocktail bar and an extensive wine list[1].\n2. **Kahana Village**: The villa complex is located near several dining options, including a great coffee shop, and fun food options[2]. The villa itself does not offer on-site dining, but the location is convenient to several nearby restaurants and cafes.\n3. **Fairmont Kea Lani**: The villa complex offers several dining options, including the Kea Lani Restaurant, which features an alfresco dining room that shows off the beauty of Wailea, Maui[3]. The resort also features the AMA Bar & Grill, which offers casual local favorites like sushi, salads, fish sandwiches, and a five-farm burger for a relaxing poolside lunch[3]. Additionally, the resort offers the MAKANA Market + Cafe, which features a variety of grab-and-go options, as well as a coffee shop[3].\n4. **Sea La Vie L510 at Wailea Beach Villas**: The villa complex is located near several dining options, including the Shops of Wailea, The Grand Wailea Resort and Spa, and The Four Seasons Resort, where you will find exceptional fine dining restaurants, fabulous shopping, pampering spas/salons, and evening entertainment[5]. Additionally, the villa offers in-villa dining options, including private chefs and catering services[5].\n\nAll of these villa complexes offer family-friendly dining options, with menus that cater to a variety of tastes and dietary restrictions. The Fairmont Kea Lani, in particular, offers a kids' menu and high chairs, making it a great option for families with young children[3]. Additionally, many of the nearby restaurants and cafes offer kid-friendly options, such as burgers, pizza, and pasta.\n\nCitations:\n[1] https://www.vrbo.com/181046\n[2] https://www.kahanavillage.com\n[3] https://www.fairmont-kea-lani.com/experience/family-travel/\n[4] https://www.outrigger.com/landing-pages/feature-page/maui-family-resorts\n[5] https://www.privateparadisevillas.com/sea-la-vie-wailea-beach-villas.aspx"
+  },
+  {
+    "role": "user",
+    "content": "Are there any nearby attractions and activities suitable for families with young children that complement the beachside experience offered by these villas."
+  }
+]