-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathUzoigwe_Chiderah_NLP_Project.py
More file actions
342 lines (240 loc) · 121 KB
/
Copy pathUzoigwe_Chiderah_NLP_Project.py
File metadata and controls
342 lines (240 loc) · 121 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
# -*- coding: utf-8 -*-
"""NLP_Project__Learner_Notebook.ipynb
Automatically generated by Colaboratory.
Original file is located at
https://colab.research.google.com/drive/11SMdLc6jZjjFvg4rs1SlNg5schW7jyfV
### **Business Context**
In today's dynamic business landscape, organizations are increasingly recognizing the pivotal role customer feedback plays in shaping the trajectory of their products and services. The ability to **swiftly and effectively respond to customer input** not only fosters enhanced customer experiences but also serves as a catalyst for growth, prolonged customer engagement, and the nurturing of lifetime value relationships.
As a dedicated Product Manager or Product Analyst, staying attuned to the voice of your customers is not just a best practice; it's a strategic imperative.
While your organization may be inundated with a wealth of customer-generated feedback and support tickets, your role entails much more than just processing these inputs. To make your efforts in managing customer experience and expectations truly impactful, you need a structured approach – a method that allows you to discern the most pressing issues, set priorities, and allocate resources judiciously.
One of the most effective strategies at your disposal as an organization is to harness the power of automated Support Ticket Categorization - **done in the modern day using Large Language Models and Generative AI.**
### **Objective**
Develop a Generative AI application using a Large Language Model to **automate the classification and processing of support tickets.** The application will aim to predict ticket categories, assign priority, suggest estimated resolution times, generate responses based on sentiment analysis, and store the results in a structured DataFrame.
### **Sample of Expected Output**
### **Please read the instructions carefully before starting the project.**
This is a commented Python Notebook file in which all the instructions and tasks to be performed are mentioned.
* Blanks '_______' are provided in the notebook that
needs to be filled with an appropriate code to get the correct result. With every '_______' blank, there is a comment that briefly describes what needs to be filled in the blank space.
* Identify the task to be performed correctly, and only then proceed to write the required code.
* Please run the codes in a sequential manner from the beginning to avoid any unnecessary errors.
* Add the results/observations (wherever mentioned) derived from the analysis in the presentation and submit the same. Any mathematical or computational details which are a graded part of the project can be included in the Appendix section of the presentation.
### **Installing Necessary Libraries and Dependencies**
"""
import warnings
warnings.filterwarnings("ignore") #This will ignore all warnings for the rest of the notebook
# Installation for GPU llama-cpp-python
!CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir -q
# Install the hugging face hub
!pip install huggingface_hub -q
#The command above installs the Hugging Face Hub library, which is a cloud based platform facilitating the sharing and discovery of pre-trained models,
#datasets, and other resources in the field of natural language processing (NLP) as well as an inference API for running them.
# The -q flag ensures a quiet installation without displaying verbose output.
"""### **Q1: Write Python code that imports the 'hf_hub_download' function from the 'huggingface_hub' library and also imports the 'Llama' class from the 'llama_cpp' library.**
**Ensure that the code is correctly formatted and ready for execution.**
"""
# complete the code to import the 'hf_hub_download' function from the 'huggingface_hub' library
from huggingface_hub import hf_hub_download
#The hf_hub_download() function is the main function for downloading files from the Hugging face Hub.
#It downloads the remote file, caches it on disk (in a version-aware way), and returns its local file path.
!pip install llama-cpp-python
#llama-cpp-python is a Python binding for llama.cpp. It supports inference for many LLMs models, which can be accessed on Hugging Face.
# complete the code to import the 'Llama' class from the 'llama_cpp' library
from llama_cpp import Llama
#A state-of-the-art foundational large language model designed to help researchers advance their work in NLP, it is an autoregressive large language models
# defining the model name or path as a string (You can find this info from hugging face website)
model_name_or_path = "TheBloke/Llama-2-13B-chat-GGUF"
# defining the model basename as a string, indicating it's in the gguf format
model_basename = "llama-2-13b-chat.Q5_K_M.gguf"
# downloading the model from the Hugging Face Hub using the 'hf_hub_download' function by specifying the 'repo_id' and 'filename'
model_path = hf_hub_download(
repo_id =model_name_or_path, # complete the code to mentioned the repo_id
filename =model_basename # complete the code to mentioned the filename
)
# complete the code to create an instance of the 'Llama' class with specified parameters
lcpp_llm = Llama(
model_path=model_path,
n_threads=2, # CPU cores
n_batch=512, # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU.
n_gpu_layers=43, # Change this value based on your model and your GPU VRAM pool.
n_ctx=4096, # Context window
)
#This above code initializes a Llama model instance (lcpp_llm) with specific configuration parameters,
#including model path, CPU threads, GPU batch size, GPU layers,
# and context window size. These parameters tailor the model's behavior and resource utilization.
"""**Comment:**
The flags generated are typically used to optimize the performance of the code based on the hardware capabilities. For instance, enabling AVX, AVX2, or AVX512 flags leverages advanced vector instructions, which can significantly accelerate certain computations on supported hardware.
### **Q2: Define the System Message**
Write a Python function called **generate_llama_response** that takes a single parameter, support_ticket_text, which represents the user's support ticket text. Inside the function, you should perform the following tasks:
Define a system message as a string and assign it to the variable system_message.
- **Combine the support_ticket_text and system_message to create a prompt string.**
*Generate a response from the LLaMA model using the lcpp_llm instance with the following parameters:*
- prompt should be the combined prompt string.
- max_tokens should be set to 256.
- temperature should be set to 0.
- top_p should be set to 0.95.
- repeat_penalty should be set to 1.2.
- top_k should be set to 50.
- stop should be set as a list containing 'INST'.
- echo should be set to False.
Extract and return the response text from the generated response.
Don't forget to provide a value for the system_message variable before using it in the function.
*What content and instructions should be included in the system message to guide the technical assistant when processing support tickets? Please provide a detailed description of the information and guidelines that the system message should contain.*
Here's a detailed breakdown of what should be included:
- **Introduction (System Role):** Begin with an introductory statement that establishes the role of the system message. In this case, it's acting as a guide for a technical assistant.
- **Ticket Categorization:** Explain the primary task of the technical assistant, which is to classify the support ticket into specific categories. In this example, the categories are:
- Technical Issues
- Hardware Issues
- Data Recovery
- **Response Options:** Clearly state that the assistant should only respond with one of the predefined categories, emphasizing that other responses are not acceptable.
- **Sub-Tasks:** Outline the secondary tasks that the technical assistant should perform once the category is identified. These sub-tasks include:
- **Creating Tags:** Instruct the assistant to create tags that will help further classify the ticket.
- **Assigning Priority:** Specify that the assistant should assign a priority level (e.g., "High" or "Normal") based on their understanding of the text.
- **Suggesting ETA:** Guide the assistant to provide an estimated time for
resolving the issue mentioned in the ticket.
- **General Instructions:** Offer general instructions that should be followed throughout the ticket processing, such as:
- **Categorization:** Reiterate that the assistant should categorize the ticket only into the predefined categories.
- **Reading Carefully:** Stress the importance of reading the support ticket text thoroughly and considering the overall sentiment before assigning priority.
- **Output Format:** Clearly specify the desired output format for the responses generated by the assistant. In this case, the output should be in JSON format.
**The output of the model should be in JSON format**
"""
def generate_llama_response(support_ticket_text):
# System message
system_message = """
[INST]<<SYS>>
As a technical assistant, your primary role is to categorize the support ticket into specific categories which are:
1. Technical Issues
2. Hardware Issues
3. Data Recovery
Please respond with only one of the predefined categories. Do not provide any other responses.
Once you've identified the category, perform the following sub-tasks:
- Create tags that further classify the ticket.
- Assign a priority level (e.g., "High" or "Normal") based on your understanding of the text.
- Provide an estimated time for resolving the mentioned issue.
General Instructions:
- Categorize the ticket only into the predefined categories mentioned above.
- Read the support ticket text thoroughly and consider the overall sentiment before assigning priority.
Output Format:
- Respond only in JSON format and no other formart like the text format is not needed, no explanation, just extract the categories and sub-task
and put them in a JSON Object
The output should be in the form of a JSON
For Categories
1. Technical Issues (as a string)
2. Hardware Issues (as a string)
3. Data Recovery (as a string)
for sub-tasks
1. tags (as a string)
2. priority (as a string)
3. estimated time (as a string)
for example this: { "category": "Data Recovery", "subtasks": { "tags": ["External Hard Drive", "Not Recognized"], "priority": "High", "estimated_time": "2-3 business days" } }
i dont need anything like "Sure, I'd be happy to help! Based on the information provided in the support ticket, I have categorized it as follows: "
or like this "Sure, I'd be happy to help! Based on the information provided, I would categorize this support ticket as "Data Recovery". Here are the sub-tasks and tags for this category: Tags: data recovery, water damage, laptop Priority: High Estimated time: 2-3 business days Please let me know if you have any further questions or concerns."
i need pure JSON object
Thank you for your assistance!
<</SYS>>[/INST]
"""
# Combine user_prompt and system_message to create the prompt
prompt = f"{support_ticket_text}\n{system_message}"
# Generate a response from the LLaMA model
response = lcpp_llm(
prompt=prompt,
max_tokens=256, # complete the code to set the max tokens to generate
temperature=0, # complete the code to set the temperature (between 0 and 1)
top_p=0.95,
repeat_penalty=1.2,
top_k=50,
stop=['INST'],
echo=False
)
# Extract and return the response text
response_text = response["choices"][0]["text"]
return response_text
"""**Comment:** This above function, generate_llama_response, takes a support ticket text as input and generates a response using the LLaMA model. The generated response includes a system message with instructions for categorizing the support ticket and performing sub-tasks. The function then extracts and returns the response text in a JSON format that includes categories and sub-task details, as specified in the system message. The code is designed to provide a concise JSON output without additional explanations.
### **Q3: Loading the Dataset**
"""
# Import the pandas library and alias it as 'pd'
import pandas as pd
# Mount Google Drive
from google.colab import drive
drive.mount('/content/drive')
# Read the CSV file into a DataFrame and store it in the 'data' variable
df = pd.read_csv('/content/drive/MyDrive/Support_ticket_text_data_mid_term.csv')
"""### **Data Overview**"""
# Check the first 5 rows of the data
df.head()
# Check the shape of the data
df.shape
"""**Comment**:There are 21 rows and 2 columns in the dataset"""
# Check for missing values in the data
df.isnull().sum(axis=0)
"""**Comment:** There are no missing values in the dataset
### **Q4: Create a new column in the DataFrame called 'llama_response' and populate it with responses generated by applying the 'generate_llama_response' function to each 'support_ticket_text' in the DataFrame**
"""
# creating a copy of the dataset
data = df.copy()
# complete the code to create a new column llama_response'
# by applying the function to each element in the 'support_ticket_text' column of the DataFrame 'data'
data['llama_response'] = data['support_ticket_text'].apply(lambda x: generate_llama_response(x))
"""**Comment:**
The message "Llama.generate: prefix-match hit" suggests that during text generation using the Llama model, a successful match with a specified prefix has been identified. This indicates that the model has recognized and responded to the input pattern.
"""
# Check the first five rows of the data to confirm whether the new column has been added
data.head()
"""**Comment:** A new coloumn llama_response has been added to the dataframe, after applying the generate_llama_response function to the support ticket text, which is the raw model response.
### **Q5: Prepare the dataset in the desired format.**
"""
import json
# # Function to parse JSON data and extract key-value pairs
def extract_json_data(json_str):
try:
data_dict = json.loads(json_str)
return data_dict
except json.JSONDecodeError as e:
print(f"Error parsing JSON: {e}")
print(f"Invalid JSON data: {json_str}")
return {}
"""
**Comment**:The above function, extract_json_data, is designed to parse a JSON string (json_str) and return a dictionary (data_dict) containing the key-value pairs from the JSON data. If a JSON decoding error occurs, it prints an error message and returns an empty dictionary."""
# Apply the function to the 'llama_response' column
data['llama_response_parsed'] = data['llama_response'].apply(extract_json_data)
# This code line adds a new column to a DataFrame named llama_response_parsed,
# where each entry is the result of applying the extract_json_data function to
# the corresponding value in the 'llama_response' column.
# This process involves parsing JSON data in the 'llama_response'
# column and storing the resulting dictionaries in the new column.
data.head()
"""**Comment:** A new column **llama_response_parsed** is added to the original dataframe, which contains the resulting dictionaries generated by parsing JSON data in the 'llama_response' column"""
# creating a new dataframe from the parsed llama response
llama_response_parsed_df = pd.json_normalize(data['llama_response_parsed'])
# This code line creates a new DataFrame, llama_response_parsed_df,
# by flattening and normalizing the JSON data stored in the 'llama_response_parsed'
# column of the existing DataFrame (data). The resulting DataFrame is structured to
# better handle and analyze the parsed Llama responses
llama_response_parsed_df.head()
"""**Comment:** The resulting DataFrame is structured to better handle and analyze the parsed Llama responses. The llama_response_parsed_df DataFrame has five columns namely the Category, tags,priority and estimated time column."""
# complete the code to concatenate the original data with the dataframe containing the parsed model output
data_with_parsed_model_output = pd.concat([data, llama_response_parsed_df], axis=1)
data_with_parsed_model_output.head()
"""**Comment:** The Original DataFrame 'data' and 'llama_response_parsed_df' are now joined or concatenated together to create a new DataFrame called data_with_parsed_model_output, which now contains eight (8) columns support_tick_id, support_ticket_text, llama_response, llama_response_parsed, category, subtasks.tags, subtasks.priority, and subtasks.estimated_time"""
# complete the code to drop the columns containing the raw model response
final_data = data_with_parsed_model_output.drop(['llama_response', 'llama_response_parsed'], axis=1)
final_data.head()
"""**Comment:** The columns llama_response, and llama_response_parsed containing the raw moded responses are being dropped remaining the Six(6) columns which are as per expected output.
### **Q6: Share your observations and insights from this exercise, and your recommendations for a business looking to adopt a solution such as this.**
**-Observations and Insights**
1. The generate_llama_response function utilizes the LLaMA model to generate responses for support ticket text.
2. The responses include system instructions and are formatted in a concise JSON output, following a predefined structure.
3. The message "Llama.generate: prefix-match hit" suggests that the Llama model successfully matches specified prefixes during text generation. This indicates that the model is designed to recognize and respond to particular input patterns or contexts.
**-Insights**
1. Leveraging the Llama model enables organizations to respond promptly and efficiently to customer input. Incorporating Artificial Intelligence (AI) through automated Support Ticket Categorization, utilizing Large Language Models and Generative AI, is imperative in the contemporary business landscape.
2. To encourage growth, prolonged customer engagement, and the nurturing of lifetime value relationships, utilization of Generative AI cannot be over emphazised, as this method that allows for the discernment of the most pressing issues, set priorities, and allocate resources judiciously, which fosters customer retention.
**-Recommendations**
1. The adoption of this solution allows businesses to automate the categorization and processing of support tickets using the LLaMA model.
2. Utilizing JSON for structured outputs enables easy extraction and analysis of information, facilitating efficient handling of support ticket data.
3. The provided structure in the final DataFrame ('data_with_parsed_model_output') makes it convenient for further analysis and reporting.
4. Regular monitoring and evaluation of the model's performance and adjustments to the parsing function may be necessary to handle variations in input patterns over time.
5. Establishing a feedback mechanism from end-users and support staff is crucial to continuously improve the model's accuracy and adaptability to evolving patterns in support ticket data.
6. Depending on the volume of support tickets and the performance of the LLaMA model, businesses should consider scalability factors, such as efficient use of computational resources and optimization of processing times.
---
"""