Not able to generate meaningful results on gsm8k_synthetic_cot

[yongzx]

Thank you for creating this repo!

May I know what should be the correct configuration for training your model/setup (since you are using an entirely different model and eval dataset than the original paper)?

I tried your default configuration as well as increasing the training steps 10x to max_steps=10000, but the output_stage_3/checkpoint-10000 has an accuracy score of 0.006 on your gsm8k_synthetic_cot. Many of the output predictions are None such as the following. It also got the answer for your infer.py example wrong.

Input: 
It was Trevor's job to collect fresh eggs from the family's 4 chickens every morning.   He got 4 eggs from Gertrude and 3 eggs from Blanche.  Nancy laid 2 eggs as did Martha.  On the way, he dropped 2 eggs.  How many eggs did Trevor have left?
Target: 9
Predicted: None

[casper-hansen]

Hi @yongzx, I am aware of this issue. I simply implemented Coconut as it was described in the paper and shared this as an early version, hoping others would benefit. There may be some bugs that are not yet discovered in the modeling code

[casper-hansen]

In particular, this parts right here is what I am most unsure about and is something I worked a lot on modifying.

1. The slicing of the last hidden state. Why not feed in the full hidden state?
   

   OpenCoconut/opencoconut/models/qwen2.py

   Line 45 in 5519bf4

   inputs_embeds = outputs.last_hidden_state[:, -1:, :]

2. Handling caching/masking in with batch size > 1 (I think this could be buggy)

• OpenCoconut/opencoconut/models/qwen2.py

  Lines 46 to 57 in 5519bf4

  	attention_mask = torch.cat(
  	(
  	attention_mask,
  	torch.ones(
  	(inputs_embeds.shape[0], 1),
  	dtype=attention_mask.dtype,
  	device=attention_mask.device,
  	),
  	),
  	dim=1,
  	)
  	past_key_values = outputs.past_key_values

• OpenCoconut/opencoconut/models/qwen2.py

  Lines 257 to 290 in 5519bf4

  	for b in range(input_ids.shape[0]):
  	insert_idx = insert_indices[b]
  	new_attention_mask.append(
  	torch.cat(
  	(
  	attention_mask[b, :insert_idx],
  	torch.ones(
  	num_thoughts - 1,
  	dtype=attention_mask.dtype,
  	device=attention_mask.device,
  	),
  	attention_mask[b, insert_idx:],
  	)
  	)
  	)
  	new_labels.append(
  	torch.cat(
  	(
  	labels[b, :insert_idx],
  	torch.full(
  	(num_thoughts - 1,),
  	-100,
  	dtype=labels.dtype,
  	device=labels.device,
  	),
  	labels[b, insert_idx:],
  	)
  	)
  	)
  	# Stack the attention masks and labels along the batch dimension
  	attention_mask = torch.stack(new_attention_mask, dim=0)
  	labels = torch.stack(new_labels, dim=0)

[yongzx]

Gotcha! thank you for being transparent – I will also look into this

[thelongestusernameofall]

In particular, this parts right here is what I am most unsure about and is something I worked a lot on modifying.

1. The slicing of the last hidden state. Why not feed in the full hidden state?
   

   OpenCoconut/opencoconut/models/qwen2.py

   Line 45 in 5519bf4

   inputs_embeds = outputs.last_hidden_state[:, -1:, :]

2. Handling caching/masking in with batch size > 1 (I think this could be buggy)

• OpenCoconut/opencoconut/models/qwen2.py

  Lines 46 to 57 in 5519bf4

  	attention_mask = torch.cat(
  	(
  	attention_mask,
  	torch.ones(
  	(inputs_embeds.shape[0], 1),
  	dtype=attention_mask.dtype,
  	device=attention_mask.device,
  	),
  	),
  	dim=1,
  	)
  	past_key_values = outputs.past_key_values

• OpenCoconut/opencoconut/models/qwen2.py

  Lines 257 to 290 in 5519bf4

  	for b in range(input_ids.shape[0]):
  	insert_idx = insert_indices[b]
  	new_attention_mask.append(
  	torch.cat(
  	(
  	attention_mask[b, :insert_idx],
  	torch.ones(
  	num_thoughts - 1,
  	dtype=attention_mask.dtype,
  	device=attention_mask.device,
  	),
  	attention_mask[b, insert_idx:],
  	)
  	)
  	)
  	new_labels.append(
  	torch.cat(
  	(
  	labels[b, :insert_idx],
  	torch.full(
  	(num_thoughts - 1,),
  	-100,
  	dtype=labels.dtype,
  	device=labels.device,
  	),
  	labels[b, insert_idx:],
  	)
  	)
  	)
  	# Stack the attention masks and labels along the batch dimension
  	attention_mask = torch.stack(new_attention_mask, dim=0)
  	labels = torch.stack(new_labels, dim=0)

the same confusion。
Maybe we can modify and try.