the current framework is mostly deterministic

[yanan1116]

hi authors, here is a concern from mine.

In the current framework, the retrieval path is mostly deterministic.

for each task being tested on the test split, given the query of each task, the following parts are deterministic:

1. First-stage similarity retrieval

• given the query text
• the embedding is deterministic
• similarity ranking over stored query embeddings is deterministic

2. Expansion from query to memory IDs

• once the top query groups are chosen
• the associated memory IDs are looked up by rule
• that step is deterministic

3. Second-stage scoring

• given candidate memories
• similarity and Q are already fixed
• hybrid scoring and sorting are deterministic

4. Final selection

• vanilla topk selection is deterministic
• tri-channel bucket selection is also deterministic

So overall, the retrieval process is largely deterministic, and i suppose that determinism is one of the reasons exploration is weak.

and in the yaml file configs/rl_alf_config.yaml , epsilon is set to 0, which means that no exploration, so overall is deterministic.
and i do not find any words talking about how to set epsilon to be optimal.

(for LLM the temperature is also 0)

It means the work is not an exploration method in the RL sense -weak exploration over the memory bank and weak credit coverage.

in other words, the process is very path-dependent:

• Epoch 1 creates the initial memory bank.
• From Epoch 2 onward, retrieval is already conditioned on what Epoch 1 produced.
• Since retrieval is mostly greedy, early-selected memories keep getting reused and updated.
• That gives the first epoch disproportionate influence on the later trajectory.

That means later epochs are not independent rounds of learning. They are mostly: refinement of an already biased bank state

this means that early good memories get amplified, and early bad or noisy memories can also get locked in, and large parts of the bank remain under-tested.

Could you please give more clarification or thoughts on this point ?

[ai-jiaqian]

Hi, thanks for the detailed read.

You're right that the retrieval path is deterministic given a fixed M —
Phase A similarity, Phase B hybrid scoring, and top-k are all deterministic,
and epsilon=0 in the ALF config reflects that we didn't implement
ε-greedy on retrieval. The "exploration (similarity) / exploitation (Q)"
phrasing in Section 4.2 is loose — similarity-based recall is really
candidate diversification, not exploration in the RL sense. We'll fix that
wording in a revision.

On scope: what we mean by "RL on memory" is the value-learning side —
updating Q from environmental rewards — rather than exploration policy
design. The two are separate axes in RL and we focused on the first. Other
agentic memory works (MemP, A-Mem, Memento) also don't do exploration on
retrieval, so we didn't treat it as the core problem here, though we agree
it's a legitimate direction.

There are a few mechanisms in the current design that relate to the
lock-in dynamic you describe, which might be worth discussing together:

• k1 > k2 recall. Phase A surfaces more candidates than Phase B selects
  (10→5 or 5→3), so memories with under-evaluated Q keep re-entering the
  ranking competition each round.

• The bank keeps growing. Every trajectory writes a new (z, e, Q_init)
  triplet, so new memories are always competing with old ones — the bank
  isn't frozen after epoch 1.

• Q dynamics give natural attrition. Memories that keep producing low
  rewards have their Q decayed by the EMA update and gradually lose Phase
  B ranking against higher-Q alternatives. High-utility memories
  accumulate reinforcement and stabilize. So the bank filters itself
  softly over time rather than staying stuck on whatever epoch 1 produced.

• Generation stochasticity. Even with deterministic retrieval, the LLM's
  generation (temperature=0.6 on HLE) produces different rewards for the
  same (s, m) across epochs, so Q estimates keep moving.

• z-score norm + similarity gating. The ablation in Appendix B.1 shows
  that removing these pushes the forgetting rate from 0.041 to 0.073.
  Without them, early Q imbalances would amplify — which matches your
  intuition. Keeping them in is how we manage that risk currently.

That said, your concern has a real edge we don't fully close: if a memory
gets unlucky on its first evaluation and Q drops below the ranking
threshold, deterministic top-k may not resurface it for re-evaluation.
Attrition works when low-Q is correct; it's less fair when low-Q is noisy.
This is a real limitation of the current implementation.

One related point worth flagging: the optimal retrieval policy we derive
in Appendix A.4.3 is actually a Boltzmann distribution
μ*(m|s) ∝ π_sim · exp(β·Q), i.e. temperature-controlled softmax sampling.
The deterministic top-k we use in the implementation is the β→∞ limit of
that. Moving to a finite-β softmax sampler, or adding ε-greedy / UCB on
top of Phase B, would address the lock-in concern directly, and this is
probably the most natural next step.

Happy to keep discussing.

[yanan1116]

hmm, i see.
UCB and other samping method like cold start problem, can help solve this issue, to provide more opportunities for those memory items that sinked below the surface .

i implement it in my side, and i do see there is a very obvious gain on qwen models (serving with vllm like openai api).
this surely helps break the class solidification, giving to those memories failed at initial epoch more chances to shine.

this is like recommendation system which expose those items to make the Q score more confident.

As for growing memory bank size, while at the same time, a very large proportion of them are of zero Q values, yeah it is a problem. no ideas for now.

[yanan1116]

by the way, frankly speaking, the success rate is already very high on alfworld with the backbone model of GPT-5-mini.

i think for alfworld, the similarities across these tasks, either within a dataset split, or between dataset split (train split and test split), are very high, which makes the memories easy to be sharable and useful.