Do not always hide atoms when using frames.

[linas]

When multiple frames are being used, the current rocks code will just hide atoms, instead of removing them, when frames are being used. This results in functionally correct behavior, but is wasteful of storage if the atom can actually be deleted.

The AtomSpace extract code currently implements the correct (reference) implementation: it will either hide an atom, when needed, or it will delete the atom, if possible. It can be used as the final arbiter of whether to delete, or not. The backend should follow this advice.

There right way to solve this is to implement the pre-delete and a post-delete calls in the backend. The atomspace calls pre-delete before doing the deletion, and rocks can gather any needed info for the deletion to happen. Next, the atomspace extracts the atom. Then it calls the post-delete hook. The hook code should look to see what the atomspace did: either the absent flag is set on the atom, or the atom is actually gone. If its only marked absent, then rocks should also hide the atom. else, rocks should delete the atom.

[sikefield3]

I'm interested in doing this.

When multiple frames are being used, the current rocks code will just hide atoms, instead of removing them, when frames are being used. This results in functionally correct behavior, but is wasteful of storage if the atom can actually be deleted.

What I found is that currently, the actual hiding takes place with _rfile->Put(...) in RocksStorage::storeMissingAtom?

The AtomSpace extract code currently implements the correct (reference) implementation: it will either hide an atom, when needed, or it will delete the atom, if possible. It can be used as the final arbiter of whether to delete, or not. The backend should follow this advice.

There are two ways to solve this:

1. implement rocks so that it tries to use the same decision logic as the atomspace, when removing. This risks having the two be subtly out of sync.

2. Have a pre-delete and a post-delete call in the backend, so that first, pre-delete is called (and rocks can gather any needed info for the deletion to happen) then the atom is extracted from the atomspace, and then the post-delete code looks to see what the atomspace did: either the absent flag is set on the atom, or the atom is actually gone. If its only marked absent, then rocks should also hide the atom. else, rocks should delete the atom.

Seems that 2 is a better solution.

This is my suggestion how this could be accomplished:
The deletion of the atom in the AS has already happened (I'm not sure about this.). In the recursive function RocksStorage::removeAtom and if we have multi-space, then we can do the following steps:

1. look into the atomspace if the absent flag is set.Handle lookupHandle(const Handle& h) const or bool Atom::isAbsent() const resp. seem to be the functions to get the absent flag ?
2. Either we delete with _rfile->Delete or do the same as in RocksStorage::storeMissingAtom. I assumed that it is enough to check for the absent flag in every recursion step, if not one has to do the whole recursion twice ?

Anyway, I hope I didn't get this entirely wrong! Maybe you could point out what i´m missing.

[linas]

Wow! Cool! Umm. Technical questions require detailed answers;

the actual hiding takes place with _rfile->Put(...) in RocksStorage::storeMissingAtom?

Yes I think so. I wish there was some documentation in the code that explained what that function did! What moron doesn't document their code??? Oh, wait ...

Here's a trick: RocksStorage::print_range prints a range of records in Rocks, in the "raw" format, as strings. You can use it to see what is actually being stored. You can use it from scheme like so:

(cog-rocks-get "a@")  ;;; print everything starting with "a@"
(define rsn (RocksStorageNode "rocks://... whatever"))
(cog-rocks-print rsn  "a@")  ; same as above

Use empty string to dump everything. If you're stepping through examples/demos, you can use this to look at the actual contents of the DB, to see if it matches up with the docs, with what you think it should be.

The deletion of the atom in the AS has already happened (I'm not sure about this.).

Well, that's the main issue. If I recall correctly, what needs to happen is that the backend needs to be warned that the deletion is going to happen; then the atomspace can delete, and then the backed can be told to finish up with whatever it needs to do. Without this two-step, there's just not enough info available to "do it right": either the atom is already gone and we've lost access to it, or its not yet gone, and we don't know what to do with it next. A kind-of chicken-and-egg problem.

Anyway, I hope I didn't get this entirely wrong!

I don't know. Maybe the most important thing is to go through the examples & demos and really be sure that you've got a clear idea of how its supposed to work. I think the code is bug-free (famous last words). The design goal was to make it all "simple and obvious" for the user, which makes the implementation rather complicated. (I recall that it was hard to get it right.) If you're going through the examples, and something seems wrong ... well, that could be a bug. If you're not sure, the cog-rocks-print debug utility is great for exploring what's actually in the DB.

[linas]

Oh, several side remarks:

• The CogStorageNode, which implements storage over the network, has almost no support at all for frames. Only half of one unit test works. The rest is a blank slate. It would be nice to get this working, but ... it would be a huge and difficult effort. It's likely to expose other chicken-n-egg issues.

• There are other performance issues. One of my favorites is to call (cog-report-counts) to print a summary report of what is in the AtomSpace. It's fast and works great in the base AtomSpace; its painfully slow when its in a higher layer. I don't know why. (my datasets contain tens of millions of atoms these days. So "painfully slow" is anything that doesn't clear 100K atom operations/second.)

• Most of the AtomSpace has been very highly tuned for performance. So almost all the juice has been squeezed out. There is only one weird trick left, that might get some more out, but it would be complicated and difficult. But this is off-topic.

[sikefield3]

Tried to test this in some - as it seemed to me - suitable example with two AS + RocksDB to see if the atom gets deleted.

In the situation of this example, we have a base + overlay AS. Now, one can attach a Rocks DB to the overlay.

(define base (cog-atomspace))
(define ovly (cog-new-atomspace base))
(cog-set-atomspace! ovly)
(define rsn (RocksStorageNode "rocks:///home/opcwdir/repodock/ForTestingStuff/foo.rdb"))
(cog-open rsn)

Now, add sth to the base and check if it is also in the overlay, then add it to rocks:

(cog-set-atomspace! base)
(List (Concept "A") (Concept "B"))
(cog-set-atomspace! ovly)
(cog-prt-atomspace)
(store-atomspace )
(cog-rocks-print rsn  "")

Then go back to base and delete some atom:

(cog-set-atomspace! base)
(cog-delete-recursive! (Concept "B"))

Now, if you look in both overlay and rocks, the deletion did occur as expected.

[sikefield3]

Later, I want to try other examples with multiple atomspaces (e.g. frame)

[linas]

Then go back to base and delete some atom:
...
the deletion did occur as expected.

It is not obvious if this is the desired outcome, or if it should even be allowed. If you think of overlays as going from past to future, i.e. the base space containing the oldest data, and the overlays as being newer, then deleting something in the base space is like changing something in the past. Should that be allowed, or not? What might a reasonable person expect to happen? (I don't yet have good answers for any of this. I kind of follow my nose, to see what "feels right".)

Some examples:

• Create (Concept "B") in the base. Then go to the overlay, and create (List (Concept "A") (Concept "B")) Now, go to the base, and delete (Concept "B") ... what should happen to the stuff in the overlay? Should it still be there? Should it be gone? Should there be a warning message "I'm sorry, Dave, I can't allow you to do that"? Should the overlay be protected from damage to the space underneath? Off the top of my head, I don't know.
• Create (Concept "B") in the base. Go to the overlay, and set some value on it (say, a truth value). Now, go to the base, and delete (Concept "B") ... what happens to the stuff in the overlay? Should it still be there with the modified value, or should it be gone?

Currently, I have only two use cases:

• The overlay is used to hold temporary, partial results, and then it is destroyed at the end. The base space holds the "permanent" results. (this is what the query subsystem does).
• There is a stack of frames, recording a history of changes. Each new layer contains deltas on the layer beneath. Lower layers, the "history", "the past", are never changed. (Some of my learning code does this).

Anything outside these two use cases is unexplored. My gut instinct says that it would be best if lower layers cannot be modified. This includes deletion. There is, however, a flag that sort of allows this; it is used to modify the base while using a temporary (because some results obtained in the temporary need to be copied back into the base.)

[sikefield3]

As a starting point for an example with subframes, I looked at the multi space example and I tried to see what happens in the rocks database, if I delete an atom:

(cog-set-atomspace! a)
(Concept "A")

Attach Rocks storage to C:

(cog-set-atomspace! c)
(define rsn (RocksStorageNode "rocks:///home/opcwdir/repodock/ForTestingStuff/foo.rdb"))
(cog-open rsn)
(store-atomspace)
(store-frames a)
(store-frames b)

Now we delete the atom from the subframe A:

(cog-delete-recursive! (Concept "A"))

The atom disappears from C (as we can check with (cog-prt-atomspace), but not from the RocksDB (cog-rocks-print rsn "a@") .

Note: Now, we have the problem that atom is still in A, which might not be intended ?

[sikefield3]

Now, let's try a different (maybe more plausible) scenario (again from the multi-space example):
delete an atom in atomspace A:
We are in the following situation:

(define a (AtomSpace))
(define b (AtomSpace))
(define c (AtomSpace a b))
(cog-set-atomspace! a)
(Concept "I'm in A")

and rocksdb links to atomspace c:

(cog-set-atomspace! c)
(define rsn (RocksStorageNode "rocks:///home/opcwdir/repodock/ForTestingStuff/foo.rdb"))
(cog-open rsn)
(store-atomspace)
(store-frames a)
(store-frames b)

Now, "I'm in A" is deleted from atomspace A (in contrast to the previous example, where the atom was deleted from C) :

(cog-set-atomspace! a)
(cog-delete-recursive! (Concept "I'm in A"))

and again it is purged from C, but not from rocksdb.

[sikefield3]

Now, let's try a different (maybe more plausible) scenario (again from the multi-space example):

Another modification from the previous example I just tried: have exactly the same atom (Concept "AB") in atomspaces A and B (is this likely to happen?).
What happens, if we delete this atom? As expected, if we want to erase (Concept "AB") from C, it is necessary to apply cog-delete-recursive! to both A and B.
But as above, the atom still remains in the rocksdb.

[linas]

This was mouldering in my email box. Sorry for the late reply. Regarding this comment: #15 (comment)

Working as designed. The Atom is not deleted, it is hidden. Let me explain. You wrote:

Now we delete the atom from the subframe A:

(cog-delete-recursive! (Concept "A"))

The atom disappears from C (as we can check with (cog-prt-atomspace), but not from the RocksDB (cog-rocks-print rsn "a@") .

Try (cog-rocks-print rsn "") and you will see:

rkey: >>a@3:<<    rval: >>(ConceptNode "A")<<
rkey: >>f@(as "(uuid . 2)")<<    rval: >>4<<
rkey: >>f@(as "(uuid . 3)")<<    rval: >>5<<
rkey: >>f@(as "(uuid . 4)" 4 5)<<    rval: >>6<<
rkey: >>k@3:4:+1<<    rval: >><<
rkey: >>k@3:6:-1<<    rval: >><<

This states that:

• There is an Atom (ConceptNode "A") and it is assigned a UUID of 3
• There are three atomspaces with (confusing, off-by-two) UUID's 4,5,6. These are the f@ (The off-by-n effect is purely due to order of creation. Try e.g. adding more atoms, before creating more atomspaces.)
• Number 4 is AtomSpace "a" and number 6 is AtomSpace "c"
• (ConceptNode "A") is alive and well in 4: that is what >>k@3:4:+1<< says.
• (ConceptNode "A") is hidden in 6: that is what >>k@3:6:-1<< says.

What happened when you did the delete, you only deleted in frame c, but not in frame a. When you say delete-recursive the "recursive" does not refer to frames, but only to containing links. (So for example, if you had (Link (Concept "a")) then a regular delete on (Concept "a") will error out, because its contained in the link. A delete-recursive will delete the link, without complaining.)

If you closed the storage node, exited, re-opened, restored, you would find that (Concept "a") is present in a, but absent in c (i.e. just as before). The idea is that one can put a bunch of stuff into the base space a, and any changes in c are just a "delta" on a. In particular, deleting from c must not delete from a.

The k@ -1 and +1 are "hide" and "expose" markers. You can continue layering more and more atomspaces, alternately creating and deleting (Concept "a") in each, and you will see these k@ hiding and exposing in each layer.

[linas]

Regardng this comment: #15 (comment)

I see an actual bug. But before I explain the bug, let me point out a variation on what you wrote, that does work correctly. If I take what you wrote and also save c, then everything is OK. (cog-rocks-print rsn "") shows that a@ for (ConceptNode "I'm in A") is there. It will always be there, forever and ever. The most that can happen is that it would be hidden with a k@ And, for this example, if I look at the k@ I see that it is indeed hidden in atomspace a. it is unmarked in atomspace b,c and therefor absent in those.

The optimization, proposed at the very top, is to just go ahead and remove the a@ since it is not visible in any frame, ever.

Along the way, I seem to have spotted an actual, real-life bug. Next comment.

[linas]

I opened issue #21 to describe the bug.