datastoreio doesn't work with entities containing keys

[kennknowles]

If I trie to map over an entity struct that contains a *datastore.Key field, I get the error:

panic: type value main.Foo must be a concrete type

 

It looks like this is because typex.IsConcrete doesn't support pointers.

Imported from Jira BEAM-5200. Original Jira may contain additional context.
Reported by: ribrdb.

[remko]

The JIRA ticket says:

This issue was marked "stale-P2" and has not received a public comment in 14 days. It is now automatically moved to P3. If you are still affected by it, you can comment and move it back to P2.

I am affected by this, so should this become a P2 again? I'm unable to use the datastore connector because almost all my kinds have embedded Keys in them, and I get a stack overflow when starting the pipeline.

This is my program:

package main

import (
	"context"
	"reflect"

	"cloud.google.com/go/datastore"
	"github.com/apache/beam/sdks/v2/go/pkg/beam"
	"github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime"
	"github.com/apache/beam/sdks/v2/go/pkg/beam/io/datastoreio"
	"github.com/apache/beam/sdks/v2/go/pkg/beam/runners/prism"
)

type Foo struct {
	Bar *datastore.Key `datastore:"bar"`
}

var fooKey string

func init() {
	fooKey = runtime.RegisterType(reflect.TypeOf((*Foo)(nil)).Elem())
}

func main() {
	beam.Init()
	p := beam.NewPipeline()
	s := p.Root()
	datastoreio.Read(s, "myproject", "Foo", 256, reflect.TypeOf(Foo{}), fooKey)
	prism.Execute(context.Background(), p)
}

I attached the output of the command.

stack.txt

Looking at the stack, I suspect that it is because reflectTypeToFieldType is recursively traversing datastore.Key, which has a *datastore.Key field, so it goes into an infinite loop.

[remko]

This is probably related to ticket #18846 (which uses datastore.Key as an example)

[remko]

I think I have found a (tedious) workaround for this issue, and that is to register a custom coder for every struct type that has a *datastore.Key. That stops the binary from crashing, and seems to work. This is obviously not very convenient.

@kennknowles Can the priority of this issue be re-evaluated? The datastore connector isn't really usable without support for Key fields in structs.