docs: clarify when topics are created automatically

[sword2ya]

As the document says, topic will be created on first use by subscribing.

 Topics are created on first use by publishing to the named topic or by subscribing to a channel on the named topic. Channels are created on first use by subscribing to the named channel.

but it does not.

my environments are as below:

# nsqd -version
nsqd v1.0.1-alpha (built w/go1.10)
# nsqlookupd -version
nsqlookupd v1.0.1-alpha (built w/go1.10)
# nsq_tail -version
nsq_tail v1.0.1-alpha
# nsqadmin -version
nsqadmin v1.0.1-alpha (built w/go1.10)

what did I do?

started nsqd , nsqlookupd and nsqadmin using default params.

then i subscribed a topic to two channels by nsq_tail:

nsq_tail --channel=c1 --topic=test --lookupd-http-address=localhost:4161
nsq_tail --channel=c2 --topic=test --lookupd-http-address=localhost:4161

then I visit http://localhost:4171/, but i saw no topics there.

after that, I published a message on topic test by to_nsq:

# to_nsq -nsqd-tcp-address=localhost:4150 -topic=test -rate=1
2018/10/24 15:48:12 Throttling messages rate to max:1/second
{}
2018/10/24 15:48:14 INF    1 (localhost:4150) connecting to nsq

after about 40 second, the c1 received the message, but c2 didn't.

here is the output of nsq_tail:

c1:

nsq_tail --channel=c1 --topic=test --lookupd-http-address=localhost:4161 
2018/10/24 15:47:53 Adding consumer for topic: test
2018/10/24 15:47:53 INF    1 [test/c1] querying nsqlookupd http://localhost:4161/lookup?topic=test
2018/10/24 15:47:53 ERR    1 [test/c1] error querying nsqlookupd (http://localhost:4161/lookup?topic=test) - got response 404 Not Found "{\"message\":\"TOPIC_NOT_FOUND\"}"
2018/10/24 15:48:58 INF    1 [test/c1] querying nsqlookupd http://localhost:4161/lookup?topic=test
2018/10/24 15:48:58 INF    1 [test/c1] (anjiawei-Vostro-3667:4150) connecting to nsqd
{}
2018/10/24 15:49:58 INF    1 [test/c1] querying nsqlookupd http://localhost:4161/lookup?topic=test

c2:

nsq_tail --channel=c2 --topic=test --lookupd-http-address=localhost:4161
2018/10/24 15:48:04 Adding consumer for topic: test
2018/10/24 15:48:04 INF    1 [test/c2] querying nsqlookupd http://localhost:4161/lookup?topic=test
2018/10/24 15:48:04 ERR    1 [test/c2] error querying nsqlookupd (http://localhost:4161/lookup?topic=test) - got response 404 Not Found "{\"message\":\"TOPIC_NOT_FOUND\"}"
2018/10/24 15:49:14 INF    1 [test/c2] querying nsqlookupd http://localhost:4161/lookup?topic=test
2018/10/24 15:49:14 INF    1 [test/c2] (anjiawei-Vostro-3667:4150) connecting to nsqd
2018/10/24 15:50:14 INF    1 [test/c2] querying nsqlookupd http://localhost:4161/lookup?topic=test

is it a bug ? i expected to create the topic on subscribing, and i expected to receive messages on publishing.

[sword2ya]

@ploxiln

[mreiferson]

When connecting to NSQ through nsqlookupd, if a topic has not yet been published to, there's no way for nsqlookupd to know which nsqd contain the topic, so that paragraph you pasted could be a little more specific under these circumstances.

[sword2ya]

But I can’t let all of channels receive the message if subscribers before first publish.
What’s your suggestions?

[ploxiln]

The case where a topic does not exist anywhere yet is somewhat special.

nsqadmin has a page where you can pre-create topics and channels, the "Lookup" page: http://localhost:4171/lookup - it makes requests to nsqlookupd to remember them. When nsqd gets a message that creates a new topic, it first asks nsqlookupd for a list of channels for that topic, and creates them before the topic starts passing messages, so all channels get the first message.

It will still take around 30 seconds or so for running consumers to find out that a topic has appeared on an nsqd which the topic did not exist on before. You can similarly pre-create the topic on the nsqd (though there's no nsqadmin UI for it, you have to use an http client like curl). When consumers start up, they'll find all nsqd that already have that topic right away (if those nsqd have had that topic for at least 30 seconds or so).

[ploxiln]

... but imagine a more "typical" scenario where you publish a test message on some server first, then start up all consumers, then a few minutes later start publishing messages for real, including on new servers ... then this all works correctly without any manual pre-create steps (though that first test message is lost for some channels)

[sword2ya]

You can similarly pre-create the topic on the nsqd (though there's no nsqadmin UI for it, you have to use an http client like curl). When consumers start up, they'll find all nsqd that already have that topic right away (if those nsqd have had that topic for at least 30 seconds or so).

Thanks, it works, I had tried it yesterday.

... but imagine a more "typical" scenario where you publish a test message on some server first, then start up all consumers, then a few minutes later start publishing messages for real, including on new servers ... then this all works correctly without any manual pre-create steps.

And you give a better suggestion.

Can you modify the documents on the design page? Because it may mislead other users.

Topics are created on first use by publishing to the named topic or by subscribing to a channel on the named topic.

[ploxiln]

Yeah, that's the take-away, the docs for this could use some more detail.

[mreiferson]

leaving this open, seems like a useful improvement to the docs...

[mreiferson]

moving this issue to https://github.com/nsqio/nsqio.github.io

[jeefs]

Now I am experiencing the same problem, please tell me how to solve it