PINGREQ still going out even when QoS 1 publishes are happening within keep alive period

[tsturzl]

Prerequisites

Note: You may remove this section prior to submitting your report.

A small team of volunteers monitors issues. Please help us to help you by making it simple to understand and, if possible,
replicate your issue. Prior to reporting a bug please:

• Test the latest release of the library.
• Search existing issues.
• Read the relevant documentation.
• Review your server configuration and logs.
• Consider testing against a different server (e.g. mqtt.eclipseprojects.io or test.mosquitto.org)
• If possible, test using another tool (e.g. MQTTX / mosquitto_sub)
  to confirm the issue is specific to this client.
• If you are unsure if you have found a bug, please consider asking on stackoverflow for a quicker response.

Bug Description

I have a client configured for a 4 second keep alive, and I'm publishing QoS 1 messages every 500ms. I still see PINGREQ going out in a packet sniffer. This means that keep alive is making unneeded ping requests. This doesn't track the behavior the documentation seems to suggest, nor does it seem to resemble the behavior of other clients such as the paho.mqtt.java client. Obviously this isn't a major behavior issue, but we noticed this behavioral difference while packet sniffing python and Java clients, and it seems python is the outlier in terms of behavior, especially since the documentation seems to suggest that keep alive should only ping in absence of other control messages.

Reproduction

I'll just provide a test script:

import paho.mqtt.client as mqtt
import time

client = mqtt.Client()

def on_disconnected(_client, _userdata, _rc):
    print("DISCONNECTED")

client.on_disconnect = on_disconnected

# enable logging, let it create it's own logger
client.enable_logger(logger=None)

print("connecting...")
client.connect("localhost", 1883, 5)
print("connected")
print("starting loop...")
client.loop_start()
print("loop started")
while True:
    print("publishing message...")
    client.publish("testtopic1", "hello world", qos=1)
    time.sleep(0.5)

Environment

• Python version: 3.10.9
• Library version: 1.6.1
• Operating system (including version): Ubuntu 22.04.03
• MQTT server (name, version, configuration, hosting details): Mosquito, but have validated against another server implementation as well

Logs

Logs are largely irrelevant. I observed this in a debugger, ping is still happening even though publish QoS 1 is happening many times in the keep alive period.

[langer-volt]

Is your client subscribing to anything? I believe a long duration of no messages in could be triggering your PINGREQ

It's a logical OR that can be triggered if <msg_in_expired> OR <msg_out_expired>
https://github.com/eclipse/paho.mqtt.python/blob/d45de3737879cfe7a6acc361631fa5cb1ef584bb/src/paho/mqtt/client.py#L3271

[tsturzl]

I don't believe we need to be subscribed on anything. Doing the exact same thing I Java results in no PINGREQ being sent. QoS 1 will always have a PUBACK on success, so it should be enough to prove the connection is open and functional bidirectionally.

[neel-infocusp]

I found that the last_msg_out timestamp is not being updated properly after a packet is sent.

Based on the current logic mentioned L3223 and L3244, the last_msg_out timestamp is updated only when there is a disconnection and the write_length value is <= 0.

Ideally, the last_msg_out timestamp should be updated after a packet is successfully sent (or partially sent).

I have added a fix for this in the PR: 903

[JamesParrott]

@neel-infocusp Thanks for your PR but there wasn't really problem here to fix in the first place, and the PR 903 breaks QoS 0 messages.

Resetting the keep alive timeout counter, mid packet upload, for packets that take longer to write then the keep alive timeout, breaks paho.mqtt.python's contract with the user, in my opinion.

A correct fulfilment of the feature request needs several supporting tests, and much more careful consideration. Paho.mqtt.python's behaviour may well differ from Java's etc. but Clients are free to make PINGREQs "at any time" under the spec