Speed up Decode and Encode timing

[xRowe]

I see there is a old thread on this topic, check if this topic can be carried on.
#362

Is there any new solution to speed up decoding and encoding timing?

Online receving Messages by python-can and decoding: canmatrix v.s cantools

Example Test Code

def on_message_received(msg):
        start = time.perf_counter_ns()

        dbc_msg = _database.frame_by_id(msg.arbitration_id)
        
        # start = time.perf_counter_ns()
        decoded = dbc_msg.decode(msg.data)
        end = time.perf_counter_ns()
        
       timings.append(end-start)

canmatrix
decode
Mean +- std dev: 49.1 us +- 36.5 us (2207 values)
Mean +- std dev: 97.4 us +- 182.5 us (12201 values)
received_message + decode
Mean +- std dev: 147.1 us +- 157.3 us (12201 values)

cantools
decode
Mean +- std dev: 30.0 us +- 29.9 us (2200 values)
Mean +- std dev: 34.8 us +- 35.9 us (12201 values)
received_message + decode
Mean +- std dev: 64.9 us +- 57.7 us (12201 values)

After looking at cantools solution, after database is parsed, it will generate a _codec info which allowed to more quickly encode/decode a message, and generate two dicts which allow quick search message by name or id.

But canmatrix, everytime when receive a raw message, I need to found it in database by using for test in self.frames:, when call decode() method, it try to upack the data by bytes_to_bitstrings() and bitstring_to_signal_list(), which also using for loop.

I do not if bitsruct may help better, and refer to cantools

[ebroecker]

Hi @xRowe

thanks for doing the measurements!

It would be easy to implement some dictionary to allow quick search message by name or id.
Thus I'm gonna implement such a hash-table which I think is a good idea.

Bitstruct on the other hand is (or at least was) not able to handle multiplexed messages.
Multiplexer enables messages(=frames/pdus) to have different layouts depending on the content of some bits.

[xRowe]

Thanks for your reply.

There are commits along with implementing the DICTs already,

canmatrix/src/canmatrix/canmatrix.py

Lines 1994 to 2002 in e7ce2aa

	def get_frame_by_id(self, id: int
	) -> typing.Union[Frame, None]:
	"""Get Frame by id.
	:param str name: Frame id to search for
	:rtype: Frame or None
	"""
	return self.frames_dict_id[id]

canmatrix/src/canmatrix/canmatrix.py

Lines 2029 to 2036 in e7ce2aa

	def get_frame_by_name(self, name): # type: (str) -> typing.Union[Frame, None]
	"""Get Frame by name.
	:param str name: Frame name to search for
	:rtype: Frame or None
	"""
	return self.frames_dict_name[name]

And I create/add the data here, but I think the way is not good enough, and not cover all the cases.

canmatrix/src/canmatrix/canmatrix.py

Lines 2075 to 2089 in e7ce2aa

	def add_frame(self, frame): # type: (Frame) -> Frame
	"""Add the Frame to the Matrix.
	:param Frame frame: Frame to add
	:return: the inserted Frame
	"""
	self.frames.append(frame)
	self.frames_dict_name[frame.name] = frame
	if frame.header_id:
	self.frames_dict_id[frame.header_id] = frame
	elif frame.arbitration_id.id:
	self.frames_dict_id[frame.arbitration_id.id] = frame
	return self.frames[len(self.frames) - 1]

Though my project is running these code, I did not run the benchmark then.

If you have better idea, please improve it.

Thanks,
Rowe

[ebroecker]

hi @xRowe

I (finally) tried some stupid hashing

#825

maybe you could have a look on it.

PS: ok, seems to have errors still...

[xRowe]

An issue relate to this topic.

For my working project, it need to decode ethernet pdu.
Example:
A pdu, is both rx and tx for a node,
Current implementation of this library, it will generate two object for each tx and rx, the objects have the same header_id, signals, etc.. but different endpoints.

If a DICT sort ethernet frames/pdu by header_id, will occur issue in such situation.

So necessary to disscus with you the solution

Another Issue
After ethernet frames parser(with the objects have same name, id, different endpoint) then put the result into CanCluster, the transmitter and receiver will get wrong.

canmatrix/src/canmatrix/formats/arxml.py

Lines 2181 to 2189 in 7fde135

	if decode_ethernet:
	result.update(decode_ethernet_helper(ea, float_factory))
	if decode_flexray:
	result.update(decode_flexray_helper(ea, float_factory))
	result.update(decode_can_helper(ea, float_factory, ignore_cluster_info))
	result = canmatrix.cancluster.CanCluster(result)