Skip to content

fetcher.fetch and pickle operations cost lots of time #1

@robotsp

Description

@robotsp

Thanks for the implementation, it is a great project.

I have a performance test, the result makes me confused, would you please help me understand.

`
while True:
socket.send(byte_message(myid, CODE_POLL, (jobid, data)), copy=False)

    msg = pickle.loads(socket.recv())
    jobid = msg['message']    # list converted to string(eg. "[id1, id2, ...]")
    if jobid is None:
        jobid = data = None
        time.sleep(1)
    else:
        data = fetcher.fetch(ast.literal_eval(jobid))

`

In the upper codelines,
`
msg = pickle.loads(socket.recv())

data = fetcher.fetch(ast.literal_eval(jobid))
`
cost 90% of time, which makes the bottlenecks of training, I wonder how to fix. Pickle seems not the best performance serialization tools but it supports most of object types in python. fetcher.fetch seems cost much time than local dataloader within the same batch size, does it make sense?

@ildoonet Thanks for your reply in advance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions