Add run time and memory usage

[spatts14]

For each analysis module, add to get estimated run time and use of memory

Want to include info on how this might scale depending on the amount of data inputted

Maybe use crude approach?

import time

# Get the time at the start of the run
start_time = time.time()

# Analysis code to run

# Get the time at the end of the run
end_time = time.time()

# Calculate the total time of the run
total_time = end_time - start_time
print(total_time)

[AdrianDAlessandro]

I'd recommend using the in-built python logging module and logging these things to Debug or Info level

[spatts14]

How do I do this? Can you show me an example on one of the scripts please?

[jianlianggao]

Alternatively, you can use psutil, an example of use is as follows,

import psutil
def get_memory_usage():
    process = psutil.Process(os.getpid())
    return process.memory_info().rss / 1024 / 1024  # Memory in MB

then you can call this function anywhere you think you want to observe the memory usage. Hope this helps.

[AdrianDAlessandro]

How do I do this? Can you show me an example on one of the scripts please?

I was only suggesting to use logging instead of the print function

Jianliang (@jianlianggao)'s suggestion above is a good one for getting the memory usage. There are tools for profiling code that might be useful here. However, what is the reason for doing this? Is it to show the user these values? Or to help with development?

[spatts14]

A few reasons

1. I thought it was part of good practice to have an idea how long and how much memory you're using in different analysis modules (inform user on deciding if they can run them locally or should run on HCP)
2. When developing, knowing when modules or aspects are taking a lot of time to run so to be mindful on how to work with them efficiently
3. As a user, they will change parameters to choose the optimal way to analyze the data. Knowing how long something will take to run (even roughly like 5 min, 1 hr, etc) is helpful for a few things...

• making decisions on a parameter (i.e if it takes 25 min to run a function when x = 100, but 10 5 min when x =80 and the results aren't much different, it might make more sense to use x from a biological and efficiency point of view)
• structuring downtime (i.e I can go work on something else this is going to take 30 minutes to run)

Thats what I was thinking. I also just wanted to learn how to include this information

[AdrianDAlessandro]

Those are all good reasons. Typically you don't include this stuff in a package, and it's up to the users to do it, but this makes sense, especially considering users will be less code-savvy.

I suggest we leave this on hold until we have a clear picture of how the overall code will be structured. I think the best way to do this is to put the timings into the main module that runs all of it, instead of into the individual modules.

[AdrianDAlessandro]

If you want to include this now, it's very easy to put the run time into the __main__.py. For example (using your code from above):

    if config.modules.format_data:
        from recode_st.format_data import run_format

        logger.info("Running Module 0 - Format")
        start_time = time.time()
        run_format(config.io)
        total_time = time.time() - start_time
        logger.info(f"Module 0 completed in {total_time:.3f} seconds")

Memory usage is more complicated, you'd need to find how to get memory usage of a function.