[Model/trade] Can we get away with train_every_n_epochs > 1?

[trentmc]

Background / motivation

We recently found that for benchmarking "can we make $ trading", we need 178 days worth of sims (50K 5-min candles). 17.8 days (5K candles) is not enough.

But running 178 days takes a long time. On Trent's MacBook Pro M3, it takes about 3 h for one run.

We can probably optimize a lot for << 3h, via the typical flow: run profiling, fix the worst offender, repeat, until diminishing returns. I'll create an issue.

But we might get a quick win: what if we set train_every_n_epochs to be >> 1? Model-building is likely the biggest bottleneck. The question is: how much does that impact "trader profitability" numbers?

TODOs

• do a benchmark comparing train_every_n_epochs at various values: 1, 100, 500, 1000; and see what the difference is.
  • Track: trader profit vs time (and final value), and predictoor profit vs time (and final value).
  • Report those results in this github issue, simply by screenshots of the plots.

[trizin]

I got very interesting results:

Predictoor profits

n = 1:

n = 10:

n = 100:

Model evaluation

n = 1:

• accuracy: 50.54%
• f1: 0.2853
• log loss: 0.9671

n = 10:

• accuracy: 50.19%
• f1: 0.2922
• log loss: 1.0091

n = 100:

• accuracy: 50.75%
• f1: 0.3927
• log loss: 1.4632

Run times

n = 1: 9m0.416s
n = 10: 7m15.635s
n = 100: 5m52.522s

Profiling:

n = 1:

      2/1    0.014    0.007  527.337  527.337 pdr_backend/sim/sim_engine.py:85(run)
10001/10000    9.704    0.001  527.043    0.053 pdr_backend/sim/sim_engine.py:101(run_one_iter)
10001/10000    0.036    0.000  191.797    0.019 pdr_backend/aimodel/aimodel_factory.py:41(build)
10001/10000    0.018    0.000  132.640    0.013 pdr_backend/aimodel/aimodel_factory.py:232(_fit)
     2004    0.048    0.000   86.279    0.043 pdr_backend/sim/sim_plotter.py:107(save_state)
    10000    0.870    0.000   69.461    0.007 pdr_backend/aimodel/aimodel_data_factory.py:60(create_xy)
    10000    0.009    0.000   47.788    0.005 pdr_backend/aimodel/aimodel.py:138(set_importance_per_var)
    10000    0.374    0.000   47.705    0.005 pdr_backend/aimodel/aimodel.py:152(_calc_importance_per_var)
    10000    0.068    0.000   39.895    0.004 pdr_backend/util/mathutil.py:52(has_nan)
    10000    0.052    0.000   20.773    0.002 pdr_backend/aimodel/aimodel.py:53(predict_ptrue)