Merge pull request #90 from LAMPSPUC/update_estimation_procedure

update estimation procedure and paper experiments

Author: andreramosfdc

.gitignore

@@ -23,3 +23,7 @@ paper_tests/m4_test/evaluate_model2.jl
 results_PROPHET/    
 results_SS/
 sarima/
+
+results_simulation/
+results_metrics/
+ssl_vs_kalman_paired_tests.csv

Project.toml

@@ -1,11 +1,12 @@
 name = "StateSpaceLearning"
 uuid = "971c4b7c-2c4e-4bac-8525-e842df3cde7b"
 authors = ["andreramosfc <andreramosfdc@gmail.com>"]
-version = "2.0.15"
+version = "2.1.0"
 
 [deps]
 Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
 GLMNet = "8d5ece8b-de18-5317-b113-243142960cc6"
+HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3"
 LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

README.md

@@ -73,8 +73,8 @@ log_air_passengers = log.(airp.passengers)
 steps_ahead = 30
 
 model = StructuralModel(log_air_passengers)
-fit!(model)
-prediction_log = forecast(model, steps_ahead) # arguments are the output of the fitted model and number of steps ahead the user wants to forecast
+StateSpaceLearning.fit!(model)
+prediction_log = StateSpaceLearning.forecast(model, steps_ahead) # arguments are the output of the fitted model and number of steps ahead the user wants to forecast
 prediction = exp.(prediction_log)
 
 plot_point_forecast(airp.passengers, prediction)
@@ -234,36 +234,63 @@ StateSpaceModels.fit!(ss_model)
 
 ## Paper Results Reproducibility
 
-The paper has two experiments (results for the M4 competition and a simulation study). To reproduce each experiment follow the instructions below:
+To reproduce the paper results, run the following experiments:
+
+### M4 Competition Test
 
-### M4 Experiment
+Evaluates SSL and SS (StateSpaceModels) benchmark models on M4 competition dataset across all granularities (Monthly, Quarterly, Daily, Hourly, Weekly, Yearly).
 
-To reproduce M4 paper results you can clone the repository and run the following commands on terminal:
+**Before running:** Add PyCall to your Julia environment:
+```julia
+using Pkg
+Pkg.add("PyCall")
+```
+
+The script also requires Python packages (`statsmodels`, `numpy`) for the SS benchmark evaluation.
 
 ```shell
 julia paper_tests/m4_test/m4_test.jl
-python paper_tests/m4_test/m4_test.py
 ```
 
-The results for SSL model in terms of MASE and sMAPE for all 48000 series will be stored in folder "paper_tests/m4_test/results_SSL". The average results of MASE, sMAPE and OWA will be saved in file "paper_tests/m4_test/metric_results/SSL_METRICS_RESULTS.csv".
+The script:
+- Downloads M4 competition datasets directly from GitHub
+- Runs SSL models with various configurations (with/without outliers, different selection methods, and alpha values)
+- Runs SS (StateSpaceModels) benchmark models using PyCall
+- Calculates metrics (MASE, sMAPE, OWA, CRPS) for all models
 
-The results for SS model in terms of MASE and sMAPE for all 48000 series will be stored in folder "paper_tests/m4_test/results_SS". The average results of MASE, sMAPE and OWA will be saved in file "paper_tests/m4_test/metric_results/SS_METRICS_RESULTS.csv".
+Results are saved in:
+- `paper_tests/m4_test/results_SSL/` - SSL model results by granularity
+- `paper_tests/m4_test/metrics_results/` - Summary metrics and benchmark results
 
-### Simulation Experiment
+### Simulation Parameter Study
 
-To reproduce the simulation results you can clone the repository and run the following commands on terminal:
+Compares SSL vs Kalman filter on simulated data.
 
 ```shell
-julia paper_tests/simulation_test/simulation.jl 0
+julia paper_tests/simulation_param/simulation.jl [repetitions] [sample_sizes]
 ```
 
-As this test takes a long time, you may want to run it in parallel, for that you can change the last argument to be number of workers to use in the parallelization:
+Example:
+```shell
+julia paper_tests/simulation_param/simulation.jl 100 60,120,240
+```
+
+Results are saved in `paper_tests/simulation_param/ssl_vs_kalman_paired_tests.csv`.
+
+### Simulation Test
+
+Large-scale simulation comparing SSL, SS, and other methods.
+
+```shell
+julia paper_tests/simulation_test/simulation.jl [num_workers]
+```
 
+Example (with 4 parallel workers):
 ```shell
-julia paper_tests/simulation_test/simulation.jl 3
+julia paper_tests/simulation_test/simulation.jl 4
 ```
 
-The results will be saved in two separated files: "paper_tests/simulation_test/results_metrics/metrics_confusion_matrix.csv" and "paper_tests/simulation_test/results_metrics/metrics_bias_mse.csv"
+Results are saved in `paper_tests/simulation_test/results_simulation/` and `paper_tests/simulation_test/results_metrics/`.
 
 
 ## Contributing