Errors are as follows :
2022-08-31 18:27:29.737977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21644 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6
I0831 18:27:29.751516 140527906674496 saver.py:1395] Restoring parameters from /tmp/run_example/tuner-1/23/model.ckpt-1000
I0831 18:27:29.763098 140527906674496 builder_impl.py:669] Assets added to graph.
I0831 18:27:29.763194 140527906674496 builder_impl.py:463] No assets to write.
I0831 18:27:29.783092 140527906674496 builder_impl.py:428] SavedModel written to: /tmp/run_example/tuner-1/23/saved_model/temp-1661938049/saved_model.pb
I0831 18:27:29.784265 140527906674496 oss_trainer_lib.py:278] Evaluation results: {'accuracy': 0.7, 'auc_pr': 1.0, 'auc_roc': 0.9999998, 'loss': 0.54577476, 'num_parameters': 9236, 'global_step': 1000}
I0831 18:27:29.786289 140527906674496 oss_trainer_lib.py:303] creating directory: /tmp/run_example/tuner-1/24
I0831 18:27:29.786396 140527906674496 oss_trainer_lib.py:350] Tuner id: tuner-1
I0831 18:27:29.786435 140527906674496 oss_trainer_lib.py:351] Training with the following hyperparameters:
I0831 18:27:29.786465 140527906674496 oss_trainer_lib.py:352] {'learning_rate': 5.270349994604263e-06, 'new_block_type': 'FULLY_CONNECTED_RESIDUAL_PROJECT_BATCHNORM', 'optimizer': 'sgd', 'initial_architecture_0': 'FULLY_CONNECTED_RESIDUAL_PROJECT', 'exponential_decay_rate': 0.9005046125313068, 'exponential_decay_steps': 1000, 'gradient_max_norm': 2, 'dropout_rate': 0.20000000596046447, 'initial_architecture': ['FULLY_CONNECTED_RESIDUAL_PROJECT']}
I0831 18:27:29.786560 140527906674496 run_config.py:549] TF_CONFIG environment variable: {'model_dir': '/tmp/run_example/tuner-1/24', 'session_master': ''}
I0831 18:27:29.786598 140527906674496 run_config.py:985] Using model_dir in TF_CONFIG: /tmp/run_example/tuner-1/24
I0831 18:27:29.786791 140527906674496 estimator.py:202] Using config: {'_model_dir': '/tmp/run_example/tuner-1/24', '_tf_random_seed': None, '_save_summary_steps': 2000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 120, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I0831 18:27:29.803411 140527906674496 estimator.py:1173] Calling model_fn.
I0831 18:27:29.803532 140527906674496 phoenix.py:355] <tensorflow_estimator.python.estimator.run_config.RunConfig object at 0x7fce0074fdf0>
I0831 18:27:29.803575 140527906674496 phoenix.py:356] /tmp/run_example/tuner-1/24
I0831 18:27:29.823276 140527906674496 controller.py:160] trial id: 24
I0831 18:27:29.823375 140527906674496 controller.py:239] intermix ensemble search mode
I0831 18:27:29.827098 140527906674496 phoenix.py:387] {'search_generator': GeneratorWithTrials(instance=<model_search.generators.search_candidate_generator.SearchCandidateGenerator object at 0x7fce26aa6280>, relevant_trials=[<model_search.metadata.trial.Trial object at 0x7fce1c78baf0>, <model_search.metadata.trial.Trial object at 0x7fce1c78bbe0>, <model_search.metadata.trial.Trial object at 0x7fce0075c5b0>, <model_search.metadata.trial.Trial object at 0x7fce1c7a6b20>, <model_search.metadata.trial.Trial object at 0x7fce1c7956a0>, <model_search.metadata.trial.Trial object at 0x7fce1c795df0>, <model_search.metadata.trial.Trial object at 0x7fce1c797850>, <model_search.metadata.trial.Trial object at 0x7fce1c797c40>, <model_search.metadata.trial.Trial object at 0x7fce1c793c40>, <model_search.metadata.trial.Trial object at 0x7fce1c794850>, <model_search.metadata.trial.Trial object at 0x7fce1c794c40>, <model_search.metadata.trial.Trial object at 0x7fce24371850>, <model_search.metadata.trial.Trial object at 0x7fce24359850>, <model_search.metadata.trial.Trial object at 0x7fce24359c40>, <model_search.metadata.trial.Trial object at 0x7fce24372850>, <model_search.metadata.trial.Trial object at 0x7fce24372c40>, <model_search.metadata.trial.Trial object at 0x7fce2434ac40>, <model_search.metadata.trial.Trial object at 0x7fce24337850>, <model_search.metadata.trial.Trial object at 0x7fce24337c40>])}
I0831 18:27:29.827618 140527906674496 coordinate_descent.py:63] Maximal depth allowed: 4
I0831 18:27:29.827801 140527906674496 search_candidate_generator.py:123] Creating new architecture:
I0831 18:27:29.827841 140527906674496 search_candidate_generator.py:124] [82 81]
I0831 18:27:29.830933 140527906674496 phoenix.py:406] {'search_generator': [<model_search.architecture.tower.Tower object at 0x7fce24351d90>]}
I0831 18:27:30.007734 140527906674496 run_config.py:549] TF_CONFIG environment variable: {'model_dir': '/tmp/run_example/tuner-1/24', 'session_master': ''}
I0831 18:27:30.007827 140527906674496 run_config.py:985] Using model_dir in TF_CONFIG: /tmp/run_example/tuner-1/24
I0831 18:27:30.243517 140527906674496 estimator.py:1175] Done calling model_fn.
I0831 18:27:30.244107 140527906674496 basic_session_run_hooks.py:558] Create CheckpointSaverHook.
I0831 18:27:30.344659 140527906674496 monitored_session.py:243] Graph was finalized.
2022-08-31 18:27:30.344949: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345355: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345549: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345726: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21644 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6
I0831 18:27:30.390913 140527906674496 session_manager.py:527] Running local_init_op.
I0831 18:27:30.404366 140527906674496 session_manager.py:530] Done running local_init_op.
I0831 18:27:30.648690 140527906674496 basic_session_run_hooks.py:628] Calling checkpoint listeners before saving checkpoint 0...
I0831 18:27:30.649056 140527906674496 basic_session_run_hooks.py:633] Saving checkpoints for 0 into /tmp/run_example/tuner-1/24/model.ckpt.
I0831 18:27:30.703323 140527906674496 basic_session_run_hooks.py:640] Calling checkpoint listeners after saving checkpoint 0...
I0831 18:27:31.163495 140527906674496 basic_session_run_hooks.py:266] loss = 1.035786, step = 0
I0831 18:27:31.420209 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 388.237
I0831 18:27:31.420408 140527906674496 basic_session_run_hooks.py:264] loss = 0.9075287, step = 100 (0.257 sec)
I0831 18:27:31.558015 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 725.577
I0831 18:27:31.558200 140527906674496 basic_session_run_hooks.py:264] loss = 1.0140789, step = 200 (0.138 sec)
I0831 18:27:31.690464 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 755.011
I0831 18:27:31.690645 140527906674496 basic_session_run_hooks.py:264] loss = 1.0821533, step = 300 (0.132 sec)
I0831 18:27:31.824359 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 746.852
I0831 18:27:31.824541 140527906674496 basic_session_run_hooks.py:264] loss = 1.1923344, step = 400 (0.134 sec)
I0831 18:27:31.954357 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 769.251
I0831 18:27:31.954540 140527906674496 basic_session_run_hooks.py:264] loss = 1.2966073, step = 500 (0.130 sec)
I0831 18:27:32.083554 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 773.977
I0831 18:27:32.083731 140527906674496 basic_session_run_hooks.py:264] loss = 1.0678508, step = 600 (0.129 sec)
I0831 18:27:32.211544 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 781.295
I0831 18:27:32.211720 140527906674496 basic_session_run_hooks.py:264] loss = 1.1640563, step = 700 (0.128 sec)
I0831 18:27:32.342442 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 763.951
I0831 18:27:32.342617 140527906674496 basic_session_run_hooks.py:264] loss = 1.2269, step = 800 (0.131 sec)
I0831 18:27:32.476507 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 745.914
I0831 18:27:32.476685 140527906674496 basic_session_run_hooks.py:264] loss = 1.2129242, step = 900 (0.134 sec)
I0831 18:27:32.605798 140527906674496 basic_session_run_hooks.py:628] Calling checkpoint listeners before saving checkpoint 1000...
I0831 18:27:32.605903 140527906674496 basic_session_run_hooks.py:633] Saving checkpoints for 1000 into /tmp/run_example/tuner-1/24/model.ckpt.
I0831 18:27:32.649677 140527906674496 basic_session_run_hooks.py:640] Calling checkpoint listeners after saving checkpoint 1000...
I0831 18:27:32.660749 140527906674496 estimator.py:361] Loss for final step: 1.1485888.
I0831 18:27:32.678622 140527906674496 estimator.py:1173] Calling model_fn.
I0831 18:27:32.678768 140527906674496 phoenix.py:355] <tensorflow_estimator.python.estimator.run_config.RunConfig object at 0x7fce0074fdf0>
I0831 18:27:32.678819 140527906674496 phoenix.py:356] /tmp/run_example/tuner-1/24
I0831 18:27:32.699427 140527906674496 controller.py:160] trial id: 24
I0831 18:27:32.699542 140527906674496 controller.py:239] intermix ensemble search mode
I0831 18:27:32.704100 140527906674496 phoenix.py:387] {'search_generator': GeneratorWithTrials(instance=<model_search.generators.search_candidate_generator.SearchCandidateGenerator object at 0x7fce26aa6280>, relevant_trials=[<model_search.metadata.trial.Trial object at 0x7fce242128e0>, <model_search.metadata.trial.Trial object at 0x7fce24212490>, <model_search.metadata.trial.Trial object at 0x7fce24348b80>, <model_search.metadata.trial.Trial object at 0x7fce1c7a7820>, <model_search.metadata.trial.Trial object at 0x7fce1c27c220>, <model_search.metadata.trial.Trial object at 0x7fce25e36b20>, <model_search.metadata.trial.Trial object at 0x7fce25e361c0>, <model_search.metadata.trial.Trial object at 0x7fce240dff70>, <model_search.metadata.trial.Trial object at 0x7fce25dbe3d0>, <model_search.metadata.trial.Trial object at 0x7fce25fe2e20>, <model_search.metadata.trial.Trial object at 0x7fce240e4160>, <model_search.metadata.trial.Trial object at 0x7fce25fe2580>, <model_search.metadata.trial.Trial object at 0x7fce1c36ea60>, <model_search.metadata.trial.Trial object at 0x7fce1c0fed60>, <model_search.metadata.trial.Trial object at 0x7fce1c36ee80>, <model_search.metadata.trial.Trial object at 0x7fce1c0feeb0>, <model_search.metadata.trial.Trial object at 0x7fce25f3f5b0>, <model_search.metadata.trial.Trial object at 0x7fce25f3f3d0>, <model_search.metadata.trial.Trial object at 0x7fce2422fac0>])}
I0831 18:27:32.704649 140527906674496 base_tower_generator.py:99] Building from existing checkpoint.
I0831 18:27:32.711285 140527906674496 phoenix.py:406] {'search_generator': [<model_search.architecture.tower.Tower object at 0x7fce24212a90>]}
I0831 18:27:32.773150 140527906674496 run_config.py:549] TF_CONFIG environment variable: {'model_dir': '/tmp/run_example/tuner-1/24', 'session_master': ''}
I0831 18:27:32.773260 140527906674496 run_config.py:985] Using model_dir in TF_CONFIG: /tmp/run_example/tuner-1/24
I0831 18:27:32.929626 140527906674496 estimator.py:1175] Done calling model_fn.
I0831 18:27:32.940950 140527906674496 evaluation.py:250] Starting evaluation at 2022-08-31T18:27:32
I0831 18:27:32.983377 140527906674496 monitored_session.py:243] Graph was finalized.
2022-08-31 18:27:32.983673: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.983907: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.984069: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.984265: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.984439: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.984585: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21644 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6
I0831 18:27:32.984675 140527906674496 saver.py:1395] Restoring parameters from /tmp/run_example/tuner-1/24/model.ckpt-1000
I0831 18:27:33.023914 140527906674496 session_manager.py:527] Running local_init_op.
I0831 18:27:33.039376 140527906674496 session_manager.py:530] Done running local_init_op.
I0831 18:27:33.158018 140527906674496 evaluation.py:269] Inference Time : 0.21697s
I0831 18:27:33.159124 140527906674496 evaluation.py:271] Finished evaluation at 2022-08-31-18:27:33
I0831 18:27:33.159208 140527906674496 estimator.py:2083] Saving dict for global step 1000: accuracy = 0.3, auc_pr = 0.48401165, auc_roc = 7.142855e-08, global_step = 1000, loss = 0.97662735, num_parameters = 86
I0831 18:27:33.209768 140527906674496 estimator.py:2143] Saving 'checkpoint_path' summary for global step 1000: /tmp/run_example/tuner-1/24/model.ckpt-1000
I0831 18:27:33.210166 140527906674496 phoenix.py:136] Saving the following evaluation dictionary.
I0831 18:27:33.210225 140527906674496 phoenix.py:137] {'accuracy': 0.30000001192092896, 'auc_pr': 0.4840116500854492, 'auc_roc': 7.142855196207165e-08, 'loss': 0.9766273498535156, 'num_parameters': 86, 'global_step': 1000}
I0831 18:27:33.210270 140527906674496 ml_metadata_db.py:161] Storing the following evaluation dictionary,
I0831 18:27:33.210300 140527906674496 ml_metadata_db.py:162] {'accuracy': 0.30000001192092896, 'auc_pr': 0.4840116500854492, 'auc_roc': 7.142855196207165e-08, 'loss': 0.9766273498535156, 'num_parameters': 86, 'global_step': 1000}
I0831 18:27:33.210334 140527906674496 ml_metadata_db.py:163] For the model in the following model dictionary,
I0831 18:27:33.210361 140527906674496 ml_metadata_db.py:164] /tmp/run_example/tuner-1/24
I0831 18:27:33.238589 140527906674496 estimator.py:1173] Calling model_fn.
I0831 18:27:33.238721 140527906674496 phoenix.py:355] <tensorflow_estimator.python.estimator.run_config.RunConfig object at 0x7fce0074fdf0>
I0831 18:27:33.238767 140527906674496 phoenix.py:356] /tmp/run_example/tuner-1/24
I0831 18:27:33.259187 140527906674496 controller.py:160] trial id: 24
I0831 18:27:33.259282 140527906674496 controller.py:239] intermix ensemble search mode
I0831 18:27:33.262947 140527906674496 phoenix.py:387] {'search_generator': GeneratorWithTrials(instance=<model_search.generators.search_candidate_generator.SearchCandidateGenerator object at 0x7fce26aa6280>, relevant_trials=[<model_search.metadata.trial.Trial object at 0x7fce25dbcd90>, <model_search.metadata.trial.Trial object at 0x7fce1c46ff40>, <model_search.metadata.trial.Trial object at 0x7fce25dbcbe0>, <model_search.metadata.trial.Trial object at 0x7fce25dbce50>, <model_search.metadata.trial.Trial object at 0x7fce1c46f8b0>, <model_search.metadata.trial.Trial object at 0x7fce1c44cf70>, <model_search.metadata.trial.Trial object at 0x7fce1c44c160>, <model_search.metadata.trial.Trial object at 0x7fce1c29ffd0>, <model_search.metadata.trial.Trial object at 0x7fce25f033d0>, <model_search.metadata.trial.Trial object at 0x7fce1c29ff10>, <model_search.metadata.trial.Trial object at 0x7fce25f3cb80>, <model_search.metadata.trial.Trial object at 0x7fce25f14cd0>, <model_search.metadata.trial.Trial object at 0x7fce00730bb0>, <model_search.metadata.trial.Trial object at 0x7fce1c10fd60>, <model_search.metadata.trial.Trial object at 0x7fce00730b20>, <model_search.metadata.trial.Trial object at 0x7fce00730d00>, <model_search.metadata.trial.Trial object at 0x7fce1c6599d0>, <model_search.metadata.trial.Trial object at 0x7fce1c659580>, <model_search.metadata.trial.Trial object at 0x7fce1c45a9a0>, <model_search.metadata.trial.Trial object at 0x7fce25e47550>])}
I0831 18:27:33.263211 140527906674496 base_tower_generator.py:99] Building from existing checkpoint.
I0831 18:27:33.267359 140527906674496 phoenix.py:406] {'search_generator': [<model_search.architecture.tower.Tower object at 0x7fce1c3d53a0>]}
I0831 18:27:33.322653 140527906674496 run_config.py:549] TF_CONFIG environment variable: {'model_dir': '/tmp/run_example/tuner-1/24', 'session_master': ''}
I0831 18:27:33.322746 140527906674496 run_config.py:985] Using model_dir in TF_CONFIG: /tmp/run_example/tuner-1/24
I0831 18:27:33.324692 140527906674496 estimator.py:1175] Done calling model_fn.
I0831 18:27:33.324922 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Classify: None
I0831 18:27:33.324964 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Regress: None
I0831 18:27:33.324995 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Predict: ['serving_default']
I0831 18:27:33.325023 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Train: None
I0831 18:27:33.325047 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Eval: None
2022-08-31 18:27:33.325256: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.325487: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.325648: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.325842: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.326005: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.326149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21644 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6
I0831 18:27:33.340039 140527906674496 saver.py:1395] Restoring parameters from /tmp/run_example/tuner-1/24/model.ckpt-1000
I0831 18:27:33.352076 140527906674496 builder_impl.py:669] Assets added to graph.
I0831 18:27:33.352158 140527906674496 builder_impl.py:463] No assets to write.
I0831 18:27:33.372737 140527906674496 builder_impl.py:428] SavedModel written to: /tmp/run_example/tuner-1/24/saved_model/temp-1661938053/saved_model.pb
I0831 18:27:33.373998 140527906674496 oss_trainer_lib.py:278] Evaluation results: {'accuracy': 0.3, 'auc_pr': 0.48401165, 'auc_roc': 7.142855e-08, 'loss': 0.97662735, 'num_parameters': 86, 'global_step': 1000}
Traceback (most recent call last):
File "test.py", line 26, in
trainer.try_models(
File "/mnt/hdd1/model_search_220825/model_search/model_search/single_trainer.py", line 78, in try_models
while oss_trainer_lib.run_parameterized_train_and_eval(
File "/mnt/hdd1/model_search_220825/model_search/model_search/oss_trainer_lib.py", line 340, in run_parameterized_train_and_eval
trial = oracle.create_trial(tuner_id)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/engine/oracle.py", line 189, in create_trial
response = self.populate_space(trial_id)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/tuners/bayesian.py", line 209, in populate_space
self.gpr.fit(x, y)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/tuners/bayesian.py", line 83, in fit
self._alpha_vector = cho_solve(self._l_matrix, self._y_train)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/tuners/bayesian.py", line 31, in cho_solve
y = solve_triangular(l_matrix, b.reshape(-1, 1), lower=True)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/tuners/bayesian.py", line 24, in solve_triangular
return tf.linalg.triangular_solve(a, b, lower=lower).numpy()
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 513, in getattr
self.getattribute(name)
AttributeError: 'Tensor' object has no attribute 'numpy'
I've installed model search. And I'm testing the code in getting start but getting errors.
I think successfully installed, but I don't know why this error occurs.
import sys
from absl import app
sys.argv = sys.argv[:1]
try:
app.run(lambda argv: None)
except:
pass
import model_search
from model_search import constants
from model_search import single_trainer
from model_search.data import csv_data
trainer = single_trainer.SingleTrainer(
data=csv_data.Provider(
label_index=0,
logits_dimension=2,
record_defaults=[0, 0, 0, 0],
filename="model_search/data/testdata/csv_random_data.csv"),
spec='model_search/configs/dnn_config.pbtxt')
trainer.try_models(
number_models=200,
train_steps=1000,
eval_steps=200,
root_dir="/tmp/run_example",
batch_size=32,
experiment_name="example",
experiment_owner="model_search_user")
Errors are as follows :
2022-08-31 18:27:29.737977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21644 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6
I0831 18:27:29.751516 140527906674496 saver.py:1395] Restoring parameters from /tmp/run_example/tuner-1/23/model.ckpt-1000
I0831 18:27:29.763098 140527906674496 builder_impl.py:669] Assets added to graph.
I0831 18:27:29.763194 140527906674496 builder_impl.py:463] No assets to write.
I0831 18:27:29.783092 140527906674496 builder_impl.py:428] SavedModel written to: /tmp/run_example/tuner-1/23/saved_model/temp-1661938049/saved_model.pb
I0831 18:27:29.784265 140527906674496 oss_trainer_lib.py:278] Evaluation results: {'accuracy': 0.7, 'auc_pr': 1.0, 'auc_roc': 0.9999998, 'loss': 0.54577476, 'num_parameters': 9236, 'global_step': 1000}
I0831 18:27:29.786289 140527906674496 oss_trainer_lib.py:303] creating directory: /tmp/run_example/tuner-1/24
I0831 18:27:29.786396 140527906674496 oss_trainer_lib.py:350] Tuner id: tuner-1
I0831 18:27:29.786435 140527906674496 oss_trainer_lib.py:351] Training with the following hyperparameters:
I0831 18:27:29.786465 140527906674496 oss_trainer_lib.py:352] {'learning_rate': 5.270349994604263e-06, 'new_block_type': 'FULLY_CONNECTED_RESIDUAL_PROJECT_BATCHNORM', 'optimizer': 'sgd', 'initial_architecture_0': 'FULLY_CONNECTED_RESIDUAL_PROJECT', 'exponential_decay_rate': 0.9005046125313068, 'exponential_decay_steps': 1000, 'gradient_max_norm': 2, 'dropout_rate': 0.20000000596046447, 'initial_architecture': ['FULLY_CONNECTED_RESIDUAL_PROJECT']}
I0831 18:27:29.786560 140527906674496 run_config.py:549] TF_CONFIG environment variable: {'model_dir': '/tmp/run_example/tuner-1/24', 'session_master': ''}
I0831 18:27:29.786598 140527906674496 run_config.py:985] Using model_dir in TF_CONFIG: /tmp/run_example/tuner-1/24
I0831 18:27:29.786791 140527906674496 estimator.py:202] Using config: {'_model_dir': '/tmp/run_example/tuner-1/24', '_tf_random_seed': None, '_save_summary_steps': 2000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 120, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I0831 18:27:29.803411 140527906674496 estimator.py:1173] Calling model_fn.
I0831 18:27:29.803532 140527906674496 phoenix.py:355] <tensorflow_estimator.python.estimator.run_config.RunConfig object at 0x7fce0074fdf0>
I0831 18:27:29.803575 140527906674496 phoenix.py:356] /tmp/run_example/tuner-1/24
I0831 18:27:29.823276 140527906674496 controller.py:160] trial id: 24
I0831 18:27:29.823375 140527906674496 controller.py:239] intermix ensemble search mode
I0831 18:27:29.827098 140527906674496 phoenix.py:387] {'search_generator': GeneratorWithTrials(instance=<model_search.generators.search_candidate_generator.SearchCandidateGenerator object at 0x7fce26aa6280>, relevant_trials=[<model_search.metadata.trial.Trial object at 0x7fce1c78baf0>, <model_search.metadata.trial.Trial object at 0x7fce1c78bbe0>, <model_search.metadata.trial.Trial object at 0x7fce0075c5b0>, <model_search.metadata.trial.Trial object at 0x7fce1c7a6b20>, <model_search.metadata.trial.Trial object at 0x7fce1c7956a0>, <model_search.metadata.trial.Trial object at 0x7fce1c795df0>, <model_search.metadata.trial.Trial object at 0x7fce1c797850>, <model_search.metadata.trial.Trial object at 0x7fce1c797c40>, <model_search.metadata.trial.Trial object at 0x7fce1c793c40>, <model_search.metadata.trial.Trial object at 0x7fce1c794850>, <model_search.metadata.trial.Trial object at 0x7fce1c794c40>, <model_search.metadata.trial.Trial object at 0x7fce24371850>, <model_search.metadata.trial.Trial object at 0x7fce24359850>, <model_search.metadata.trial.Trial object at 0x7fce24359c40>, <model_search.metadata.trial.Trial object at 0x7fce24372850>, <model_search.metadata.trial.Trial object at 0x7fce24372c40>, <model_search.metadata.trial.Trial object at 0x7fce2434ac40>, <model_search.metadata.trial.Trial object at 0x7fce24337850>, <model_search.metadata.trial.Trial object at 0x7fce24337c40>])}
I0831 18:27:29.827618 140527906674496 coordinate_descent.py:63] Maximal depth allowed: 4
I0831 18:27:29.827801 140527906674496 search_candidate_generator.py:123] Creating new architecture:
I0831 18:27:29.827841 140527906674496 search_candidate_generator.py:124] [82 81]
I0831 18:27:29.830933 140527906674496 phoenix.py:406] {'search_generator': [<model_search.architecture.tower.Tower object at 0x7fce24351d90>]}
I0831 18:27:30.007734 140527906674496 run_config.py:549] TF_CONFIG environment variable: {'model_dir': '/tmp/run_example/tuner-1/24', 'session_master': ''}
I0831 18:27:30.007827 140527906674496 run_config.py:985] Using model_dir in TF_CONFIG: /tmp/run_example/tuner-1/24
I0831 18:27:30.243517 140527906674496 estimator.py:1175] Done calling model_fn.
I0831 18:27:30.244107 140527906674496 basic_session_run_hooks.py:558] Create CheckpointSaverHook.
I0831 18:27:30.344659 140527906674496 monitored_session.py:243] Graph was finalized.
2022-08-31 18:27:30.344949: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345355: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345549: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345726: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:30.345873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21644 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6
I0831 18:27:30.390913 140527906674496 session_manager.py:527] Running local_init_op.
I0831 18:27:30.404366 140527906674496 session_manager.py:530] Done running local_init_op.
I0831 18:27:30.648690 140527906674496 basic_session_run_hooks.py:628] Calling checkpoint listeners before saving checkpoint 0...
I0831 18:27:30.649056 140527906674496 basic_session_run_hooks.py:633] Saving checkpoints for 0 into /tmp/run_example/tuner-1/24/model.ckpt.
I0831 18:27:30.703323 140527906674496 basic_session_run_hooks.py:640] Calling checkpoint listeners after saving checkpoint 0...
I0831 18:27:31.163495 140527906674496 basic_session_run_hooks.py:266] loss = 1.035786, step = 0
I0831 18:27:31.420209 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 388.237
I0831 18:27:31.420408 140527906674496 basic_session_run_hooks.py:264] loss = 0.9075287, step = 100 (0.257 sec)
I0831 18:27:31.558015 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 725.577
I0831 18:27:31.558200 140527906674496 basic_session_run_hooks.py:264] loss = 1.0140789, step = 200 (0.138 sec)
I0831 18:27:31.690464 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 755.011
I0831 18:27:31.690645 140527906674496 basic_session_run_hooks.py:264] loss = 1.0821533, step = 300 (0.132 sec)
I0831 18:27:31.824359 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 746.852
I0831 18:27:31.824541 140527906674496 basic_session_run_hooks.py:264] loss = 1.1923344, step = 400 (0.134 sec)
I0831 18:27:31.954357 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 769.251
I0831 18:27:31.954540 140527906674496 basic_session_run_hooks.py:264] loss = 1.2966073, step = 500 (0.130 sec)
I0831 18:27:32.083554 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 773.977
I0831 18:27:32.083731 140527906674496 basic_session_run_hooks.py:264] loss = 1.0678508, step = 600 (0.129 sec)
I0831 18:27:32.211544 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 781.295
I0831 18:27:32.211720 140527906674496 basic_session_run_hooks.py:264] loss = 1.1640563, step = 700 (0.128 sec)
I0831 18:27:32.342442 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 763.951
I0831 18:27:32.342617 140527906674496 basic_session_run_hooks.py:264] loss = 1.2269, step = 800 (0.131 sec)
I0831 18:27:32.476507 140527906674496 basic_session_run_hooks.py:717] global_step/sec: 745.914
I0831 18:27:32.476685 140527906674496 basic_session_run_hooks.py:264] loss = 1.2129242, step = 900 (0.134 sec)
I0831 18:27:32.605798 140527906674496 basic_session_run_hooks.py:628] Calling checkpoint listeners before saving checkpoint 1000...
I0831 18:27:32.605903 140527906674496 basic_session_run_hooks.py:633] Saving checkpoints for 1000 into /tmp/run_example/tuner-1/24/model.ckpt.
I0831 18:27:32.649677 140527906674496 basic_session_run_hooks.py:640] Calling checkpoint listeners after saving checkpoint 1000...
I0831 18:27:32.660749 140527906674496 estimator.py:361] Loss for final step: 1.1485888.
I0831 18:27:32.678622 140527906674496 estimator.py:1173] Calling model_fn.
I0831 18:27:32.678768 140527906674496 phoenix.py:355] <tensorflow_estimator.python.estimator.run_config.RunConfig object at 0x7fce0074fdf0>
I0831 18:27:32.678819 140527906674496 phoenix.py:356] /tmp/run_example/tuner-1/24
I0831 18:27:32.699427 140527906674496 controller.py:160] trial id: 24
I0831 18:27:32.699542 140527906674496 controller.py:239] intermix ensemble search mode
I0831 18:27:32.704100 140527906674496 phoenix.py:387] {'search_generator': GeneratorWithTrials(instance=<model_search.generators.search_candidate_generator.SearchCandidateGenerator object at 0x7fce26aa6280>, relevant_trials=[<model_search.metadata.trial.Trial object at 0x7fce242128e0>, <model_search.metadata.trial.Trial object at 0x7fce24212490>, <model_search.metadata.trial.Trial object at 0x7fce24348b80>, <model_search.metadata.trial.Trial object at 0x7fce1c7a7820>, <model_search.metadata.trial.Trial object at 0x7fce1c27c220>, <model_search.metadata.trial.Trial object at 0x7fce25e36b20>, <model_search.metadata.trial.Trial object at 0x7fce25e361c0>, <model_search.metadata.trial.Trial object at 0x7fce240dff70>, <model_search.metadata.trial.Trial object at 0x7fce25dbe3d0>, <model_search.metadata.trial.Trial object at 0x7fce25fe2e20>, <model_search.metadata.trial.Trial object at 0x7fce240e4160>, <model_search.metadata.trial.Trial object at 0x7fce25fe2580>, <model_search.metadata.trial.Trial object at 0x7fce1c36ea60>, <model_search.metadata.trial.Trial object at 0x7fce1c0fed60>, <model_search.metadata.trial.Trial object at 0x7fce1c36ee80>, <model_search.metadata.trial.Trial object at 0x7fce1c0feeb0>, <model_search.metadata.trial.Trial object at 0x7fce25f3f5b0>, <model_search.metadata.trial.Trial object at 0x7fce25f3f3d0>, <model_search.metadata.trial.Trial object at 0x7fce2422fac0>])}
I0831 18:27:32.704649 140527906674496 base_tower_generator.py:99] Building from existing checkpoint.
I0831 18:27:32.711285 140527906674496 phoenix.py:406] {'search_generator': [<model_search.architecture.tower.Tower object at 0x7fce24212a90>]}
I0831 18:27:32.773150 140527906674496 run_config.py:549] TF_CONFIG environment variable: {'model_dir': '/tmp/run_example/tuner-1/24', 'session_master': ''}
I0831 18:27:32.773260 140527906674496 run_config.py:985] Using model_dir in TF_CONFIG: /tmp/run_example/tuner-1/24
I0831 18:27:32.929626 140527906674496 estimator.py:1175] Done calling model_fn.
I0831 18:27:32.940950 140527906674496 evaluation.py:250] Starting evaluation at 2022-08-31T18:27:32
I0831 18:27:32.983377 140527906674496 monitored_session.py:243] Graph was finalized.
2022-08-31 18:27:32.983673: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.983907: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.984069: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.984265: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.984439: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:32.984585: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21644 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6
I0831 18:27:32.984675 140527906674496 saver.py:1395] Restoring parameters from /tmp/run_example/tuner-1/24/model.ckpt-1000
I0831 18:27:33.023914 140527906674496 session_manager.py:527] Running local_init_op.
I0831 18:27:33.039376 140527906674496 session_manager.py:530] Done running local_init_op.
I0831 18:27:33.158018 140527906674496 evaluation.py:269] Inference Time : 0.21697s
I0831 18:27:33.159124 140527906674496 evaluation.py:271] Finished evaluation at 2022-08-31-18:27:33
I0831 18:27:33.159208 140527906674496 estimator.py:2083] Saving dict for global step 1000: accuracy = 0.3, auc_pr = 0.48401165, auc_roc = 7.142855e-08, global_step = 1000, loss = 0.97662735, num_parameters = 86
I0831 18:27:33.209768 140527906674496 estimator.py:2143] Saving 'checkpoint_path' summary for global step 1000: /tmp/run_example/tuner-1/24/model.ckpt-1000
I0831 18:27:33.210166 140527906674496 phoenix.py:136] Saving the following evaluation dictionary.
I0831 18:27:33.210225 140527906674496 phoenix.py:137] {'accuracy': 0.30000001192092896, 'auc_pr': 0.4840116500854492, 'auc_roc': 7.142855196207165e-08, 'loss': 0.9766273498535156, 'num_parameters': 86, 'global_step': 1000}
I0831 18:27:33.210270 140527906674496 ml_metadata_db.py:161] Storing the following evaluation dictionary,
I0831 18:27:33.210300 140527906674496 ml_metadata_db.py:162] {'accuracy': 0.30000001192092896, 'auc_pr': 0.4840116500854492, 'auc_roc': 7.142855196207165e-08, 'loss': 0.9766273498535156, 'num_parameters': 86, 'global_step': 1000}
I0831 18:27:33.210334 140527906674496 ml_metadata_db.py:163] For the model in the following model dictionary,
I0831 18:27:33.210361 140527906674496 ml_metadata_db.py:164] /tmp/run_example/tuner-1/24
I0831 18:27:33.238589 140527906674496 estimator.py:1173] Calling model_fn.
I0831 18:27:33.238721 140527906674496 phoenix.py:355] <tensorflow_estimator.python.estimator.run_config.RunConfig object at 0x7fce0074fdf0>
I0831 18:27:33.238767 140527906674496 phoenix.py:356] /tmp/run_example/tuner-1/24
I0831 18:27:33.259187 140527906674496 controller.py:160] trial id: 24
I0831 18:27:33.259282 140527906674496 controller.py:239] intermix ensemble search mode
I0831 18:27:33.262947 140527906674496 phoenix.py:387] {'search_generator': GeneratorWithTrials(instance=<model_search.generators.search_candidate_generator.SearchCandidateGenerator object at 0x7fce26aa6280>, relevant_trials=[<model_search.metadata.trial.Trial object at 0x7fce25dbcd90>, <model_search.metadata.trial.Trial object at 0x7fce1c46ff40>, <model_search.metadata.trial.Trial object at 0x7fce25dbcbe0>, <model_search.metadata.trial.Trial object at 0x7fce25dbce50>, <model_search.metadata.trial.Trial object at 0x7fce1c46f8b0>, <model_search.metadata.trial.Trial object at 0x7fce1c44cf70>, <model_search.metadata.trial.Trial object at 0x7fce1c44c160>, <model_search.metadata.trial.Trial object at 0x7fce1c29ffd0>, <model_search.metadata.trial.Trial object at 0x7fce25f033d0>, <model_search.metadata.trial.Trial object at 0x7fce1c29ff10>, <model_search.metadata.trial.Trial object at 0x7fce25f3cb80>, <model_search.metadata.trial.Trial object at 0x7fce25f14cd0>, <model_search.metadata.trial.Trial object at 0x7fce00730bb0>, <model_search.metadata.trial.Trial object at 0x7fce1c10fd60>, <model_search.metadata.trial.Trial object at 0x7fce00730b20>, <model_search.metadata.trial.Trial object at 0x7fce00730d00>, <model_search.metadata.trial.Trial object at 0x7fce1c6599d0>, <model_search.metadata.trial.Trial object at 0x7fce1c659580>, <model_search.metadata.trial.Trial object at 0x7fce1c45a9a0>, <model_search.metadata.trial.Trial object at 0x7fce25e47550>])}
I0831 18:27:33.263211 140527906674496 base_tower_generator.py:99] Building from existing checkpoint.
I0831 18:27:33.267359 140527906674496 phoenix.py:406] {'search_generator': [<model_search.architecture.tower.Tower object at 0x7fce1c3d53a0>]}
I0831 18:27:33.322653 140527906674496 run_config.py:549] TF_CONFIG environment variable: {'model_dir': '/tmp/run_example/tuner-1/24', 'session_master': ''}
I0831 18:27:33.322746 140527906674496 run_config.py:985] Using model_dir in TF_CONFIG: /tmp/run_example/tuner-1/24
I0831 18:27:33.324692 140527906674496 estimator.py:1175] Done calling model_fn.
I0831 18:27:33.324922 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Classify: None
I0831 18:27:33.324964 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Regress: None
I0831 18:27:33.324995 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Predict: ['serving_default']
I0831 18:27:33.325023 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Train: None
I0831 18:27:33.325047 140527906674496 export_utils.py:166] Signatures INCLUDED in export for Eval: None
2022-08-31 18:27:33.325256: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.325487: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.325648: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.325842: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.326005: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-31 18:27:33.326149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21644 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6
I0831 18:27:33.340039 140527906674496 saver.py:1395] Restoring parameters from /tmp/run_example/tuner-1/24/model.ckpt-1000
I0831 18:27:33.352076 140527906674496 builder_impl.py:669] Assets added to graph.
I0831 18:27:33.352158 140527906674496 builder_impl.py:463] No assets to write.
I0831 18:27:33.372737 140527906674496 builder_impl.py:428] SavedModel written to: /tmp/run_example/tuner-1/24/saved_model/temp-1661938053/saved_model.pb
I0831 18:27:33.373998 140527906674496 oss_trainer_lib.py:278] Evaluation results: {'accuracy': 0.3, 'auc_pr': 0.48401165, 'auc_roc': 7.142855e-08, 'loss': 0.97662735, 'num_parameters': 86, 'global_step': 1000}
Traceback (most recent call last):
File "test.py", line 26, in
trainer.try_models(
File "/mnt/hdd1/model_search_220825/model_search/model_search/single_trainer.py", line 78, in try_models
while oss_trainer_lib.run_parameterized_train_and_eval(
File "/mnt/hdd1/model_search_220825/model_search/model_search/oss_trainer_lib.py", line 340, in run_parameterized_train_and_eval
trial = oracle.create_trial(tuner_id)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/engine/oracle.py", line 189, in create_trial
response = self.populate_space(trial_id)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/tuners/bayesian.py", line 209, in populate_space
self.gpr.fit(x, y)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/tuners/bayesian.py", line 83, in fit
self._alpha_vector = cho_solve(self._l_matrix, self._y_train)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/tuners/bayesian.py", line 31, in cho_solve
y = solve_triangular(l_matrix, b.reshape(-1, 1), lower=True)
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/keras_tuner/tuners/bayesian.py", line 24, in solve_triangular
return tf.linalg.triangular_solve(a, b, lower=lower).numpy()
File "/home/nagakura/anaconda3.8.8_pytorch/envs/model_search_220825/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 513, in getattr
self.getattribute(name)
AttributeError: 'Tensor' object has no attribute 'numpy'
Note
Getting Started has number_models=200, but the result is the same.
If I set number_models to less than 20, the error does not occur. is there a limit of 20 for number_models?