|
楼主 |
发表于 2024-3-16 23:27:03
|
显示全部楼层
训练点击后出错了,这是什么情况?
Error: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating a buffer of 205521156 bytes
[[node src_dst_opt_1/random_uniform_8/RandomUniform (defined at F:\DeepFaceLab\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\framework\ops.py:1762) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[concat_6/concat/_121]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating a buffer of 205521156 bytes
[[node src_dst_opt_1/random_uniform_8/RandomUniform (defined at F:\DeepFaceLab\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\framework\ops.py:1762) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations.
0 derived errors ignored.
Original stack trace for 'src_dst_opt_1/random_uniform_8/RandomUniform':
File "threading.py", line 884, in _bootstrap
File "threading.py", line 916, in _bootstrap_inner
File "threading.py", line 864, in run
File "F:\DeepFaceLab\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
debug=debug)
File "F:\DeepFaceLab\_internal\DeepFaceLab\models\ModelBase.py", line 193, in __init__
self.on_initialize()
File "F:\DeepFaceLab\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 327, in on_initialize
self.src_dst_opt.initialize_variables (self.src_dst_trainable_weights, vars_on_cpu=optimizer_vars_on_cpu, lr_dropout_on_cpu=self.options['lr_dropout']=='cpu')
File "F:\DeepFaceLab\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 44, in initialize_variables
lr_rnds = [ nn.random_binomial( v.shape, p=self.lr_dropout, dtype=v.dtype) for v in trainable_weights ]
File "F:\DeepFaceLab\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 44, in <listcomp>
lr_rnds = [ nn.random_binomial( v.shape, p=self.lr_dropout, dtype=v.dtype) for v in trainable_weights ]
File "F:\DeepFaceLab\_internal\DeepFaceLab\core\leras\ops\__init__.py", line 206, in random_binomial
random_ops.random_uniform(shape, dtype=tf.float16, seed=seed) < p,
File "F:\DeepFaceLab\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\random_ops.py", line 245, in random_uniform
rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2)
File "F:\DeepFaceLab\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\gen_random_ops.py", line 860, in random_uniform
name=name)
File "F:\DeepFaceLab\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "F:\DeepFaceLab\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "F:\DeepFaceLab\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3371, in create_op
attrs, op_def, compute_device) |
|