How to change batch size?

brian2 · April 29, 2020, 2:05pm

I have a GPU with 6GB memory, and received out of memory error. I reduced the batch size in the config file by

import json
from utils.generic_utils import load_config
CONFIG = load_config('config.json')
CONFIG['datasets'][0]['path'] = '../LJSpeech-1.1/'
CONFIG['output_path'] = '../'
CONFIG['batch_size'] = 8
CONFIG['eval_batch_size'] = 8
with open('config.json', 'w') as fp:
    json.dump(CONFIG, fp)

but still, I get the memory error

RuntimeError: CUDA out of memory. Tried to allocate 106.00 MiB (GPU 0; 5.93 GiB total capacity; 3.20 GiB already allocated; 25.44 MiB free; 3.47 GiB reserved in total by PyTorch)

mrthorstenm · April 30, 2020, 6:38pm

Seems you’ve overwritten value in code instead of using configfile value.

github.com

mozilla/TTS/blob/2e2221f146f1ca301a3b2b547d8f26b9009676de/config.json#L53




// DISTRIBUTED TRAINING

"distributed":{

    "backend": "nccl",

    "url": "tcp:\/\/localhost:54321"

},



"reinit_layers": [],    // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.



// TRAINING

"batch_size": 32,       // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.

"eval_batch_size":16,   

"r": 7,                 // Number of decoder frames to predict per iteration. Set the initial values if gradual training is enabled.  

"gradual_training": [[0, 7, 64], [1, 5, 64], [50000, 3, 32], [130000, 2, 32], [290000, 1, 32]], //set gradual training steps [first_step, r, batch_size]. If it is null, gradual training is disabled. For Tacotron, you might need to reduce the 'batch_size' as you proceeed.

"loss_masking": true,         // enable / disable loss masking against the sequence padding.



// VALIDATION

"run_eval": true,

"test_delay_epochs": 10,  //Until attention is aligned, testing only wastes computation time.

"test_sentences_file": null,  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.

I’m in doubt that reaching alignment with a value of 8 is possible.

brian2 · April 30, 2020, 7:51pm

here says about 9.5GB GPU memory is needed for standard training (batch_size 32), but it is possible to train with 4GB GPU memory with a batch_size 8-12.

I have 6GB GPU memory, and I think I have enough memory for batch_size 16.

What is the standard way to change the training batch_size?

mrthorstenm · April 30, 2020, 8:16pm

config.json in repo root directory is the right place. I’m just confused on the code snipplet on your first post which includes code not config.

brian2 · April 30, 2020, 8:45pm

I am following the instruction given here.

The config file is re-created, and my code attempts to set new batch_size. I can verify that the batch_size in the generated config file is changed, but I still get the memory errors as mentioned above.

mrthorstenm · April 30, 2020, 9:33pm

Juypter Notebook is not needed.
Try starting with readme on mozilla tts repo at https://github.com/mozilla/tts