Hi Lissy, your suggestion worked like a charm 
but the “allow_growth” flag in config does not work for some reason. However, I looked into the config.py in training/deepspeech_training/utils and set the “per_process_gpu_memory_fraction=.1”, this limits tensorflow to use only 10% of the GPU.
Pasting the exact code snippet(from config.py) that worked for me:
c.session_config = tfv1.ConfigProto(allow_soft_placement=True,
log_device_placement=FLAGS.log_placement,
inter_op_parallelism_threads=FLAGS.inter_op_parallelism_threads,
intra_op_parallelism_threads=FLAGS.intra_op_parallelism_threads, gpu_options=tfv1.GPUOptions(per_process_gpu_memory_fraction=0.1))
I think in the next release, you should add a flag for the user to set how much GPU they want DS to consume.
PS: I am using DS==0.7.1
Hope it helps more users who are looking for something similar!