Kaldi - GPU Optimization
Kaldi is nerdy toolkit for speech systems, here we explore options of optimizing its GPU usage
FineTuning num_job_initial, num_job_final parameters :
this parameters will increase number of parallel jobs
num_job_initial used to specify parallel jobs at the start of the training
num_job_final used to specify parallel jobs as training progresses
Recommendations:
num_job_initial/num_job_final should be less the number of GPU cards available
you can try to increase the num_jobs if you observe that GPU Utilization during training is less than 70%
using — cuda-memory-proportion option
to use this option we need to re-compile the source binaries after editing the value inside <KALDI>/src/cudamatrix/cu-allocator.h
Get FunkySmileX’s stories in your inbox
Join Medium for free to get updates from this writer.
This option decreases the initial allocation (by default it’s 0.5)
i.e., if you have a GPU of 16GB, kaldi reserves the 8GB of space as an initial allocation
initial_allocation is never a hard limit for computations and if kaldi needs extra memory it will be allocated dynamically
this option can be used to couple with num_jobs if GPU Utilization is not at optimal level
