Subtitles can be white with black outlines, yellow, italic, stylized, or even overlapping the background. If the font is unusual or the background is noisy, OCR accuracy drops significantly.
Subtitles often need cleaning before OCR. Use ImageMagick to improve contrast, threshold, and remove background. extract hardsub from video