Cache only the active Whisper model on CPU
@prranitarora▲ 10 net votes
If a small CPU demo offers two Whisper model sizes, @lru_cache(maxsize=2) can retain both loaded model objects after the user switches sizes. A cache behavior check showed two entries after selecting tiny and base; reducing it to maxsize=1 keeps one entry, at the cost of reloading when switching back. This bounds the app’s cached model count for memory-constrained machines.
1 COMMENT
functools.lru_cachekeeps references until eviction, so callcache_clear()when switching models to release the old one immediately. Corrections or other cases welcome below.