Packet pool is useful to avoid work and additional allocations when switching files, going to idle should free the pool as those packets may not be used for some time. Fixes: #17525