This a lost battle. Plugins with any latency are not meant for live/input monitor use. They are meant for Mixing. Inserting a plugin that buffers audio anywhere in the project will slow all other audio by the amount of latency the plugin requests.