How to choose prefetch count value for RabbitMQ
In this article I am going to present few rules I use when selecting
prefetch count value for RabbitMQ. Those rules are based on the experience I gained when working with RabbitMQ. You should treat them as a guidance and starting point, remembering that each application, network and queue topology is different. Let’s start with quick explanation what
prefetch count is.
RabbitMQ allows consumers to specify the size of the limit of unacknowledged messages on a queue basis. This value is used to specify how many messages is send to the consumer and cached by RabbitMQ client library. Once the buffer is full the RabbitMQ will wait with delivering new messages to that consumer until it sends ACKs / NACKs. The main purpose for pre fetching messages is to optimise message delivering. The ideal situation is where there is a message delivered and waiting for the processor to be consumed.
Of course, caching messages on the consumer side has repercussions. All pre-fetched messages are removed from the queue and become invisible to other processors. Hence setting
prefetch count to a high value will skew message distribution between different consumers.
Below are rules for selecting
prefetch count depending on the consumer type.
Fast, single consumer
If there is only one consumer processing messages quickly you definitely want all messages to be pre-fetched. The optimum value will probably be something higher than 20, but setting it to anything above 50 will rarely delivery more value. Remember that pre-fetched messages are cached on the client side consuming its resources. High value will also restrict your ability to peek messages.
Multiple fast consumers
When you have multiple consumers which are processing messages fast you need to find value which keeps messages waiting for consumers as well as allows for balanced message delivery between consumers. Low value will penalise consumers as they need to wait for messages. High value will damage load balancing. I usually use anything between 20 and 30.
For slow consumers message delivery time is negligible. My advice is to set
prefetch count to 1 and let RabbitMQ load balance message between consumers.
Application crash and buffered messages
Let’s see what happens to buffered messages when the application crashes. Because RabbitMQ automatically puts all not ACKed messages back to the queue, if the application crashes then all cached messages will simply go back to the top of the queue.
Setting prefetch count
prefetch count in EasyNetQ
var bus = RabbitHutch.CreateBus("host=localhost;prefetchcount=30");
You may find more information about prefetch count in following articles: