is it possible to change flushing interval of outgoing messages

[Morten Herman Langkjaer]

We have a setup where 25 or more clients send messages to a server, that
dispactehes/copies the messages to each of the other clients, that are
unaware of each others pressence. When monitoring our system, it seems that
MSMQ in stressed periods, start to stack up a few messages locally, and thes
sending them as a bulk message or something like that. in the beginning it
may be e.g. 4 messages per outgoing queue times 25 clients, resulting in 100
outgoing messages being gathered before being flushed to the network. After a
while, possibly hours of stress, the number of queued up messages rise to be
10000 (400) messages per client. The odd thing is that the number is not
constant, it may go from 0 to 400, back to 0 then 1000 then 0 then 2000 etc.
so Message queueing catches up, but somehow seems to wait a little before
sending away the messages.

This may seem reasonable as a design decission to awoid chatty talk over the
network connection. However, our setup guarantees that the network connection
is stable, and that the network transfer ofg the data amount will not be the
problem. Is there anyway to configure that message querue should not stack up
messages temporarily, or to change the threshold so the amount of messages
gatherd is smaller? our clients suffer from the penalty it is not to have the
messages immediately, and somtimes break over stress for having toi handle
many messages in a short amount of time due to some other shared ressources.,

[John Breakwell (MSFT)]

Hi Morten,

MSMQ doesn't delay sending messages (except when a connection has been
dropped or reset, for example).
If the messages are staying in the outgoing queue then it means MSMQ hasn't
physically been able to send them yet or MSMQ is waiting on acknowledgements
before progressing.

For example, there may not be enough free threads available to handle the
different outgoing queues.
Up to 16 threads are available for handling messages although the default
value may be lower depending on how many CPUs the sending machine has.
You may want to raise this value on the server to see if it helps.
QMThreadNo
http://technet2.microsoft.com/windowsserver/en/library/6e36b98f-75c6-4c93-8bc6-3bf706612a2d1033.mspx?mfr=true

New messages can only be sent when old messages have been acknowledged.
If you are sending many messages to a busy destination then it is
concievable that the destination cannot send the Session Ack back to the
sender in time.
The default that MSMQ waits is 64 messages. Increasing this value may give
more time for the destination to acknowledge messages.
MaxUnackedPacket. Normally the destination will send a Session Ack at the
half way value (32 messages).
The sender will suspend the session if no Ack is received for 64 messages.
If the session is broken, all the unacked messages are re-sent.
http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/regentry/58682.mspx?mfr=true
Note the page says "If the queue is too small, messages might be lost."
Ignore this - it is wrong.

"However, our setup guarantees that the network connection is stable, and
that the network transfer of the data amount will not be the problem."
Although I admire your confidence in the network, have you investigated the
traffic? For example, any RESET packets?

"our clients suffer from the penalty it is not to have the messages
immediately, and sometimes break over stress for having to handle many
messages in a short amount of time due to some other shared resources."
So is the server of the right specification for the workload you are
expecting it to handle?

Cheers
John Breakwell

Post by Morten Herman Langkjaer
We have a setup where 25 or more clients send messages to a server, that
dispactehes/copies the messages to each of the other clients, that are
unaware of each others pressence. When monitoring our system, it seems that
MSMQ in stressed periods, start to stack up a few messages locally, and thes
sending them as a bulk message or something like that. in the beginning it
may be e.g. 4 messages per outgoing queue times 25 clients, resulting in 100
outgoing messages being gathered before being flushed to the network. After a
while, possibly hours of stress, the number of queued up messages rise to be
10000 (400) messages per client. The odd thing is that the number is not
constant, it may go from 0 to 400, back to 0 then 1000 then 0 then 2000 etc.
so Message queueing catches up, but somehow seems to wait a little before
sending away the messages.
This may seem reasonable as a design decission to awoid chatty talk over the
network connection. However, our setup guarantees that the network connection
is stable, and that the network transfer ofg the data amount will not be the
problem. Is there anyway to configure that message querue should not stack up
messages temporarily, or to change the threshold so the amount of messages
gatherd is smaller? our clients suffer from the penalty it is not to have the
messages immediately, and somtimes break over stress for having toi handle
many messages in a short amount of time due to some other shared ressources.,

[Morten Herman Langkjaer]

Hi John,

In regards to the number of processing threads, we will try to adjust this
to a higher number, in our next scheduled run. I read on your site that for
Win 2003 server the default is 5 x number of processors. is this meant as 5x
number of physical proccesors in processor sockets on the motherboard, or is
it the number of processor kernels on the physical processor mounted on the
motherboard.?

When you say that "MSMQ is waiting on acknowledgements before progressing"
and also says, "The default that MSMQ waits is 64 messages", does that mean
that MSMQ will send up to 64 messages to a remote machine without having
acknowledgements for the messages, and then waits for the first
acknowledgement on the specific queue before sending the next message. If so,
what does this mean when we run a setup using only transactional queues,
where the order of the messages are guaranteed to be the same as when sending
the messages. Does it mean anything for performance, if we change the value
of the number of unacknowledged messages?

regardung network traffic, when you refer to RESET packets, is that at MSMQ
specific that is sent, and do you have any suggestions on how to monitor this?

regarding the clients breaking under stress, you ask if the server is of the
right soecification. this is the case. What happens is that we have set som
timeouts for acknowledgements for processig of messages from queues. If a
client is not responding to an incoming queue, it is automatically
"unsunbscribed" (to accomodate scenatios where the client has died
unexpectatly, e.g. process kill). When having messages received in large
bulks instead of a continous flow, our clients may sometimes get above the
threshold limit for processing messages. The threshold can not just be raised
in our setup, due to other constraints, and end user requirements. we are
aware that our clients should process messages faster in some scenarios, but
the bulking of messages instead of a continuous flow seems to be the reason
for our clients to get behind.

Cheers
Morten Herman Langkjaer

Post by John Breakwell (MSFT)
Hi Morten,
MSMQ doesn't delay sending messages (except when a connection has been
dropped or reset, for example).
If the messages are staying in the outgoing queue then it means MSMQ hasn't
physically been able to send them yet or MSMQ is waiting on acknowledgements
before progressing.
For example, there may not be enough free threads available to handle the
different outgoing queues.
Up to 16 threads are available for handling messages although the default
value may be lower depending on how many CPUs the sending machine has.
You may want to raise this value on the server to see if it helps.
QMThreadNo
http://technet2.microsoft.com/windowsserver/en/library/6e36b98f-75c6-4c93-8bc6-3bf706612a2d1033.mspx?mfr=true
New messages can only be sent when old messages have been acknowledged.
If you are sending many messages to a busy destination then it is
concievable that the destination cannot send the Session Ack back to the
sender in time.
The default that MSMQ waits is 64 messages. Increasing this value may give
more time for the destination to acknowledge messages.
MaxUnackedPacket. Normally the destination will send a Session Ack at the
half way value (32 messages).
The sender will suspend the session if no Ack is received for 64 messages.
If the session is broken, all the unacked messages are re-sent.
http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/regentry/58682.mspx?mfr=true
Note the page says "If the queue is too small, messages might be lost."
Ignore this - it is wrong.
"However, our setup guarantees that the network connection is stable, and
that the network transfer of the data amount will not be the problem."
Although I admire your confidence in the network, have you investigated the
traffic? For example, any RESET packets?
"our clients suffer from the penalty it is not to have the messages
immediately, and sometimes break over stress for having to handle many
messages in a short amount of time due to some other shared resources."
So is the server of the right specification for the workload you are
expecting it to handle?
Cheers
John Breakwell

Post by Morten Herman Langkjaer
We have a setup where 25 or more clients send messages to a server, that
dispactehes/copies the messages to each of the other clients, that are
unaware of each others pressence. When monitoring our system, it seems that
MSMQ in stressed periods, start to stack up a few messages locally, and thes
sending them as a bulk message or something like that. in the beginning it
may be e.g. 4 messages per outgoing queue times 25 clients, resulting in 100
outgoing messages being gathered before being flushed to the network. After a
while, possibly hours of stress, the number of queued up messages rise to be
10000 (400) messages per client. The odd thing is that the number is not
constant, it may go from 0 to 400, back to 0 then 1000 then 0 then 2000 etc.
so Message queueing catches up, but somehow seems to wait a little before
sending away the messages.
This may seem reasonable as a design decission to awoid chatty talk over the
network connection. However, our setup guarantees that the network connection
is stable, and that the network transfer ofg the data amount will not be the
problem. Is there anyway to configure that message querue should not stack up
messages temporarily, or to change the threshold so the amount of messages
gatherd is smaller? our clients suffer from the penalty it is not to have the
messages immediately, and somtimes break over stress for having toi handle
many messages in a short amount of time due to some other shared ressources.,

[John Breakwell (MSFT)]

Hi Morten

The number of CPUs that QMThreadNo uses for calculations should be the
number that appears to the O/S.
The number should be the logical and not physical amount of processors.
Easy test is to go to Task Manager and see how many CPU windows appear.

I've posted on the blog to cover how the Session Acks work:
http://blogs.msdn.com/johnbreakwell/archive/2008/02/09/what-does-msmq-do-when-it-doesn-t-receive-a-session-ack.aspx
Yes, MSMQ will wait if Session Acks don't arrive quickly enough to keep the
messages flowing smoothly.
If you increase this value then it means you can continue to send messages
whilst waiting for the Session Ack to arrive.
If the Session ACK still doesn't arrive before AckTimeout is reached then
all the messages in the list are re-sent.
If QMThreadNo is large when this happens then there will be a lot of
re-sent messages.
Transactional messages also use Order Acks (to confim messages are delivered
in sequence) and Storage Acks (to confirm persistent messages are written to
disk).

RESET packets are at the TCP level and are used by any network application
(for example, MSMQ) to terminate a TCP connection.
Any network monitoring software should be able to filter on these without
too much problem.

From your explanation it sounds like the machines are the right
specification for normal use but peak volumes may be too much to handle in
the time required.
There is no guarantee that tweaking the system will allow the machines to
process these peaks fast enough to beat the deadlines.

Cheers
John Breakwell

Post by Morten Herman Langkjaer
Hi John,
In regards to the number of processing threads, we will try to adjust this
to a higher number, in our next scheduled run. I read on your site that for
Win 2003 server the default is 5 x number of processors. is this meant as 5x
number of physical proccesors in processor sockets on the motherboard, or is
it the number of processor kernels on the physical processor mounted on the
motherboard.?
When you say that "MSMQ is waiting on acknowledgements before progressing"
and also says, "The default that MSMQ waits is 64 messages", does that mean
that MSMQ will send up to 64 messages to a remote machine without having
acknowledgements for the messages, and then waits for the first
acknowledgement on the specific queue before sending the next message. If so,
what does this mean when we run a setup using only transactional queues,
where the order of the messages are guaranteed to be the same as when sending
the messages. Does it mean anything for performance, if we change the value
of the number of unacknowledged messages?
regardung network traffic, when you refer to RESET packets, is that at MSMQ
specific that is sent, and do you have any suggestions on how to monitor this?
regarding the clients breaking under stress, you ask if the server is of the
right soecification. this is the case. What happens is that we have set som
timeouts for acknowledgements for processig of messages from queues. If a
client is not responding to an incoming queue, it is automatically
"unsunbscribed" (to accomodate scenatios where the client has died
unexpectatly, e.g. process kill). When having messages received in large
bulks instead of a continous flow, our clients may sometimes get above the
threshold limit for processing messages. The threshold can not just be raised
in our setup, due to other constraints, and end user requirements. we are
aware that our clients should process messages faster in some scenarios, but
the bulking of messages instead of a continuous flow seems to be the reason
for our clients to get behind.
Cheers
Morten Herman Langkjaer

Post by John Breakwell (MSFT)
Hi Morten,
MSMQ doesn't delay sending messages (except when a connection has been
dropped or reset, for example).
If the messages are staying in the outgoing queue then it means MSMQ hasn't
physically been able to send them yet or MSMQ is waiting on
acknowledgements
before progressing.
For example, there may not be enough free threads available to handle the
different outgoing queues.
Up to 16 threads are available for handling messages although the default
value may be lower depending on how many CPUs the sending machine has.
You may want to raise this value on the server to see if it helps.
QMThreadNo
http://technet2.microsoft.com/windowsserver/en/library/6e36b98f-75c6-4c93-8bc6-3bf706612a2d1033.mspx?mfr=true
New messages can only be sent when old messages have been acknowledged.
If you are sending many messages to a busy destination then it is
concievable that the destination cannot send the Session Ack back to the
sender in time.
The default that MSMQ waits is 64 messages. Increasing this value may give
more time for the destination to acknowledge messages.
MaxUnackedPacket. Normally the destination will send a Session Ack at the
half way value (32 messages).
The sender will suspend the session if no Ack is received for 64 messages.
If the session is broken, all the unacked messages are re-sent.
http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/regentry/58682.mspx?mfr=true
Note the page says "If the queue is too small, messages might be lost."
Ignore this - it is wrong.
"However, our setup guarantees that the network connection is stable, and
that the network transfer of the data amount will not be the problem."
Although I admire your confidence in the network, have you investigated the
traffic? For example, any RESET packets?
"our clients suffer from the penalty it is not to have the messages
immediately, and sometimes break over stress for having to handle many
messages in a short amount of time due to some other shared resources."
So is the server of the right specification for the workload you are
expecting it to handle?
Cheers
John Breakwell
"Morten Herman Langkjaer"

Post by Morten Herman Langkjaer
We have a setup where 25 or more clients send messages to a server, that
dispactehes/copies the messages to each of the other clients, that are
unaware of each others pressence. When monitoring our system, it seems that
MSMQ in stressed periods, start to stack up a few messages locally, and thes
sending them as a bulk message or something like that. in the beginning it
may be e.g. 4 messages per outgoing queue times 25 clients, resulting
in
100
outgoing messages being gathered before being flushed to the network. After a
while, possibly hours of stress, the number of queued up messages rise
to
be
10000 (400) messages per client. The odd thing is that the number is not
constant, it may go from 0 to 400, back to 0 then 1000 then 0 then 2000 etc.
so Message queueing catches up, but somehow seems to wait a little before
sending away the messages.
This may seem reasonable as a design decission to awoid chatty talk
over
the
network connection. However, our setup guarantees that the network connection
is stable, and that the network transfer ofg the data amount will not
be
the
problem. Is there anyway to configure that message querue should not
stack
up
messages temporarily, or to change the threshold so the amount of messages
gatherd is smaller? our clients suffer from the penalty it is not to
have
the
messages immediately, and somtimes break over stress for having toi handle
many messages in a short amount of time due to some other shared ressources.,