MSMQ over HTTP messages stay in outgoing queue

Discussion:

(too old to reply)

r***@jonassoftware.com

2009-01-12 16:00:17 UTC

I am running into a problem where MSMQ over HTTP is not working on a
specific server. We have multiple development environments that work
but it’s failing in our QA environment.
The symptom is that the sender’s message stays in the outgoing queue
and never processes. If I try to hit the website directly through IE,
all I get is a generic HTTP 500 error. So I seem to be able to reach
the server, but then the message just disappears into the ether
without any errors or logs.
Notes:
• Windows 2003, so its MSMQ 3.0
• I have confirmed MSMQ is properly setup by calling the queue
directly using MSMQFirstAidKit. So it does seem to be specific to the
MSMQ HTTP component.
• There doesn’t seem to be anything in the IIS logs to indicate the
message arrived. So it almost seems like its being blocked and/or
processed by something else. At one point, I pointed it at another
website and it did show up in the log, so I am fairly certain it is
getting to the server. For example:
2009-01-10 02:17:38 W3SVC1 S261207DC2EW02 192.168.254.97 POST /msmq/
private$/club - 555 - 192.168.1.206 HTTP/1.1 - - - 192.168.254.97 404
0 64 0 3461 3343
• There aren’t any errors in the Event Viewer that seems to be
related.
• I can’t see any lockdown tools (i.e. URLScan, firewalls, anti-virus)
that might be conflicting. There aren’t any hardware firewalls
between the two servers that could be stopping the message.
• There is software on there that is not on the other environments.
ColdFusion and httpZip which could be conflicting. The problem is
that I can’t remove either one without breaking the rest of the site.
So I have to figure out how to work around it.
• The default website is on port 555. So when you install it through
Windows Components, it installs it there. Ideally, I would also like
to move it the other website on port 80. I’ve tried to move it using
various techniques with no effect:
o Creating a virtual directory that has the MQISE.dll setup
o Going directly into the metabase and moving the entry into the
other site.
So I am at the point where I am completely stumped on how to get it
working. Is there anything I could do to trace through the issue? I
was going to install some sort of HTTP tracing utility next and see
what that returns.
Is there any other type of logging or techniques people can think of
to solve this problem?

John Breakwell (MSFT)

2009-01-12 16:33:21 UTC

Permalink

Hi

"If I try to hit the website directly through IE, all I get is a generic
HTTP 500 error."
I wouldn't bother troubleshooting MSMQ until you can browse the web server
without error.
Are you browsing to the website or browing to the MSMQ queue path?
You will get an error if you try to browse the MSMQ queue path using IE as
IE is not sending data that looks like a message.

"• I have confirmed MSMQ is properly setup by calling the queue
directly using MSMQFirstAidKit. So it does seem to be specific to the
MSMQ HTTP component."
All respect to Cogin but their MSMQFirstAidKit is not going to confirm that
MSMQ is properly set up. The tools in their kit allow you to manipulate
messages which does not mean MSMQ is set up correctly. To get a detailed
health check them you need to generate some TMQ reports:

887220 Description of TMQTools utilities for Microsoft Message Queuing
http://support.microsoft.com/default.aspx?scid=kb;EN-US;887220

I've collected a few blog posts together that may help you get off the
ground:
http://blogs.msdn.com/johnbreakwell/archive/tags/MSMQ+over+HTTP/default.aspx

Cheers
John Breakwell (MSFT)

Randar Puust

2009-01-12 16:56:01 UTC

Permalink

Hello John,

• In regards to the HTTP 500 error, I can browse to every other page on the
web server. In the past, I have used this just to make sure there isn't
something like URLScan that is blocking the message before it gets to the
HTTP component. For example, I just hit a server which works and it returned
a 403.2. I know it is not a way to confirm its working, but at least it
tells me that a firewall isn't blocking it. In this case, I can hit an HTML
page in the root directory, but as soon as I do anything below the MSMQ
folder, it fails.
• Thanks for the link to the TMQTools. That is my next angle. Not sure if
it matters, but I should have mentioned that I am running in Workgroup mode.
Some of the tools (i.e. TMQ Site) probably don't apply, but I'm willing to
try anything at this point.
• I actually found your blog this morning (which is an impressive collection
of posts!) but none of the scenarios seem to apply. I'm using IP address as
opposed to the name and I can't think of any reason the ack wouldn't return
to the server, etc. Although I am going to change the code to use the Dead
Letter Queue and see if that reveals anything.

Thanks for the help.

Post by John Breakwell (MSFT)
Hi
"If I try to hit the website directly through IE, all I get is a generic
HTTP 500 error."
I wouldn't bother troubleshooting MSMQ until you can browse the web server
without error.
Are you browsing to the website or browing to the MSMQ queue path?
You will get an error if you try to browse the MSMQ queue path using IE as
IE is not sending data that looks like a message.
" I have confirmed MSMQ is properly setup by calling the queue
directly using MSMQFirstAidKit. So it does seem to be specific to the
MSMQ HTTP component."
All respect to Cogin but their MSMQFirstAidKit is not going to confirm that
MSMQ is properly set up. The tools in their kit allow you to manipulate
messages which does not mean MSMQ is set up correctly. To get a detailed
887220 Description of TMQTools utilities for Microsoft Message Queuing
http://support.microsoft.com/default.aspx?scid=kb;EN-US;887220
I've collected a few blog posts together that may help you get off the
http://blogs.msdn.com/johnbreakwell/archive/tags/MSMQ+over+HTTP/default.aspx
Cheers
John Breakwell (MSFT)

Randar Puust

2009-01-12 17:30:00 UTC

Permalink

I ran the TMQTools. The only thing that seemed odd, was it having issues
with some registry keys. But they appear to just be warnings, except the
last one is a bit strange.

!!! Warning: Failed to get value of registry key:
SOFTWARE\Microsoft\MSMQ\Parameters\MachineCache\MachineJournalQuota, return
value: 2
!!! Warning: Failed to get value of registry key:
SOFTWARE\Microsoft\MSMQ\Parameters\MachineCache\MachineQuota, return value: 2
!!! Warning: Failed to get value of registry key:
SOFTWARE\Microsoft\MSMQ\Parameters\Security\SecureDSCommunication, return
value: 2
!!! Warning: Failed to find all appropriate entries in the registry
--- Going to Find unknown registry values
Value HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters\SetupStatus: 0x0
(of type REG_DWORD)
Value HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters\MsmqRootPath:
C:\WINNT\system32\msmq (of type REG_SZ)
Value HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters\DsEnvironment:
0x2 (of type REG_DWORD)
!!! Warning: Failed to find the correct type of the key
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters\Workgroup, the key is
of type:4 and should be of type: 0

It is of type D_WORD and is set to 1. This is the same as all of the other
servers. Is it just confused because I am actually running in Workgroup mode?

Randar

Post by Randar Puust
Hello John,
• In regards to the HTTP 500 error, I can browse to every other page on the
web server. In the past, I have used this just to make sure there isn't
something like URLScan that is blocking the message before it gets to the
HTTP component. For example, I just hit a server which works and it returned
a 403.2. I know it is not a way to confirm its working, but at least it
tells me that a firewall isn't blocking it. In this case, I can hit an HTML
page in the root directory, but as soon as I do anything below the MSMQ
folder, it fails.
• Thanks for the link to the TMQTools. That is my next angle. Not sure if
it matters, but I should have mentioned that I am running in Workgroup mode.
Some of the tools (i.e. TMQ Site) probably don't apply, but I'm willing to
try anything at this point.
• I actually found your blog this morning (which is an impressive collection
of posts!) but none of the scenarios seem to apply. I'm using IP address as
opposed to the name and I can't think of any reason the ack wouldn't return
to the server, etc. Although I am going to change the code to use the Dead
Letter Queue and see if that reveals anything.
Thanks for the help.

Randar Puust

2009-01-13 07:41:01 UTC

Permalink

I was finally to get past my issues. It was not a single issue, but a
combination of a few. So I thought I would share my findings.
• The MSMQ HTTP component has to be on the default website. Although it can
be on a different port (in our case 555), it would not work unless it’s on
the Default Website (i.e. 1).
• The known issue with the local RPC call and finding its own name
(http://support.microsoft.com/default.aspx?scid=kb;EN-US;952571). Thanks
John for posting the reference to that.
• Because we are running on a non-standard port, you have to also change the
code so that it does not ignore the message if the ports do not match in the
EndPoints. i.e. [ServiceBehavior(AddressFilterMode = AddressFilterMode.Any)]

A few other tricks that really helped me solve the last EndPoint issue.
• Make sure you log the ServiceHost.UnknownMessageReceived event. I wasn’t
so it wasn’t returning errors when the message was being rejected.
• Use the WCF trace, which showed me the AddressFilter mismatch error. i.e.

<system.diagnostics>
<sources>
<source name="System.ServiceModel"
switchValue="Information, ActivityTracing"
propagateActivity="true">
<listeners>
<add name="traceListener"
type="System.Diagnostics.XmlWriterTraceListener"
initializeData= "E:\Logs\WCFTraces.txt" />
</listeners>
</source>
</sources>
</system.diagnostics>

Cheers,
Randar

Post by Randar Puust
I ran the TMQTools. The only thing that seemed odd, was it having issues
with some registry keys. But they appear to just be warnings, except the
last one is a bit strange.
SOFTWARE\Microsoft\MSMQ\Parameters\MachineCache\MachineJournalQuota, return
value: 2
SOFTWARE\Microsoft\MSMQ\Parameters\MachineCache\MachineQuota, return value: 2
SOFTWARE\Microsoft\MSMQ\Parameters\Security\SecureDSCommunication, return
value: 2
!!! Warning: Failed to find all appropriate entries in the registry
--- Going to Find unknown registry values
Value HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters\SetupStatus: 0x0
(of type REG_DWORD)
C:\WINNT\system32\msmq (of type REG_SZ)
0x2 (of type REG_DWORD)
!!! Warning: Failed to find the correct type of the key
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters\Workgroup, the key is
of type:4 and should be of type: 0
It is of type D_WORD and is set to 1. This is the same as all of the other
servers. Is it just confused because I am actually running in Workgroup mode?
Randar

John Breakwell (MSFT)

2009-01-13 10:06:49 UTC

Permalink

Hi Randar

Thanks very much for the excellent review of how you resolved the problem.
Much appreciated.

Cheers
John Breakwell (MSFT)

Post by Randar Puust
I was finally to get past my issues. It was not a single issue, but a
combination of a few. So I thought I would share my findings.
• The MSMQ HTTP component has to be on the default website. Although it
can
be on a different port (in our case 555), it would not work unless it’s on
the Default Website (i.e. 1).
• The known issue with the local RPC call and finding its own name
(http://support.microsoft.com/default.aspx?scid=kb;EN-US;952571). Thanks
John for posting the reference to that.
• Because we are running on a non-standard port, you have to also change
the
code so that it does not ignore the message if the ports do not match in the
EndPoints. i.e. [ServiceBehavior(AddressFilterMode =
AddressFilterMode.Any)]
A few other tricks that really helped me solve the last EndPoint issue.
• Make sure you log the ServiceHost.UnknownMessageReceived event. I wasn’t
so it wasn’t returning errors when the message was being rejected.
• Use the WCF trace, which showed me the AddressFilter mismatch error.
i.e.
<system.diagnostics>
<sources>
<source name="System.ServiceModel"
switchValue="Information, ActivityTracing"
propagateActivity="true">
<listeners>
<add name="traceListener"
type="System.Diagnostics.XmlWriterTraceListener"
initializeData= "E:\Logs\WCFTraces.txt" />
</listeners>
</source>
</sources>
</system.diagnostics>
Cheers,
Randar

Post by Randar Puust
I ran the TMQTools. The only thing that seemed odd, was it having issues
with some registry keys. But they appear to just be warnings, except the
last one is a bit strange.
SOFTWARE\Microsoft\MSMQ\Parameters\MachineCache\MachineJournalQuota, return
value: 2
SOFTWARE\Microsoft\MSMQ\Parameters\MachineCache\MachineQuota, return value: 2
SOFTWARE\Microsoft\MSMQ\Parameters\Security\SecureDSCommunication, return
value: 2
!!! Warning: Failed to find all appropriate entries in the registry
--- Going to Find unknown registry values
Value HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters\SetupStatus: 0x0
(of type REG_DWORD)
C:\WINNT\system32\msmq (of type REG_SZ)
Value
0x2 (of type REG_DWORD)
!!! Warning: Failed to find the correct type of the key
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters\Workgroup, the key is
of type:4 and should be of type: 0
It is of type D_WORD and is set to 1. This is the same as all of the other
servers. Is it just confused because I am actually running in Workgroup mode?
Randar

Post by Randar Puust
Hello John,
• In regards to the HTTP 500 error, I can browse to every other page on
the
web server. In the past, I have used this just to make sure there isn't
something like URLScan that is blocking the message before it gets to the
HTTP component. For example, I just hit a server which works and it returned
a 403.2. I know it is not a way to confirm its working, but at least it
tells me that a firewall isn't blocking it. In this case, I can hit an HTML
page in the root directory, but as soon as I do anything below the MSMQ
folder, it fails.
• Thanks for the link to the TMQTools. That is my next angle. Not
sure if
it matters, but I should have mentioned that I am running in Workgroup mode.
Some of the tools (i.e. TMQ Site) probably don't apply, but I'm willing to
try anything at this point.
• I actually found your blog this morning (which is an impressive
collection
of posts!) but none of the scenarios seem to apply. I'm using IP address as
opposed to the name and I can't think of any reason the ack wouldn't return
to the server, etc. Although I am going to change the code to use the Dead
Letter Queue and see if that reveals anything.
Thanks for the help.