Help Wanted: Receive Locations Disabling Unexpectedly

A colleague of mine is running into an issue with a large number of receive locations on a single port and I would appreciate some community advice on how to resolve the issue.

He is running approx. 1400 file Receive Locations on a single Receive Port and after starting all of the locations, they slowly start to disable themselves with the following error message in the Event Log (note, the share isn’t actually called ‘sharename’ in the environment):

The receive location “33183-FTP Receive Location” with URL “\sharenameFTP33183IN*.*” is shutting down. Details:”The FILE receive location \sharenameFTP33183IN*.* exhausted the network retry attempts.”.

For reference, the BizTalk environment is based on 2 servers participating in a group (Enterprise Edition), with the 1400 receive locations using a single Host (one Host Instance per server). The receive locations are polling a Windows SMB share hosted on a separate system; the authentication credentials are confirmed as being correct.

We have increased the maximum number of NetBIOS connections from the default of 50 to 2000 (well within the limit of 65,000) after reading this blog entry and KB article 810886. We have also turned the file Receive Location polling off as per Rasmus Kristensen newsgroup entry.

Unfortunately, I’ve yet to take a look at the environment, but my initial thoughts are:

  • The host instances are exhausting their available threads / memory – creating additional Hosts and moving the Receive Locations out to multiple Receive Ports may stop this problem.
  • There are still NetBIOS connection issues – the ‘exhausted the network retry attempts’ seems to imply that this may be the case.
  • Are we exceeding the maximum number of usable Receive Locations per Receive Port? Does anyone know how many locations a port can contain?

I’ll be traveling to the client early site next week to investigate (I imagine that PerfMon and I will become close friends), however in the meantime it would be good to get any advice from the community – have I missed anything obvious here??

Any comments or suggestions are greatly welcomed.

Nick.

Advertisements

6 thoughts on “Help Wanted: Receive Locations Disabling Unexpectedly

  1. Nick,

    For what its worth, we experienced similar behavior to what you describe, but running under BizTalk 2002, running under Windows Server 2003. I do know that it was far less than the number of receive locations you are running…maybe around 120?

    I believe you are already on the right track, and I’ll try to relate the details (as much as I can remember) from our encounter. Hopefully its not a rabbit hole!

    I believe I understand that you are experiencing the behavior on both machines.

    Monitor (perfmon) the Redirector Current Commands counter. We would see this meet, or exceed, the default limit, which lead us to bump the registry setting. However, I don’t remember if it was also required to make this change on the computer that actually hosted the share as well. I would monitor both machines (BizTalk servers and file servers) to get a better picture.

    The value that we set in the registry had to be much higher than the number of receive locations, due to that is the max commands for the entire box and there are a lot of commands issued that are just part of either Windows housekeeping, or just plain outside the scope of your application(s).
    e.g. give yourself a good buffer. Again, I want to say that we made this change on both source and target machines.

    However, all that said and done, we still wound up spreading the hosting of those receive locations across additional machines. My memory doesn’t help me determine if that was the ultimate way we ‘resolved’ it, or if we did that as a ‘fail safe’.

    I have no idea if there are a max number of receive locations/port threshold being exceeded (unless it would be related to memory, etc).

    Good luck!
    Z

  2. Zach,
    Thanks for the comment, I do think there is more going on with the NetBIOS/SMB connections that my colleague is seeing and I hope to be able to get a better idea when I get some PerfMon data next week.

    To clarify my original posting, yes we are seeing the behavior on both machines and I have noticed an MSDN article that indicates there is a max. number of receive ports: 1024, which would lead me to suggest that there is also a max number of receive locations, both per port and across the whole BizTalk Group.

    Nick.

  3. I wish I had some advice to give you on the issue, but it looks like due to network or other constraints there is a max number of receive locations, and you have passed it.

    In case your colleague is using thousands of file adapters to pick up files from a hierarchy of folders, I do have a “recursive” file adapter that would cut down significantly on the number of receive locations needed. I would be happy to share it with your colleague if it would help.

  4. Hi everyone,
    Just though I would let you know that I was running into the same issue at our oganizations. We are using BTS2004 with Windows Server 2003 (On both the BTS and the file server) with about 150 file received location pointing to the same server. I made the changes as described but it didn’t help one bit. In fact we didn’t even have those registry entries on our servers.

    What I did do was (since the server we were having a problem with had FTP server already loaded) I moved about %50 of our file transports over to FTP transports and our Shutdowns have completedly stopped over the last week. I know it’s not the best solution but it is one that definitely works!!

    Good luck!
    John

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s