Announcing a new BizTalk 2006 Message Archiving Component

Update: The Message Archiving Component is now maintained by Atomic-Scope, visit http://www.atomic-scope.com/downloadtrial for more information.

A couple of days ago I uploaded a revised version of my BizTalk 2006 Message Archiving Component onto CodePlex – version 0.2 can be downloaded from the CodePlex website at http://www.codeplex.com/btsmsgarchcomp.

The component works with BizTalk 2006 & BizTalk 2006 R2 and allows messages to be written to the file system (either locally or over a network share) for archiving, using message context-properties to define the archive path. It is written in a streaming fashion, is designed for large message consumption and can handle Xml, binary and flat-files.

The component can be executed in either the Decode (receive pipeline) or Encode (send pipeline) stages – when used on the receive side, the original message can be archived before BizTalk has started to manipulate the message; on the send side, the final message before delivery via the send adapter can be captured. If message archiving fails, the message will still continue downstream.

As per the previous version, the archive path is constructed using ‘macros’ (e.g. %ReceivePortName%) which relate to message Context Properties. In the previous version, these macro’s were hard-coded into the component so changing or adding a new macro was painful, requiring a rebuild of the code base. The new version continues the theme of macros, however they have moved into an Xml configuration file, which can be configured at run-time, allowing any context-property (either a known BizTalk context property or a custom adapter/pipeline property) to be used. An example of the configuration file is shown below:

Macro Definition File Sample

These macros are used when defining the archive file path in the component configuration (in the BizTalk Server Admin Console) – see the screenshot below. The archive file path is constructed using the ArchiveDirectory and ArchiveFilename properties. Macros can be repeated and/or used in both properties at the same time (however I can think of scenarios where you may not wish to use say ReceivePortName in the archive filename).

Note: When configuring the component properties, you are specifing the macro name, not the context property name. This is useful when you are using a single configuration file and want to read the ReceivedFileName context property from both the FTP and FILE adapters for example (both use the same property name, but different property namespaces) – in this scenario, you would call the macro’s ‘ReceivedFileNameFILE’ and ‘ReceivedFileNameFTP’ to avoid ambiguity.

The example Macro Definition Xml file shown above details macros for the System Context Properties ReceivePortName and ReceiveLocationName. To use these macros in the pipeline component, assign them in the Archive Directory or Archive Filename property, as below.

Component Configuration - Macro Examples These macro’s (%ReceivePortName%, %ReceiveLocationName% and %MessageID%) will be expanded as a message is delivered via the port to their real-world values, writing the archived message to (for example):

..Test Xml Dasm Decode StageTest Xml Dasm Decode Stage Location364a4dee-c698-4ddd-af14-d44e66f3bd5d.xml

In the example above we’re using the %MessageID% macro to build the archive filename. This is a special macro as the value does not come from the message Context, but is taken directly from the message itself, similar to using %MessageID%.xml in a Send FILE adapter.

The OverwriteExistingFile property indicates to the component whether an existing file should be overwritten if, once the archive path has been constructed from the Context Property macros, the full archive path resolves to an existing file. If set to true, the existing file will be overwritten; if false, the new file will be written to the same directory, but with an additional GUID before the file extension.

Finally, the MacroDefinitionsFile property defines the location to the Xml configuration file detailing the macro Context Property substitutions. A different file can be used per pipeline, or a single global file can be used. A more detailed example than that shown above – which includes macro’s for commonly used Message Tracking, System and File adapter properties – is available to download.

With regards to permissions, you will need to ensure that the user running the BizTalk service has read permissions on the macro definitions Xml configuration file and read/write permissions on the root of the archive directory.

Although the component is production ready it is not yet feature complete. If you are using the previous version, can I ask you to download a copy and try it out – please let me have any feedback via the comments on this post, or via the Issue Tracker pages of the CodePlex project.

Advertisements

4 thoughts on “Announcing a new BizTalk 2006 Message Archiving Component

  1. Mikael,
    Thanks for the pointer. I’ve re-worked the component to use the ForwardOnlyEventingReadStreaem class and will be releasing an updated version 0.3 on CodePlex later today.

    Cheers, Nick.

  2. I have a question,
    Which method introduces less performance impact?
    1. Using tracking on send and receive ports?
    2. Using this component?
    Thank you for your attention.

  3. Mehdi ,
    There are several strategies that can be employed to track and archive messages, two of which you have mentioned above:

    Tracking on the Send and Receive Ports

    * Allows you to capture messages before they enter a receive port (i.e. in their native format when they come off the ‘wire’) or once they are about to be written to the Message Box;
    * Allows you to capture messages before they are sent by a Send Port (i.e. just after retrieval from the Message Box) or before they are sent (in their native format) by the adapter;
    * Allows you to capture both message bodies and message context;
    * Tracking message context properties requires much less overhead than tracking message bodies;
    * Tracked messages are viewable in the Health and Activity (HAT) tool;
    * Tracked messages are copied from the BizTalkMsgBoxxDb database to the BizTalkDTADb database every minute by the TrackedMessages_Copy_BizTalkMsgBoxDb job

    In a nutshell, using message tracking introduces a significant performance overhead, primarily because there is some much additional database work.

    BizTalk Message Archiving Pipeline Component

    * Uses a forward-only event streaming model which means that an event is placed on the message stream as it passes through the component which is them fired by later pipeline components as the stream is actually read (e.g. by another component or on writing to the Message Box); This is essentially the most efficient way of writing a pipeline component – one that gets other components to do its work for it!!!
    * Message bodies are written to the file system or UNC path directly in a spooling manner (there is no support for message context properties at present, although this is planned);
    * The component archives the raw message as a stream of bytes irrespective of what format it is in – there is no attempt to serialise it to a database etc.
    * There is no database involved, incurring network and disk I/O (on the database server) and no SQL Agent jobs to copy data from one database to the other – once the message has completed processing by the pipeline, the body will be present in your archive directory.

    I hope this has given you a rough understanding of the pro’s and con’s of the two archiving methods – my personal preference is to go for the BizTalk Message Archiving Pipeline Component because of the overhead that message tracking introduces.

    Rgds, Nick.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s