BizTalk 2006 Message Archiving Component – More Efficient Stream Handling

Update: The Message Archiving Component is now maintained by Atomic-Scope, visit http://www.atomic-scope.com/downloadtrial for more information.

I’ve re-worked the BizTalk 2006 Message Archiving Component as a true streaming pipeline component – utilising a forward only eventing stream – to more efficiently use process resources when archiving large files, as suggested by Mikael Håkansson. The updated version is on CodePlex as version 0.3.

Writing of the archive file is now accomplished during the read-event of the FOE stream, essentially spooling the archive file to disk (or network share) as the stream is read by downstream components or the Messaging Agent itself. The BizTalk process memory footprint using this new code-base (while testing with a large (10Mb) message) is 90% more efficient than previous versions.

All other functionality remains unchanged.

Announcing a new BizTalk 2006 Message Archiving Component

Update: The Message Archiving Component is now maintained by Atomic-Scope, visit http://www.atomic-scope.com/downloadtrial for more information.

A couple of days ago I uploaded a revised version of my BizTalk 2006 Message Archiving Component onto CodePlex – version 0.2 can be downloaded from the CodePlex website at http://www.codeplex.com/btsmsgarchcomp.

The component works with BizTalk 2006 & BizTalk 2006 R2 and allows messages to be written to the file system (either locally or over a network share) for archiving, using message context-properties to define the archive path. It is written in a streaming fashion, is designed for large message consumption and can handle Xml, binary and flat-files.

The component can be executed in either the Decode (receive pipeline) or Encode (send pipeline) stages – when used on the receive side, the original message can be archived before BizTalk has started to manipulate the message; on the send side, the final message before delivery via the send adapter can be captured. If message archiving fails, the message will still continue downstream.

As per the previous version, the archive path is constructed using ‘macros’ (e.g. %ReceivePortName%) which relate to message Context Properties. In the previous version, these macro’s were hard-coded into the component so changing or adding a new macro was painful, requiring a rebuild of the code base. The new version continues the theme of macros, however they have moved into an Xml configuration file, which can be configured at run-time, allowing any context-property (either a known BizTalk context property or a custom adapter/pipeline property) to be used. An example of the configuration file is shown below:

These macros are used when defining the archive file path in the component configuration (in the BizTalk Server Admin Console) – see the screenshot below. The archive file path is constructed using the ArchiveDirectory and ArchiveFilename properties. Macros can be repeated and/or used in both properties at the same time (however I can think of scenarios where you may not wish to use say ReceivePortName in the archive filename).

Note: When configuring the component properties, you are specifing the macro name, not the context property name. This is useful when you are using a single configuration file and want to read the ReceivedFileName context property from both the FTP and FILE adapters for example (both use the same property name, but different property namespaces) – in this scenario, you would call the macro’s ‘ReceivedFileNameFILE’ and ‘ReceivedFileNameFTP’ to avoid ambiguity.

The example Macro Definition Xml file shown above details macros for the System Context Properties ReceivePortName and ReceiveLocationName. To use these macros in the pipeline component, assign them in the Archive Directory or Archive Filename property, as below.

These macro’s (%ReceivePortName%, %ReceiveLocationName% and %MessageID%) will be expanded as a message is delivered via the port to their real-world values, writing the archived message to (for example):

..Test Xml Dasm Decode StageTest Xml Dasm Decode Stage Location364a4dee-c698-4ddd-af14-d44e66f3bd5d.xml

In the example above we’re using the %MessageID% macro to build the archive filename. This is a special macro as the value does not come from the message Context, but is taken directly from the message itself, similar to using %MessageID%.xml in a Send FILE adapter.

The OverwriteExistingFile property indicates to the component whether an existing file should be overwritten if, once the archive path has been constructed from the Context Property macros, the full archive path resolves to an existing file. If set to true, the existing file will be overwritten; if false, the new file will be written to the same directory, but with an additional GUID before the file extension.

Finally, the MacroDefinitionsFile property defines the location to the Xml configuration file detailing the macro Context Property substitutions. A different file can be used per pipeline, or a single global file can be used. A more detailed example than that shown above – which includes macro’s for commonly used Message Tracking, System and File adapter properties – is available to download.

With regards to permissions, you will need to ensure that the user running the BizTalk service has read permissions on the macro definitions Xml configuration file and read/write permissions on the root of the archive directory.

Although the component is production ready it is not yet feature complete. If you are using the previous version, can I ask you to download a copy and try it out – please let me have any feedback via the comments on this post, or via the Issue Tracker pages of the CodePlex project.

A Bad Day for UK Software Patents….

An old uni friend, Paul Russell, has a worrying blog post about a recent High Court ruling over UK Software Patents – software can’t be patented in the UK, but it would appear that the court upheld an appeal from Symbian following the rejection of an application the software firm made to the Intellectual Patent Office (IPO):

“Symbian filed for a patent relating to the way computers use a library of functions that can be accessed by programs. The claim was discarded by the IPO last July, on the grounds that it related to software-based innovation – the IPO says patents are not available in the UK for solely computer-based programs, only for inventions that use software programs.” – Computing.

It would appear that the presiding Judge ruled in favour of the appeal as the IPO exlusions put UK software companies at an unfair disadvantage compared with rival developers in Europe, where the legal landscape works differently. The European Patent Office (EPO) has already approved Symbian’s application.

Apparently the IPO plans to contest the ruling at the Court of Appeal:

“on the grounds that the judge did not follow procedures created following a benchmark case in 2006.”

All is not lost.

Can I get some BI with this?

While I’m lying in bed dying of flu for the second time this year (grrr!!), here is a little light-hearted look at which tastes better: SQL Server 2005 or Oracle 11g. Click on the screenshot or go to www.databasetastechallenge.com.

“Now…. Who’s still thirsty!”

Testing Pipeline Components – The Pipeline Testing Library

I’ve spent some time recently working almost exclusively with BizTalk pipeline components and took the opportunity to use the excellent Pipeline Testing Library produced by Tomas Restrepo (more info is available here and here). If Microsoft were writing this post, they would probably say that ‘It Rocks!’. I’m British, so I’ll be a little more reserved and simply say that, imho, this is the single best tool for developing and testing custom pipeline components, period.

The library provides a framework for unit-testing components in the standard NUnit or MSTest manner, without having to go to Pipeline.exe or doing a full deployment to a BizTalk instance itself – both of which aren’t particularly quick or agile. Furthermore, the library lets you seamlessly integrate with MSBuild, without jumping through lots of hoops to get a similar effect.

The library allows you to programatticaly create pipelines, create instances of components and assign them to their respective stages (including your own custom components and the familiar out-of-the-box components), or load an existing BizTalk pipeline without the need to deploy the supporting BizTalk project.

A very simple example from my recent testing looks something like the following:

Here, I’m creating an instance of my archiving component (the component I’m testing) and the Xml Disassembler (specifying the document type it should accept). I create a new receive pipeline and assign the archiving component to the decode stage and the XmlDasm to the disassembler stage. After creating a sample message using the MessageHelper helper, I add context properties that mimic the adapter I’m trying to test (in this case the FILE adapter) and execute the pipeline. I finally test to check that the archiving component wrote the file to the archive in the expected location, with the expected filename.

If you want to test using Flat-Files, simply use a flat-file disassembler and specify the schema:

Powerful stuff – no more long-winded MSBuild tasks to deploy your solution, configure several possible adapters and push messages to those adapters over different protocols, simply to test a custom pipeline component on your development workstation (and lose 5-10 minutes in the process). Don’t get me wrong, all this should be happening in your smoke and UAT environments, but for the developer (attempting) to work in an agile manner on a BizTalk project, Tomas’ Pipeline Testing library is priceless.

Configuring Subversion & Apache for Web-Based Repository Access

If you are running Subversion under Linux (Ubuntu) and need to configure a (SSL) web-based repository access via Apache2, Viktor Zigo has a particularly good writeup on his blog at Installation of Subversion on Ubuntu, with Apache, SSL, and BasicAuth.

Yet More Firefox 3 Goodness: Better ‘Remember Password’ Functionality

It’s always the little things that make a product even better and I personally think that the new ‘Remember Password’ feature is just one of those enhancements.

I have a huge number of usernames and passwords used across a variety of different sites and one thing that niggled me with previous versions of Firefox and IE was the ‘Remember Password’ dialog – it will popup before you’ve logged into the site – before you’re sure that the password is correct! ‘Yes Firefox, I would like to remember the password, but I don’t know if its correct yet….’

The new ‘Remember Password’ feature presents itself in the top bar of the browser (see above) in a non-blocking fashion, allowing you see whether the login has been successful before instructing Firefox to remember the password.