Scatter-Gather Pattern with SAP BAPI Transactions

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.



This article is the third installment of a series on SAP BAPI transactions in BizTalk.


In SAP BAPI Transactions Walkthrough, an orchestration receives a number of BAPI transaction requests one at a time. In the second post, Debatching SAP BAPI Transactions, the same orchestration is modified to accommodate messages containing a potentially large number of BAPI transaction requests; these messages are debatched by using an in-orchestration receive pipeline to create the BAPI transaction messages sent to the SAP server.


A common point between these two previous setups is that the BAPI transactions are sent serially, and the responses from the SAP server are received in the same order. The next variation, presented here, corresponds to a situation where neither constraint is required: BAPI transactions can processed concurrently, and responses do not need a specific order. To this end, we implement the Scatter-Gather pattern. In the new design, a parent orchestration starts a separate child orchestration instance for each BAPI transaction, and a self-correlating port is used for getting the responses back. The child orchestration implements a variation of the  Content-Based Router pattern whereby message content determines subsequent BAPI actions and connection states.




As explained in Run BAPI Transactions in SAP using BizTalk Server, the message context property Microsoft.Adapters.SAP.BiztalkPropertySchema.ConnectionState is used to map a BAPI transaction to a Logical Unit of Work (LUW) on the SAP server side. So, the first transaction has this property set to "OPEN", subsequent transactions then use "REUSE", and the last transactions have either "CLOSE" or "ABORT".


To delve a little into the BizTalk internals at a high-level, a series of transaction messages (i.e. OPEN-REUSE-REUSE-…-CLOSE) requires the same connection for outbound communication with the SAP server, and this connection is mapped to an LUW on the SAP server side by interchange id and send port id. In our current scenario, debatched messages share the same interchange ids which get propagated throughout the process, and the created BAPI transactions are sent from the same send port, so they can be grouped in the same LUW. In layman's term, for BAPI transactions to be in the same LUW, (from OPEN to CLOSE/ABORT) they need to come from the same "place" (the interchange id) and to be sent out  to the same "destination" (the send port) with the same route (the "session").


The use of interchange id + send port id is compatible with the "session" paradigm provided by the SAP Connector for Microsoft .NET (NCo) to expose the BAPI Transaction Model documented by SAP. Note that the latter mentions some restrictions on combining BAPIs in the same LUW. For instance, it is not possible to make two write accesses on the same instance within one LUW. However, it is possible to create several instances of the same object type within an LUW, as is the case in our scenario. Of particular interest here, BizTalk allows asynchronous and parallel processing of BAPIs, which can be an attractive proposition for instance for communication with an external server and associated latency.


As a side-note, it is the opportunity to mention that:


Implementing the Scatter-Gather Pattern


Let's start with the main orchestration. The first stage is the debatching pipeline explained in detail in the previous post Debatching SAP BAPI Transactions.


Orchestration receives

Debatching pipeline produces

<ns0:RequestsInfo xmlns:ns0="…"> <IsCommit>true</IsCommit> <Orders> <ns2:Order xmlns:ns2=""> <ORDER_HEADER_IN> <DOC_TYPE>TA</DOC_TYPE> <SALES_ORG>1000</SALES_ORG> <DISTR_CHAN>12</DISTR_CHAN> </ORDER_HEADER_IN> <ORDER_ITEMS_IN> <MATERIAL>DPC1020</MATERIAL> </ORDER_ITEMS_IN> <ORDER_PARTNERS> <PARTN_ROLE>AG</PARTN_ROLE> <PARTN_NUMB>0001012</PARTN_NUMB> </ORDER_PARTNERS> </ns2:Order> <ns2:Order xmlns:ns2=""> <ORDER_HEADER_IN> <DOC_TYPE>TA</DOC_TYPE> <SALES_ORG>1000</SALES_ORG> <DISTR_CHAN>12</DISTR_CHAN> </ORDER_HEADER_IN> <ORDER_ITEMS_IN> <MATERIAL>DPC1020</MATERIAL> </ORDER_ITEMS_IN> <ORDER_PARTNERS> <PARTN_ROLE>AG</PARTN_ROLE> <PARTN_NUMB>0001012</PARTN_NUMB> </ORDER_PARTNERS> </ns2:Order> </Orders> </ns0:RequestsInfo> <ns0:CREATEFROMDAT2 xmlns:ns0="" xmlns:ns3=""> <ns0:ORDER_HEADER_IN> <ns3:DOC_TYPE>TA</ns3:DOC_TYPE> <ns3:SALES_ORG>1000</ns3:SALES_ORG> <ns3:DISTR_CHAN>12</ns3:DISTR_CHAN> <ns3:DIVISION></ns3:DIVISION> </ns0:ORDER_HEADER_IN> <ns0:ORDER_ITEMS_IN> <ns3:BAPISDITM> ... </ns3:BAPISDITM> </ns0:ORDER_ITEMS_IN> <ns0:ORDER_PARTNERS> ... </ns0:ORDER_PARTNERS> </ns0:CREATEFROMDAT2> ---------------------------------------------- <ns0:CREATEFROMDAT2 xmlns:ns0="" xmlns:ns3=""> <ns0:ORDER_HEADER_IN> <ns3:DOC_TYPE>TA</ns3:DOC_TYPE> <ns3:SALES_ORG>1000</ns3:SALES_ORG> <ns3:DISTR_CHAN>12</ns3:DISTR_CHAN> <ns3:DIVISION></ns3:DIVISION> </ns0:ORDER_HEADER_IN> <ns0:ORDER_ITEMS_IN> <ns3:BAPISDITM> ... </ns3:BAPISDITM> </ns0:ORDER_ITEMS_IN> <ns0:ORDER_PARTNERS> ... </ns0:ORDER_PARTNERS> </ns0:CREATEFROMDAT2>


After pipeline processing, a variable named "BAPIOrders" contains a list of BUS2032.CREATEDAT2 objects. This is summarized below.




In the subsequent orchestration stage, instead of sending the BAPI transactions directly to the LOB send-receive port as was previously done, single BAPI transaction messages BUS2032.CREATEFROMDAT2 are passed as parameter to a Start Orchestration shape. This pattern is referred to as "scatter" since the messages are scattered out to child orchestrations, where the send-receive exchanges with the SAP server happen.


Note: The Start Orchestration shape allows true parallelism. This is not the case for instance with the Parallel shape, which does not make guarantees on "multi-threaded-like" execution . More info is available in a previous blog article and in How the Parallel Actions shape works.

Parallelism is also not the case with the Call Orchestration shape, which is synchronous.


In the main (parent) orchestration, the LOB send-receive port that was used for communicating with the SAP server in previous implementations is now replaced by a self-correlating direct bound port, which gathers data from the child orchestration instances asynchronously. The exact steps on how to implement the Start Orchestration shape and the self-correlating direct bound port are presented in detail in How to Use Self-Correlating Direct Bound Ports.


In our case, the child orchestration ChildBAPIOrchestration has 4 parameters:



The self-correlating port used for sending SAP responses back to the main orchestration. It is a standard parameter in the Scatter-Gather pattern.

ConnectionState The connection state (OPEN/REUSE) in the child orchestration for the BAPI transaction messages sent to the SAP server.
XmlMessage A message of type XmlDocument corresponding to the BAPI transaction message (i.e. CREATEFROMDAT2, BAPI_COMMIT_MESSAGE etc.) debatched from the pipeline.
MessageType The type of the message in the child orchestration to decide which operation to use in the LOB send port (explained in the next section).


Parameters for the BUS2032.CREATEDAT2 messages are set as follows (note the assignment of typed BAPIMessage to XmlMessage, and the use of the BTS.MessageType property):


BAPIConnectionState = "OPEN"; // if this is the first message, "REUSE" otherwise BAPIMessage = BAPIOrders.Get(BAPIOrdersCount); // BAPIOrdersCount is the loop index XmlMessage = BAPIMessage; MessageType = BAPIMessage(BTS.MessageType);




Parameter Processing in the Child Orchestration


The child orchestration uses the MessageType parameter to route messages to the proper send port operation. This is done in a Decide shape with the following condition:


MessageType == "http://Microsoft.LobServices.Sap/2007/03/Bapi/BUS2032/#CREATEFROMDAT2"


Note that the message type has the format <namespace>#<root node name>.

The BAPI transaction message is then constructed with:


BAPICreateFromData = XmlMessage; BAPICreateFromData(Microsoft.Adapters.SAP.BiztalkPropertySchema.ConnectionState) = ConnectionState;


The transaction message BAPICreateFromData is processed in the WCF-SAP send-receive port, SAPSendReceivePort. The response from the SAP server, which contains the newly-created document id, is forwarded back to the parent orchestration via the self-correlating send port SelfCorrelatingSP.





As explained in SAP BAPI Transactions Walkthrough, the parent orchestration extracts the document ids from the SAP server's responses and saves them for latter reuse with BUS2032.GETSTATUS messages in the last stage.


Child Orchestration Design


The previous section showed what happens for the BUS2032.CREATEFROMDAT2 method. We reuse the same orchestration design for BAPI_TRANSACTION_COMMIT, BAPI_TRANSACTION_ROLLBACK, GETSTATUS: The same child orchestration handles all message types with a Decide shape to dispatch received XmlMessages to the corresponding execution path based on the value of the MessageType parameter. This is similar to the Content-Based Router pattern.






Main Orchestration


Child orchestration instances are started in multiple stages:

  • Stage 2: when the BAPI transaction messages are retrieved from the local list;
  • Stage 3: to execute (1) BAPI_TRANSACTION_COMMIT or (2) BAPI_TRANSACTION_ROLLBACK actions;
  • Stage 4: to execute the GETSTATUS action based on the document ids received back from the child orchestration instances in Stage 2.


Putting together all stages of our main orchestration, now version 3.0:




It is important to note that while there is no guarantee on the execution order of the child orchestration instances, the parent orchestration will not proceed to the commit/rollback stage until all responses have been received on the self-correlating port. This prevents channels from closing too early, thereby avoiding the "Inner Channel" errors mentioned for instance in Problems Handling SAP-Transactions.


Concluding Remarks


If the Start Orchestration shapes are replaced by Call Orchestration, the main orchestration will process everything in the same thread, and responses from the SAP server will be received in chronological order. If it is the desired behavior, it is an easy way to make the current orchestration equivalent to the one previously presented in Debatching SAP BAPI Transactions.


Besides sequencing, another factor to consider in the choice of a Call or Start Orchestration shape is performance, as has been discussed elsewhere. For instance, the gain from parallel processing could be lessened by the latency induced by the Start shape as messages go through the message box.


Last, the rollback path would be well-suited in the catch exception block of a scope shape. An interesting way to extend the topic presented here could be to see how error handling is affected by debatching and asynchronous processing.


Sample Code


All code used in this article is attached.




SAP BAPI Transactions Walkthrough

Debatching SAP BAPI Transactions

Scatter-Gather pattern

How to Use Self-Correlating Direct Bound Ports

BAPI Transaction Model documented by SAP

Content-Based Router pattern


For more general info on the SAP Adapter:

Operations on BAPIs in SAP

Run BAPI Transactions in SAP using BizTalk Server

Message Schemas for BAPI Operations

SAP Adapter documentation

Registry setting to enable BAPI transactions


REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.