1 Document Header
1.1 Confidentiality and Copyr= ight Statement
All in= formation contained in this document is confidential to Keystroke IT Limited and is intended for use only within Keystroke ESB Architecture. No part of this document may be reproduced by any means, nor transmitted, nor translated into a machine language or other language without the permission of Keystroke IT Limited.<= /span>
1.2 Foreword
This d= ocument is delivered pursuant to the Keystroke IT ESB Architecture.
1.3= Abstract
This document describes the ESB - X=
ML
Message And Transport Interoperability Architecture – an open
specification for Enterprise Sevices.
1.4 Document Control
|
|
Name |
Role |
Date |
Signature |
|
Compiled by |
Minas Casiou |
Solution Architect |
9 September 2003 |
|
|
Approved by |
|
|
|
|
|
Authorised by |
|
|
|
|
|
Accepted by |
|
|
|
|
1.5 Distribution
|
Role/Department<= o:p> |
Location |
No. of Copies |
|
CSC = Solutions Development Group |
|
|
1.6= Amendment Record
|
Issue |
Version |
Date =
|
Actioned by |
Description |
|
First Draft |
1.0= |
9 September 2003<=
/span> |
Minas Casiou |
<=
/span>
|
| <=
/span>
|
|
|
|
|
1.7 Table of Contents
=
Keystroke
IT - ESB Envelope. =
1.1 Confidentiality and Copyright Statement
2.4 Envelope Field definitions
1.8 Table of Tables
Table 1 Acronyms &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; &nbs=
p; &=
nbsp; 12
Table
2 Glossary of Terms =
&nb=
sp; =
&nb=
sp; =
&nb=
sp; =
&nb=
sp; =
<=
span
style=3D'mso-no-proof:yes'>12
Table
3 Call Synchronization Types &nb=
sp; =
&nb=
sp; =
&nb=
sp; =
&nb=
sp; =
<=
span
style=3D'mso-no-proof:yes'>12
Table
4 Common MIME Types =
&nb=
sp; =
&nb=
sp; =
&nb=
sp; =
&nb=
sp;
1.9 Table of Figures
Error! No ta= ble of figures entries found.
2 = Introduction
2.1 Purpose
The purpose of this= document is to give a detailed description of the ESB Envelope and the associated ESBEnvelope.xsd XML Schema. It will cover the specification and the current implementations.
2.2 Background
CSC requires a unif= ied Enterprise Services and Enterprise Application Integration (EAI) architectu= re to integrate existing LOB services to make them more accessible from client applications. There is also a desire within CSC to have a common integration architecture across all of its systems, that is reusable across all CSC projects that involve systems integration of any type. Underpinning this architecture is a standard envelope format used to encapsulate the data. Th= is is the ESBEnvelope. The ESBEnvelope addresses some of the issues we’ve come across with previous envelope implementations.
As expected, for the Services vision to be achieved, this envelope must be used by client system= s. Additionally, the client-side framework that supports some of these basic messaging principles and abstractions must also be used. In addition, a gen= eral envelope concept must be supported by service engines. They should be able = to impose some general requirements from an envelope, rather than dictate its form. In this way, differing envelopes can be supported.
The following has b= een assimilated as a result of 4 years of research & development, and a num= ber of implementations using the architectural principles surrounding integrati= on described here. It has proved to be an architecturally robust solution to the enormous area of systems integration. The main drawback is the lack of direct tool support and hence a relatively high barrier to entry. This barrier continue= s to be reduced as standards based technologies evolve. It is note yet clear as = to whether this will ever become a mainstream solution. Also, it is not for the feint hearted or for those who fear stepping out on a limb. It is for those= who architecturally and technically excel, and have adequate Business and ICT b= acking to make long term decisions and a commitment to make it work. Therein lies = the challenge. One thing’s certain though - those who choose to embrace i= t wholeheartedly will be deeply rewarded in their efforts.
2.3 Objectives
The main objectives= of the ESB Envelope being used with the ESB Service Engines & ESB Clients and = are listed below:
- To define a message format to encapsulate all data requir= ed for a request.
- To provide an envelope interface specification (IEnvelope interface) for each target platform so as to allow implementation & concurrent support of multiple envelopes, each with different XML Sche= mas formats.
- To provide a separate header & body section for the envelope in order to allow clients and service engines to perform gene= ric processing on the data contained within the header.
- To allow versioning of the envelope (via the XML Namespac= e).
- To utilize only internet standard protocols, data formats= and technologies, and hence use widely available tools to manipulate the envelope & data within it.
- To allow the data format to be transmitted via different transport protocols.
- To allow implementations to easily extend the data contai= ned within the envelope.
- To allow the inclusion of multiple inline documents as we= ll as multiple attachments to be included and referenced in the envelope.
- To provide metadata about the data contained in the envel= ope (envelope manifest).
- To provide an internal and external message correlation mechanism.
- To include addressing information to facilitate asynchron= ous messaging.
- To allow a client to specify (indirectly via aliases) whi= ch system the request should go to.
- To allow versioning of services. We will need to provide adequate fields for a client to populate and for the service engine to query so as to invoke the appropriate handler(s) for a given request.<= /span>
- To allow implementations to provide a customisable securi= ty solution as required.
- To allow implementations to include custom session manage= ment mechanisms.
- To include fields to give service engines the ability to execute a request on more that one system. These are optional fields.<= /span>
- To facilitate the transmission of ad-hoc custom fields as required for development, testing & maintaining state for the serv= ice within the service engine.
- To provide a uniform approach to all messaging services &= #8211; Request/Reply, Eventing etc.
- To abstract business data requirements away from technical implementation details.
![](3D"ESBEnvelopes_files/image002.jpg")
2.1 Envelope Field definition= s
2.2 ESB Envelope
1.&n= bsp; header
1.1.= delivery
1.1.1.1.1.1.1. senderMessageID
a messageID that is= specified by the client for the purposes of correlating the message between the client & service engine.
1.1.1.2. receiverMessageID<= /span>
a messageID that is= created by the service engine for the purposes of ensuring that all messages handle= d by the service engine itself are guaranteed to be unique. It should not rely on the clients to do this.
1.1.1.3. sent=
a timestamp reflect= ing the time the message was sent
1.1.1.4. received
a timestamp reflect= ing the time the message was received
1.1.1.5. subject
a comment that can = be applied to the message for descriptive purposes
1.1.2.1.1.2.1. address
A URL of some sort (understood by client & service engine) that identifies the endpoint fo= r a client request (and an entry point to the service engine). The address identifies where the request is going. Used more with asynchronous messages, but can also be used to override default settings.
1.1.2.2. addressType=
an optional identif= ier that specifies the transport type. It overrides any addressType information that= can be derived from the ‘address’ field.
1.1.2.3. state
A set of fields that identify possible state information for long running transactions (business processes).
1.1.2.3.1. &nb= sp; handle
1.1.2.3.2. &nb= sp; process
1.1.2.3.3. &nb= sp; referenceID=
1.1.3.1.1.3.1. address
URL that describes = the origin of a message (or where the response should go).=
1.1.3.2. addressType=
1.1.3.3. state
1.1.3.3.1. &nb= sp; handle
1.1.3.3.2. &nb= sp; process
1.1.3.3.3. &nb= sp; referenceID=
1.1.4.A value that identi= fies the
Synchronous =3D Bla= nk, 0 or 2
Asynchronous =3D 1<= /span>
Balking =3D 4 (Not implemented in ESB)
Timeout =3D 8(Not i= mplemented in ESB)
Synchronous and Asynchronous are mutually exclusive. Balking and Timeout can be added to ei= ther Sync or Async for those features.
1.2.= manifest
1.2.1.1.2.1.1. name=
XML Schema Name of = the XML document being added to the envelope. The convention is that the name of fi= rst document in the envelope is used as one of the lookup parameters by the ESB Service Engine.
1.2.1.2. description=
1.2.1.3. documentNamespace<= /span>
The namespace of th= e XSD Schema that this document is an instance of.
1.2.2.1.2.2.1. index
A numeric, zero ind= exed value.
1.2.2.2. filename
The local path or U= RL where the attachment can be found.
1.2.2.3. description=
1.2.2.4. type=
A MIME type that can help the application adaptor process the attachment.=
1.3.= systems – collectio= n
This section specif= ies which system a request should get executed against. Usually there’s o= nly one. Sometimes, the same request can get sent off to different systems. In = this case, the client will populate multiple systems and the service engine will then invoke the set of adaptors required for each system. For asynchronous requests (from the client to the Service Engine), this is a fairly trivial task. For synchronous requests (between the client to the Service Engine), where the Service Engine needs to send out the requests in parallel and correlate & merge the data, the preferred mechanism is to do the merging using XSLT, and to configure the whole process within the Service Engine. N= ote that this is advanced functionality and is usually handled above the integration layer, at the same level where things like pub-sub are handled. ESB.NET is aimed at being a multi-level Service Engine (Integration Layer, simple co-ordination layer & as a solid foundation for a high throughpu= t, configurable application layer). Depending upon which mode ESB is being use= d, the advanced features will or will not be used.
1.3.1.1.3.1.1. adminID
1.3.1.2. systemID
1.3.1.3. url<= /p>
1.3.1.4. environment=
an alias for an ins= tance of a particular system
1.3.1.5. systemType<= /span>
an alias for a name= or type of system
1.3.1.6. systemVersion
an alias for the ve= rsion of a system
1.4.= service
1.4.1.the name of a servi= ce – eg. ClientSearch
1.4.2.an optional version= of the service
1.4.3.an optional context= which can be used to execute different application adaptors (or which can cause t= hem to behave differently). For example, a ‘retrieve client details’ from the internet and a ‘retrieve client details’ from within a call centre may return/filter out data. For a LOB system, an update to the = LOB system which also updates a CRM system, may be different to the same update done by the CRM system itself. A different context can be applied, to in th= is case, specify the client type.
1.4.4.1.5.= custom
A name/value pair collection that can be used by the service engine to quickly/efficiently st= ore state on a particular message.
1.5.1.1.5.2.2.&n= bsp; body=
2.1.= documentBodies - collecti= on
2.1.1.This is the actual = body of a request or response. It is usually in XML format. Other formats are recommended to be sent as attachments for performance reasons.
2.3 Usage
The envelope is use= d in the following manner:
- A client constructs the document to be sent to the service engine. This is usually an XML document which itself is defined by an = XSD Schema.
- A client constructs a new envelope object. At this point,= the client can specify the underlying envelope, or allow the system to pick the default envelope implementation.
- The client sets the addressing information, adds any docu= ments and attachments to the envelope and sends it off to the server. The manifest etc. are set by the methods to add the documents & attachments to the envelope. The envelope object calls the underlying envelope implementation objects to do the work in each case.
- The client then sets the parameters that are used by the = ESB Service Engine to determine which set of adaptors to execute for this request. These are the following fields:
XMLSchemaName - Nam= e of the default (first) document in the manifest. It is legal for this to be empty (i.e. for there to be no document in the manifest & body), as you can s= end down information as attachments only.
SystemType – = can have multiples
SystemVersion ̵= 1; can have multiples
ServiceName<= /span>
ServiceVersion (opt= ional)
ServiceContext (opt= ional)
These six fields (2 optional) are used by the service engine to construct the pipeline of application adaptors to be executed for a particular request. This decouples the client from the service engine and gives it a lot of flexibility to determine which service to invoke.
The system type &am= p; system version fields can be specified multiple times so that a request can= be sent off to multiple systems (each with its own pipeline (set of adaptors)) with the one request.
Functionality to se= nd off multiple different requests within the one envelope has not been catered fo= r to date. I’m not sure the complexity is warranted, as we have not had a = need for it yet.
If a large number o= f these types of messages are required, they can be catered for by using attachments though. The attachments themselves can be envelopes. You can then configure= in an adaptor which reads the request envelope and invokes the service engine (using the RequestHandlerProxy) and submit the envelopes back into the Serv= ice Engine for processing. This would be a trivial task to write. Again, I̵= 7;m not sure of a reason to do it though.
What we have had ne= eds for though, are the ability to have multiple requests of the same type to the s= ame system. For example, we can process individual contributions for a Member o= f a Superannuation fund, or we can process the entire contributions for an Employer. In the case of the latter, we have 2 choices:
1)Have an XSD schem= a for contributions that allows multiple members & have that information inli= ne in the envelope
2)Same as 1 except = that we accept the contributions list as an attachment to the envelope. The envelope can be upwards of 100Mbytes in size & won’t affect the Service Engine’s general performance as the attachment is stored to disk by t= he Transport adaptor, and a relatively small envelope is passed around. The application adaptors still have a reference to the attachment when they rec= eive the envelope request.
We usually do the l= atter.
There are also case= s where we need to have multiple documents (same, different…) in the body of = an envelope.
As an example, an u= pdate procedure may require that the results of an enquiry are presented at the time of the update (for data consistency checking). In this case, the envelope can have= the new data as the first document in the envelope, and the results of a previo= us enquiry can be sent down in the second (or other position) document in the envelope.
- The envelope is sent to a transport adaptor (or with the = ESB client case, to the RequestHandlerProxy object). The adaptor will read= the appropriate information & determine where to send the envelope. Be= fore sending, the envelope’s serialization method is called & the envelope is converted to its XML serialized representation. This must conform to the XSD for the envelope. Any documents in the envelope are also serialized by the envelope.
- The request is received by an ESB transport adaptor. The transport adaptor strips off the packaging required to transport the message and submits the request off to the ESB Core for processing.
- When the request comes into the Service Engine, it looks = at the envelope header & determines whether the message should be handles synchronously or asynchronously. It is queued if async behaviour is required, otherwise processed immediately.
- ESB has a multi-stage pipeline which is executed for a re= quest. The last two are of significance at this level. The first stage sets up the environment/features required for the request. This is usually a f= ixed set of components that implement the usually internal ISystemEntryPoint interface. The second stage of the pipeline is where you configure in = the set of application adaptors (implement the IProcessMsg interface) that= get executed for a particular request. You can also specify adaptors that = get executed for all requests etc.
- Once the pipeline manager has applied the rules & gen= erated the pipeline to be executed for this request, it caches the result and invokes the application adaptors (or forwards the request off as configured…) in turn, according to the sync/async rules defined = in the pipeline.
2.4 Notes
- The envelope will contain only XML data as specified in t= he schema. Additional non-XML data can be base 64 encoded & added inl= ine if not very large, or preferably added as an attachment reference to t= he envelope. Transferring the envelope and associated attachments is left= to the individual transport adaptors to deal with.
2.5 ESB envelope Interface
In order to achieve= the above objectives, the following major design decisions were made.
- The only data format exposed to clients is XML. It is def= ined by way of XSD Schemas only. Other data can be passed around as attachments.
- A concept of a transport adaptor is imposed on the client= . A client must communicate with the ESB over one or more agreed internet standard transports. Currently supported transports are raw HTTP and S= OAP.
- The concept of an envelope. The concept of an encapsulati= ng envelope is used to identify all of the data required for a particular exposed service. The envelope has a header and a body and a manifest t= hat describes all the documents in the body. The body can have multiple XML documents in line in the body (all named in the manifest), and multiple documents referenced external to the envelope (as attachments, and all named in the manifest).
Using the above res= traints, all of the objectives can be met at the interface level.
2.6 ESB System implementation
The implementation = of ESB can be split into at least 4 logically different areas within a system (or = as it turns out within the ESB architecture[1]).
1.&n= bsp; As the integration layer.=
2.&n= bsp; As a simple co-ordination= /notification layer that sits atop the integration layer.
3.&n= bsp; A simple services layer t= hat implements business process related software services.=
4.&n= bsp; As a subset or starting p= oint of an application layer (a session manager slots in neatly and transparently into the framework).
At the physical implementation of ESB, there are a few patterns that are used to provide the features required by the interface. They are not important from a client integration point of view. They are only of importance to ESB and the Syste= ms it integrates to. These patterns & services help to decouple the adapto= rs being written from other adaptors in the system. They can be chained togeth= er and executed serially or in parallel as configured by the Pipeline Manager = etc. Pub-sub allows for notifications to interested parties of certain events.
The major ones of t= hese are:
- The Pipeline design pattern
- The Publisher and Subscriber design pattern=
Other relevant serv= ices and patterns
- Provider Pattern
- XSLT manager
- logging, instrumentation, monitoring & request level tracing
- remote management
- clustering (network load balancing)<= /li>
- generic test harness
…
3 = References<= /a>
(Removed)
4 = Glossary of Terms
Those abbreviations that are specific to this topic are expanded below and, where applicable, expanded on their first occurrence in the text:
|
API |
Application Program Interface |
|
CMM |
Capability Maturity Model for Software<= o:p> |
|
COTS |
Commercial off the Shelf |
|
CSC |
Computer Sciences Corporation |
|
DB |
Database |
|
e3, e4 |
Enabler for Extended |
|
EAI |
|
|
GUI |
Graphical User Interface |
|
HA |
High Availability |
|
HTTP |
Hypertext Transfer Protocol |
|
IP |
Internet Protocol |
|
Keystroke |
Keystroke IT Limited |
|
LOB |
Line Of Business |
|
PKI |
Public Key Infrastructure |
|
POC |
Proof of Concept |
|
RDBMS |
Relational database management System |
|
SI |
System Integration |
|
SIT |
System Integration and Testing |
|
|
Service Level Agreement |
|
SOAP |
Simple Object Access Protocol |
|
SSL |
Secure Sockets Layer |
|
TA |
Terminal Adapter |
|
TBA |
To be agreed |
|
TBC |
To be confirmed |
|
TCP |
Transmission Control Protocol |
|
TI |
Technical Infrastructure |
|
TP |
Transaction Processing |
|
XML |
Extensible Markup Language |
|
XSD |
XML Schema Definition |
|
XSLT |
Extensible Stylesheet Language
Transformation |
|
WS-* |
Web Services Interoperability set of
Specifications |
Table 1= A= cronyms
|
Envelope |
Extensible Data encapsulation design
Pattern which facilitates the packaging of multiple documents (Inline XML
Data and referenced attachments). |
|
Service
Oriented Architectural Pattern where a system controls itself via a simple
clearly defined interface and does not allow external access to its inter=
nal
data. Things like Transactions are managed by the system itself and
distributed transactions are handled by compensating transactions. |
Table 2=
Glossary of Terms
|
Symbol |
Message |
Description |
|
|
Simple |
Don't care how it's handled. Usua= lly asynchronous. |
|
|
Synchronous |
The sender effectively blocks unt= il the handler completes. This is a normal function call in a single-threaded application. |
|
|
Asynchronous |
The handler returns immediately, = but the actual work is done in the background and the sender can move on to other tasks while processing goes on. |
|
|
Balking |
The receiving object can refuse t= o accept the message request. This could happen if an "active" object's message queue fills, for example. |
|
|
Timeout |
The message handler typically blo= cks, but will return after a predetermined amount of time, even if the work of the handler is not complete. |
![](3D"ESBEnvelopes_files/image003.gif")
![](3D"ESBEnvelopes_files/image004.gif")
![](3D"ESBEnvelopes_files/image005.gif")
![](3D"ESBEnvelopes_files/image006.gif")
![](3D"ESBEnvelopes_files/image007.gif")
Table 3= Call Synchronization Types
|
Content type description |
AttachmentType (MIME type) |
|
Plain text |
text/plain |
|
XML data |
text/xml |
|
ZIP files |
application/zip |
|
MPEG video |
video/mpeg |
|
GIF image |
image/gif |
|
JPEG image |
image/jpeg |
Table 4=
Common MIME Types
[1]= ESB (formerly known as XMATIA – XML Message And Transport Interope= rability Architecture) was conceived in= late 1999 by Minas Casiou (which, is about 2 years before the e3/e4 stuff came a= bout etc.) As it turns out, the e3/e4 stuff identifies an integration and a co-ordination services as part of its architecture. The fact that the two w= ere developed separately, is a testimony to the importance of a clean interoperability layer. A low barrier to entry means it can be adopted more widely, especially within CSC’s systems & implementations.=
COMMERCIAL IN CONFIDENCE
COMMERCIAL IN CONFIDE=
NCE