Importance of Messaging for Enterprise applications

Communication happens between multiple applications using four major approaches.
1. File Transfer.
One application writes a file and another application reads it later. Applications need to agree on file location and name to exchange information.

2. Shared Database
Multiple application share information using a common database. This way application doesn't have to exchange information as it is a single source of data.

3. Remote Procedure
One application exposes some functionality remote so another application can access it remotely as a remote procedure call. This is real-time synchronous communication

4. Messaging.
One application publishes or sends a message to common message channel (Queue) and another application cand read or consume message from the message channel later. This application must agree on channel and Message format to do this asynchronous communication.

Challenges of communication between applications.

As a developer, while developing a software we are taking many assumptions.
For example, If we are developing an application which has to run an individual machine then we assume that while running that application it will always get enough memory and CPU cycle.  An application has never had to wait for any other resource etc.

While developing a software system which has to run on the different machine for different part of the application, we assume that communication between two system is reliable.  The network is efficient with good latency. TCP communication over network runs in a reliable manner.

Applications rarely live in isolation. any application has to communicate with other application directly or indirectly.  while working on this kind of distributed application, developers make a false assumption.  This is called "Fallacies of distributed application"

Fallacies of distributed application

https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

The network is reliable.
There can be many problems with a network like a cable cut, switch power cut, network h/w failure, firewall issue etc.  Also, there can be some security attack. Applications like e-commerce highly affected by this issues.
As a developer, we need to consider that network is unreliable and we need to address that too.

Latency is zero.
Latency is a how much time it takes for data to move from one place to another.
We develop an application on a local machine and considering LAN latency or local machine latency and considering latency is zero.  But when an application goes into production environment then the scenario is different and application results into slower in performance or transaction failure.
Ajax and Async/await are patterns are little use in this scenario.

Bandwidth is infinite.
Bandwidth is capacity of a network to transfer data (how much data can be transferred).
Bandwidth is continuously improving the network nowadays. Sometimes when bandwidth is lower then network packet loss ratio is high.
Considering bandwidth is not infinite, we should limit the size of information we send over the wire.
Always test application in a production environment for this kind of reason.

The network is secure.
The Network is far from secure.
1. General Internet attack trends are showing a 64% annual rate of growth
2. The average company experienced 32 attacks per week over the past 6 months
3. Attacks during weekdays increased in the past 6 months" [RipTech].

As a developer, we always think of security and awareness of all security and the implications. Sometimes provides message-level security for more confidential information considering network is unsecured.

Topology doesn't change.

When we deploy an application on production environment its topology may change. on Network we have to change servers,  load balancers,  Active directory settings etc. Sometimes we change the whole vendor with different network architecture (AWS/Azure).
The application should work in any of the topology change.

The application should not depend on specific endpoints or routes. It should be always prepared for renegotiating endpoints or provide discovery services.
Another strategy is to abstract the physical structure of the network. Instead of depending on specific IP address, the application should depend on DNS name.

There is one administrator.
If we have an application subsystem which outside of your administrative control then it can be an issue for your application.
We have integrated third-party pricing supplier API. Sometimes they change their security protocol and strategy without even informing us in advance.  That results in failure of a part of our application. Sometimes they change data model also without informing in advance. Sometimes they let us know in advance about their change.
This type of changes is not in control of one administrator.  We should develop our application in a way that it can very less depend on their model structure. Also, we can provide two version of implementation. One can work with an older version and another for the new version so the application can decide automatically which one needs to use until it finals all the changes.

Transport cost is zero.
When information transfer over the network it has to be serialized into bits. This conversion takes server resources and also add latency.
Also, All networking instrument like the router, subnet, load balancer has its own cost. Some cloud providers also charge based on data transferred across applications.
Developers normally ignore this costs.  They should consider this aspect also and develop the cost-effective application.

The network is homogeneous.
The network is not same everywhere. Even at home in the Similar network also may use different systems like Linux and windows. This can be an issue of data formate, protocols etc.
The application should use a standard protocol like XML/JSON.  The application should be used both according to client request it should respond in a particular format


Messaging System can save some of the above scenarios.

What is a messaging system?

We can take an example of telephone voice mail system.  One person calls to another person and leaves a voicemail. That voicemail queued to a message queue(mailbox). once person wanted to listen to voicemail, it dequeues a message from the queue.

Messaging definition: Messaging is a technology that enables high-speed, asynchronous, a program to program communication with reliable delivery.

Applications communicate by sending packets of data called Messages to each other.
Channels(queues) are logical pathways that connect that program and convey a message.
Sender or producer is an application that sends messages by writing the message to a channel. Reciever or Consumer is an application receives a message by reading from the channel.

Messaging capability provided by separate software called Messaging system or Message Oriented Middleware (MOM).  NServiceBus, MassTransit.

Messaging system manages messages while send/publish a message to consume the message. It also uses a database to persists message and state. An administrator has to configure a messaging system with messaging channel that defines the path of communication between sender application and receiver application. The primary purpose of the messaging system to move messages from sender application to receiver application in a reliable fashion. While transmitting a message from sender to receiver, at any point, failure occurs then Messaging system retry to transmit until it succeeded.

Message transmits in following steps:

1. Create: sender creates a message using message model class and populates with data.
2. Send: The sender add a message to the channel.
3. Deliver:  Message system moves the message from senders machine to receivers machine.
4. Receive: The receiver reads the message from the channel.
5. Process: Receiver extract data from the message.

Advantages of Messaging system

1. Remote Communication reliable
Messaging enables separate applications to communicate and transfer data.

2. Plateform / Language integration.
Messages can be consumed by an application written in a different language and deployed on a different platform as it message formate is standard like XML or JSON.

3. Asynchronous communication
Messaging provides fire and forgets communication. The application doesn't need to wait until it finishes execution.

3. variable timing:
In synchronous communication, both applications have to wait for each other to complete a transaction.
In messaging scenario, both applications can work on at their own pace.  no one has to wait for each other.

4. Throttling
In an RPC application it is a single point of the receiver of requests and When there is too many requests comes to a single receiver then it may crash the application. In Messaging scenario it only accepts message from queue whenever it is ready.

5. Reliable communication.
    Covering all fallacies,  Messaging system uses to store and forward approach to transmitting messages. Sending system stores messages and receiver system also stores a message. Messaging system resends messages until it received by receiver successfully.

6. Disconnect operation:
Some applications are designed to work in a disconnected environment.  Messages are queued. Whenever it connects to a network it can start consuming or sending messages.

7. Mediation
Messaging system can act as a mediator between two application.

8. Thread management.
By Asynchronous communication, Threads doesn't need to be blocked until other applications finish its work.  Request/Response scenario also can be implemented by the messaging system.

Challenges of Messaging system

Complex programming model
Implementation of messaging system is little complex then RPC or another synchronous system. Debugging is complex.  Requires little learning curve for new developers.

Sequence issues
Messaging system guarantee message delivery but not in proper sequence.  You need to take special care of it. Implement saga and correlation can resolve this issue.

Synchronous scenario
In many cases, we need to provide instant feedback to the client about performed operation result.
We need to implement request/response or Push notification to cover up this.


Ref: Book: Enterprise-Integration-Patterns-Designing-Deploying
https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
www.rgoarchitects.com/Files/fallacies.pdf