In our previous post(https://zxer.in/playing-with-kubernetes/) we have briefly discussed about docker and deploying applications in kubernetes. We built a backend and a front end service. And we were able to access the front-end from internet. But we haven't connected both services together. If you remember we hard-coded the backend IP address in our front-end nodejs code to access the currency convertor API. But when we are deploying in kubernetes cluster we don't know what the IP address is going to get assigned for our application. If we are deploying multiple instances, then the problem becomes even complex as we need to distribute traffic to multiple IPs.

A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them - sometimes called a micro-service. The set of Pods targeted by a Service is (usually) determined by a Label Selector.  In our deployment we grouped front-end using selector

matchLabels:
      app: currency-app-frontend

What this means is all the pods with selector app : currency-app-frontend  will be grouped as a single service.  There are 2 primary modes for service discovery.

1. DNS

Each service defined in cluster gets a DNS name. And services can be resolved by the name from within the same namespace. And pods from other name space can access by adding namespace to the DNS path. In our case currency-app-backend-service.default.svc.cluster.local. So we can access the backend service by changing the backend IP to its DNS name and it will be accessible from cluster.

const express = require('express');
const request = require('request');


// Constants
const PORT = 8080;
const HOST = '0.0.0.0';
// Backend IP changed to DNS name
const BACKEND = 'currency-app-backend-service.default.svc.cluster.local'

// App
const app = express();
app.set('view engine', 'ejs');

app.get('/', (req, res) => {
  res.render('pages/index');
});

app.get('/currency', (req, resp) => {
  console.log('http://'+BACKEND+"/?currency="+req.query.currency+" API request send");
  request('http://'+BACKEND+"/?currency="+req.query.currency, { json: true }, (err, res, body) => {
  if (err) { return console.log(err); }
  resp.send(body);
});
});

app.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);```

2. Environment Variable

When a Pod is run on a Node, the kubelet adds a set of environment variables for each active Service.  While deploying services, we can set the environment values to be set in yaml file. In our deployment we can set SERVICE_IP and SERVICE_PORT as environment variables and access it from other PODs in same cluster.

Access the shell by following command by providing pod name

 kubectl --kubeconfig=currency-app-demo.yaml exec -it currency-app-frontend-deployment-95f9cf5-wbn24 sh

then print the variables using printenv

# printenv
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.245.0.1:443
NODE_VERSION=8.15.0
HOSTNAME=currency-app-frontend-deployment-95f9cf5-wbn24
YARN_VERSION=1.12.3
HOME=/root
TERM=xterm
KUBERNETES_PORT_443_TCP_ADDR=10.245.0.1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.245.0.1:443
KUBERNETES_SERVICE_HOST=10.245.0.1
PWD=/usr/src/app

This means any service a POD wants to communicate with should be created before otherwise the environment variables won't be populated.

Publishing Services - Service Types

If you want to make your service from outside (outside cluster) you need to publish the service. There are many ways to do that

1. ClusterIP
   Exposes the service on a cluster-internal IP. Choosing this value makes the service only reachable from within the cluster. This is the default ServiceType. This what we have used in our first tutorial. Once our backend service got assigned with a ClusterIP, we hardcoded that in our front-end application.
2. NodePort
   Exposes the service on each Node’s IP at a static port. This will be accessible even from out side of cluster. A ClusterIP service, to which the NodePort service will route, is automatically created. That mean even if there is no POD running on a particular NOD, we will be able to access the service using NodeIP:NodePort.
3. LoadBalancer
   We can expose any service by creating a LoadBalancer. Whenever we create a Service with ServiceType as LoadBalancer underlying cloud provider(AWS/Google Cloud/Azure/DO) will create a separate load balancer and balance the traffic between the service nodes. This is the easiest method but you might have to pay separately for it.

 Kubernetes is an open source platform that automates container operations. originally developed and designed by engineers at Google. Considering the adoption rate among cloud providers and companies, we can clearly see that k8s has won the container orchestration. That means if you are working with distributed systems you are going to interact with this technology.

Even though you can play around with kubernetes in your local system using minikube, it will not do justice to the  power of k8s since you are building a cluster with single node. Recently digitalocean announced that they are releasing managed kubernetes on the platform. And luckily they had given me a 100$ credit for 60 days to try out products from DO.  So I have decided to build a sample real world application using kubernetes to learn more about k8s.

We are going build currency convertor  app. It will have a front-end, a backend. We will deploy it in a multi node cluster with load balancing.

Here are the we are planning to build.

1. A customer facing "front-end" application.
2. A "backend" application which front-end talks to
3. Public load balancer so that users can access the front-end
4. Different deployments strategies

Before we talk about container orchestration,  lets have a look at container. Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. Similar to the physical containers, once you containerise your application, you can run it without worrying about the host. Even though there are many technologies for containerisation(lxc, rkt), we are going to use Docker since it is the most popular container technology.

Lets say you are building a nodejs application, if you don't have container technology then you need to procure a machine, install desired operation system (with correct version), set up nodejs run time, configure all the environment. And if you plan to have a second instance of the same application you need to repeat the procedure and make sure that everything same. But if you have packaged your application using any of the container technology, you can run it without worrying about where its going run.

Backend

we will build a simple REST API backend for our currency-convertor app. For brevity we will use a in-memory map to store the existing rates and expose a REST API to get the conversion rate for a currency. I am choosing go for this as go compiles to machine code and we don't have to install any runtime to run the application inside docker image. We can choose alpine linux as the base image as it is just 3.98 MB in size. Here is our backend code.

import (
	"encoding/json"
	"log"
	"net/http"
	"os"
)

type CurrencyResponse struct {
	Rate float64
	Host string
}

var currencyMap = map[string]float64{
	"usd": 1,
	"cad": 1.12,
	"inr": 72.3,
}

func handler(w http.ResponseWriter, r *http.Request) {
	currency := r.FormValue("currency")
	value := currencyMap[currency]
	if value != 0 {
		resp := CurrencyResponse{
			Rate: value,
			Host: os.Getenv("HOSTNAME"),
		}
		json.NewEncoder(w).Encode(resp)
	} else {
		w.WriteHeader(http.StatusBadRequest)
		w.Write([]byte("Not found"))
	}
}

func main() {
	http.HandleFunc("/", handler)
	log.Fatal(http.ListenAndServe(":8888", nil))
}

As you can see from the code, it starts a web-server on port 8888 and exposes single API. When it receives a request with a query parameter currency it return the value from map. If there is no value found in the map return Not found.

Before building the docker image lets build and verify the code

#build 
go build -o docker-backend

#run
./docker-backend 

You can verify the application by going to localhost:8888/?currency=usd from browser or using curl. And you should see response similar to below image.

Now that we have a running application, lets build a docker image out of it. For that we need DockerFile which contains all the instructions to create an image. As you can see its pretty simple and self-explanatory.

FROM golang:1.11-alpine AS build

WORKDIR /src/
COPY main.go go.* /src/
RUN CGO_ENABLED=0 go build -o /bin/demo

FROM scratch
COPY --from=build /bin/demo /bin/demo
ENTRYPOINT ["/bin/demo"]

Now we can build the image by executing

docker image build -t currency-app-backend .

then run the image using

docker container run -p 9999:8888 currency-app-backend

As you remember we are running our application in port 8888. But thats inside the container. While running the docker image we need to specify which port from host machine to be mapped to the internal port. In this command we are mapping port 9999 to containers 8888.
This can be verified by accessing  localhost:9999/?currency=usd. We should get the same response as we got when we run the application directly.

Front end

Now we need to build a user friendly front-end where a user can easily access the currency conversion rates. We will build a simple nodejs application with express. For simplicity we will use simple buttons for each currency and on clicking them we will try to fetch conversion rate from the backend API.  As you can see from the code we are using a parameter "BACKEND" to connect to the backend. But here we don't know what will be the IP address of backend service is going to be once its deployed in kubernetes. There are different ways to solve this issue and we will discuss some of them.

app.get('/currency', (req, resp) => {
  console.log('http://'+BACKEND+"/?currency="+req.query.currency+" API request send");
  request('http://'+BACKEND+"/?currency="+req.query.currency, { json: true }, (err, res, body) => {
  if (err) { return console.log(err); }
  resp.send(body);
});

Deploying to kubernetes cluster

Now that we are ready with our docker images for both front-end and backend, we need to deploy it to the cluster. We will use DigitalOcean managed kubernetes service. Creating a cluster is just few clicks. Select k8s version, data center region, number of nodes and and choose a name for your cluster and click create cluster button. Wait for couple of minutes as the cluster is getting created.

Once cluster is created successfully we need to connect the cluster. For that we need to install kubectl, the official Kubernetes command-line tool and The cluster configuration file, which contains authentication certificates. I have downloaded them and kept it in the same folder as "currency-app-demo-kubeconfig.yaml" Once you have both you can verify using

kubectl --kubeconfig=currency-app-demo-kubeconfig.yaml get nodes

You should see some thing similar to following image. You can see there are three node all are in ready state.

Deploying our service

Now we have our 3 node kubernetes cluster ready and our application is dockerised. Before we deploy our application we need to publish our image to a container registry. For now we will use Docker Hub. Before pushing the image we will tag it

docker tag currency-app-frontend vmuneeb/currency-app-frontend:1.0.0

then push it to your account

docker push vmuneeb/currency-app-frontend:1.0.0

Now our image is now available to use from docker hub.

Kubernetes uses deployment controller for deploying applications. As per official documentation "A Deployment controller provides declarative updates for Pods and ReplicaSets.You describe a desired state in a Deployment object, and the Deployment controller changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments." We can use a simple yaml file to create a deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: currency-app-frontend-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: currency-app-frontend
  template:
    metadata:
      labels:
        app: currency-app-frontend
    spec:
      containers:
        - name: currency-app-frontend-image
          image: vmuneeb/currency-app-frontend:1.0.0

most of it are self explanatory. One thing we need to look is the selectors. This labels will be used extensively for selecting services later. Labels can be used to organize and to select subsets of objects in k8s. We can apply this yaml file using

kubectl --kubeconfig=currency-app-demo-kubeconfig.yaml apply -f docker-frontend/yaml/deployment.yaml 

If everything is success, you will see a success message on the terminal.
You can verify this buy checking the available pods

 kubectl --kubeconfig=currency-app-demo-kubeconfig.yaml get pods

You can see we have one(as we have given only 1 as the number of replicas in our deployment.yaml) running pod.

Now our application is running successfully in the cluster. But we can't access it from outside world. There are many methods to expose applications like NodePort, ClusterIP, LoadBalancer. In this tutorial we will try to create a LoadBalancer to expose the service.

To expose the application to outside world we can create a load balancer. For that we will use a service controller. Here is the sample service.yaml.

apiVersion: v1
kind: Service
metadata:
  name: currency-app-frontend-service
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: currency-app-frontend

As you can see, we are creating a load balancer which listens on port 80 and redirect traffic to port 8080 of all the pods which has label as app=currency-app-frontend. Apply the yaml file.

 kubectl --kubeconfig=currency-app-demo-kubeconfig.yaml apply -f docker-frontend/yaml/service.yaml

If you go to network tab and check loadbalancer section, you can see there is a load balancer created. If this was executed on AWS of google cloud, they will create a similar load balancer service for you.

Once it is successfully created you can check by accessing the IP address from any browser. You should see the simple HTML page  we created for front-end application rendered.

Yay!!! So we have our first application deployed in kubernetes cluster and exposed using a load balancer. We will discuss about service discovery, deployment strategies and other topics in another post.

Processes are the basic building blocks in Elixir. These are used for building distributed fault tolerant systems. These can be compared to "actors" in AKKA (AKKA actors are inspired from Erlang). Processes are isolated from each other, run concurrent to one another and communicate via message passing.

You can start a process by using spawn and you will get a PID back. This PID can be used to send and receive messages to the corresponding message. Lets take a simple example

defmodule Example do
  def listen do
    receive do
     {:ok, "hello"} -> IO.puts("World")
    end

  listen
 end
end

then you can test it in iex session

iex> pid = spawn(Example, :listen, [])
#PID<0.108.0>

iex> send pid, {:ok, "hello"}
World
{:ok, "hello"}

Recently I tried to build a online multiplayer tic-tac-toe game using Elixir. Where each game was handled by a process. Whenever someone starts a new game, dynamically we create a process and we keep the reference to this process. And each user action has to be communicated to the corresponding process.

Since each game needs to store the game state, I used GenServer to start the process. A GenServer is a process like any other Elixir process and it can be used to keep state, execute code asynchronously and so on. The GenServer behaviour abstracts the common client-server interaction. You can see basics of GenServers https://hexdocs.pm/elixir/GenServer.html.

You can have thousands of processes running in the system (could be in the same node or in distributed manner). Number of PIDs we need to keep track can be a problem.

One way would be to start the GenServer with a name and refer it by that. From the documentation we can see that GenServer can be registered with

• atom : GenServer is registered locally with the given name
• {:global, term} : GenServer is registered globally with the given term using the functions in the :global module. Mostly used in distributed mode
• {:via, module, term} : GenServer is registered with the given mechanism and name. One such example is the :global module which uses these functions for keeping the list of names of processes and their associated PIDs that are available globally for a network of Elixir nodes. Elixir also ships with a local, decentralized and scalable registry called Registry for locally storing names that are generated dynamically.

Since we need to dynamically create processes for each game, we can't use the first option.

The second option is used to register a process globally, across multiple nodes. This also means it requires synchronization across the entire cluster, which is not what we are looking right now.

Third option expects a module with register_name/2, unregister_name/1, whereis_name/1 and send/2. That means, once we provide a module with these methods, system will register PIDs once it started, will use whereis_name to find the PID to send message and use unregister_name to remove once the process is stopped.

In that module we can use a simple Map, where we use the game name(randomly created unique for each game) as the key and PID as the value.

• register_name : Add to the Map
• unregister_name : remove to the Map
• whereis_name : get from Map

Since we are building a fault tolerant system, there is another issue we need fix. If any process gets killed/crashed or restarted, our Map is unaware of this and will contain dead process PID. This can be easily solved by monitoring the started PIDs by the Registry the we will get :DOWN message with PID and other information.
We just need to remove this from our MAP.

Here is a final Registry module we came up with Registry.

Finally the code can be found here (https://github.com/vmuneeb/elixir-multiplayer-tictactoe)

You can check the final game here.

Earlier when I was working with AKKA actors, there was ActorSelection for keeping track of actor references. Given an actor path with address information you could get hold of an ActorRef to any actor. And you could use any other data structure to store these references for future use. In this example you could see that all the available worker actors references were kept in a HashMap.
But it looks like in the newer version of AKKA, they got rid of that and introduced Registry service similar to this. Official Documentaion here.

I have a small droplet in digital ocean(where this blog is hosted). Recently I was trying to run an application and it was getting killed immediately without any error message. After some googling it turned out that system was throwing memory out of exception. dmesg showed me

dmesg

Even though the application was small, the framework I used had a minimum memory requirement which was more than I had in the droplet(512 mb).

The Solution was to create SWAP memory. SWAP memory is a holder in hard disk which can be used by OS to store data which exceeds RAMs capacity.
Eventhogh it will add more RAM space for OS to work on, this will be much slower since we are accessing hard disk.
But in our case we know that it will not be an issue since our application has small memory footprint.

How to

sudo swapon -s

This command will show you any existing SWAP file. If there is no entries that means there is no SWAP file enabled.

Now we can create a SWAP file under root(/swap).

sudo dd if=/dev/zero of=/swapfile bs=128m count=10

here we are trying to write zeros with block size of 128 mb for 10 times. Make sure that your block size is smaller than the available RAM in your system.

Set permission. ie Allow only root to read and write

sudo chmod 600 /swapfile

Now make the newly created file as a SWAP file

sudo mkswap /swapfile

Check if its created

sudo swapon -s

You can check with free -m also.

Recently Elixir and their web framework Phoenix has been picking up lot of interest, especially from HN. I decided to take a look and it looks very exciting. Some time back when I was playing with XMPP, I came across Erlang. But it had a very deep learning curve and the syntax was off-putting at that time.

Elixir runs on the Erlang VM giving complete access to Erlang’s ecosystem and has better syntax. Apart from approachable syntax, here are the three things which are exciting for me.

1. Pure functional language.
   I come from a JAVA world, recently I have been playing around with Scala and the idea of functional programming is tempting.

2. Actor based.
   In work we have been doing a lot work with AKKA actors and it seems excitingly simple and efficient for some of the distributed, fault tolerant tasks without any complexities of multi threaded application.

3. Channel.
   From Phoenix documentaion "Channels are a really exciting and powerful part of Phoenix that allow us to easily add soft-realtime features to our applications. Channels are based on a simple idea - sending and receiving messages". The default transport mechanism is via WebSockets which will fall back to LongPolling if WebSockets are not available and there are javascript,SWIFT and Android client.

Best way to learn something new is always by doing. Back in 2012 I was building an online test preparation platform with collaborative features. Basic Idea was students should be able to join study groups and once they start a study session, everyone in the group gets questions and they can answer/discuss in realtime. And there would be an admin bot which will manage the session like delivering questions, keeping the score board, ban/block users, etc.
Initially we found XMPP fitting perfectly for our requirment. We picked ejabbered as the XMPP server and strophe.js on client and in the backend we used Smack library from openfire.
Our main application was in PHP MYSQL, and XMPP server in Erlang, there was another backend code written in JAVA. We faced a lot of issues gluing all the systems together.
Later we had complete re-write in nodejs and socket.io. Architecture and development became much simpler since both server and client where running javascript.

I will be trying to build a POC for the above mentioned using Elixir(Phoenix framework to be precise) and may be an Android/iOS client too if I get enough time.

For a recent project, I had to integrate AKKA TCP with our REST server built on play framework. We faced and solved many issues while trying to scale the application for high concurrency. Here are the details

Requirment:
We were building REST API. For one particular API, the request was then routed to a TCP server(http request transformed to XML)
And once TCP server sends the response back,return the response to the same http request.

First method

was straight forward, whenever we get HTTP request

1. convert it to XML.
2. Send to Server using java
3. Iteratively check for response for pre-configured number of times.
4. If no response, report failure
5. If we get response from server within the time, report the result to user.

Issues

1. Sequential. Only one HTTP request will be served at a time.
2. Polling at the TCP server.

Solution

Use reactive library for TCP communication.

Second method

Since we were using play framework for AKKA was our first choice. Akka library provides actor based programming for concurrent computation. AKKA has built in support for handling TCP( AKKA TCP).

Connecting and sending messages to server is pretty straight forward as you can see from the example code.

public class Client extends UntypedActor{
    final InetSocketAddress remote;
public static Props props(InetSocketAddressremote) {
  return Props.create(Client.class, remote);
}
//Constructor takes remote server IP address
public Client(InetSocketAddress remote) {
   this.remote = remote;
//All of the Akka I/O APIs are accessed through manager objects.
   final ActorRef tcp = Tcp.get(getContext().system()).manager();
//Using the manager object, try to connect to the server. Response from this connect will be handled by onRecieve method below
   tcp.tell(TcpMessage.connect(remote), getSelf());
}
@Override
public void onReceive(Object msg) throws Exception {
if (msg instanceof CommandFailed) {
//If the tcp connect fails
  listener.tell("failed", getSelf());
  getContext().stop(getSelf());
} else if (msg instanceof Connected) { //If the tcp connected successfully
//Once connection is made, we need to send a register message to the connection to make it active.
getSender().tell(TcpMessage.register(getSelf()), getSelf());
//Change the behaviour of this connection to "connected" method.
  getContext().become(connected(getSender()));
 }
}
//Actual code block responsible for server communication.  
private Procedure<Object> connected(final ActorRef connection) {
return new Procedure<Object>() {
  @Override
  public void apply(Object msg) throws Exception {
/*Once the actor has changed its behaviour to this Procedure, all the messages addressed to this actor will be processed here. In the first *if* block, upon recieving a string, actor writes it to the connection
*/
    if (msg instanceof String) {          connection.tell(TcpMessage.write((ByteString.fromString(istMessage)) msg), getSelf());      
    } else if (msg instanceof CommandFailed) {
      // OS kernel socket buffer was full        
    } else if (msg instanceof Received) {
/* 
Received event indicates data arrived from the server. Code is self explanatory
*/
      listener.tell(((Received) msg).data(), getSelf());       
    } else if (msg instanceof ConnectionClosed) {
      getContext().stop(getSelf());
     }
   }
  };
 }
}

In the initial iteration, we tried a simple straight forward solution using AKKA TCP.

1. One TCP actor connects to TCP server and receives from server.
2. On http request, using ask protocole send message to tcp actor. Actor make the request, send it to server.
3. Once response received from server, send it back to message originator. We might have to keep the sender actor ref for this.

Or we can use the completion stage. While making the request, send one completion stage also with the request to Actor.
Actor saves the future in memory. Once we get the response, get the future we saved earlier, and make it complete or exception based on the result.

In controller we can have something like

 public CompletionStage<Result> tcpWithFuture() {
    String message = getXMLMessageFromHttpRequest(request);
    CompletableFuture<String> future =new CompletableFuture<>();
    MessageProtocol message = new MessageProtocol(future, message);
    clientActor.tell(message,ActorRef.noSender());
    return future.thenApply(res-> {
        return ok(res.toString());
    });
}

and in TCP actor we need to keep the futures somewhere. For brevity we can have a HashMap.

if (msg instanceof MessageProtocol) {      
String toSend = ((MessageProtocol) msg).message;            connection.tell(TcpMessage.write(ByteString.fromString(toSend)), getSelf());      
} else if (msg instanceof Received) {
   String data = (Received) msg).data();   
   int requestId = getRequestId(data);
   if(map.contains(requestId)) {
     CompletableFuture future = map.get(requestId);
     String responseData = getResponseData(data);
     future.complete(responseData);
     map.remove(requestId)
  }
}

Issues

This solution was working fine for minimal load/concurrency. After 100 concurrent users we started to get issues.
As you can see from the sample code, we started getting socket buffer full error.

else if (msg instanceof CommandFailed) {
      // OS kernel socket buffer was full  
    LOG.error("OS kernel socket buffer was full");      
 }

The basic model of the TCP actor is that it has no internal buffering. Most of the time it happens when you are not handling the packet buffering and there are more packets to be sent. In our case that was the issue, not the system network buffer.

Solution

In the above method we are doing all the processing in the same TCP actor. So we can make most of processing parallel. Looking at the code we found that most of the processing was to

1. Construct XML request from http request.
2. Create http response from the response received from TCP server.

AKKA provides very easy to use tools and methods for concurrency. And AKKA encourages to use as many actors we need. In our case we can create multiple actors to process input request and another set of actors to process output response from TCP server.

to create multiple actors, we can use AKKA routing. First create an actor to process the request

public class RequestHandler extends UntypedActor {

@Override
public void onReceive(Object msg) throws Exception {
    if (msg instanceof String) {
        // Do request processing
....
....
getSender().tell(processedOutput, getSelf());
    }
}

}

Then create configuration

akka.actor.deployment {
  /RequestHandler {
  router = round-robin-pool
  nr-of-instances = 10
 }
}

then create multiple actors using

requestProcessor =   system.actorOf(Props.create(RequestHandler).withRouter(new           FromConfig()), "RequestHandler");

Notice the router parameter in config which says round-robin-pool. This means when we run this, there will be 10 identical actors created, and messages send to this actor will be delivered in round-robin fashion. AKKA provides many other algorithms like

1. akka.routing.RoundRobinRoutingLogic
2. akka.routing.RandomRoutingLogic
3. akka.routing.SmallestMailboxRoutingLogic
4. akka.routing.BroadcastRoutingLogic
5. akka.routing.ScatterGatherFirstCompletedRoutingLogic
6. akka.routing.TailChoppingRoutingLogic
7. akka.routing.ConsistentHashingRoutingLogic

Or you can even create your own routing logic.

Similiarly we can create multiple actors to process the output response from TCP server.

These changes helped to increase the concurrency of our application. But soon we crossed 1000 concurrent users on a minimal hardware, we started getting the old error again. Network buffer full!!. Again we have the buffering issue. AKKA TCP connection no internal buffering (i.e. one write at a time, meaning it can buffer one write until it has been passed on to the O/S kernel in full). Congestion needs to be handled at the user level, for both writes and reads. To solve this we can use any of the solution provided by AKKA itself ([throttling](The basic model of the TCP connection actor is that it has no internal buffering (i.e. it can only process one write at a time, meaning it can buffer one write until it has been passed on to the O/S kernel in full). Congestion needs to be handled at the user level, for both writes and reads.)). Once we implemented the throttling, we were able to manage much higher concurrency.

A sample project implementing above mentioned methods can be found here

On my previous assignment I was tasked with building a payment system for a large enterprise. Since it was an enterprise there were a bunch of subsystems to be integrated. One of the subsystem was a SOAP server!!..aaargh!!.

First step

There is no straight forward support for SOAP in play. In a servlets based solution supporting SOAP would be much simpler.

Since we were using play framework 2.5.x as our application framework, we need to make all the SOAP calls to be reactive to get maximum performance out of play framework.

To test the SOAP performance we will use a simple hello world SOAP server. This server has a single method which responds to requests. To monitor the performance better responses are delayed by 5 seconds.

Once we run the above application, a simple SOAP server will be available at localhost:3000. We can get the WSDL by navigating to http://127.0.0.1:3000/hw?wsdl from a browser. Or you can use wget.

  wget http://127.0.0.1:3000/hw?wsdl -O simple_soap_server.wsdl 

First we need to generate stub classes from the WSDL files with async support. We used wsimport tool with an addition configuration for generating async methods. Following external binding was used.

<bindings
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
 xmlns="http://java.sun.com/xml/ns/jaxws">
   <bindings node="wsdl:definitions">
     <enableAsyncMapping>true</enableAsyncMapping>
   </bindings>
</bindings>

Save the above binding to async.xml and use below command to generate stub.

wsimport helloworld.wsdl -keep -wsdllocation wsdl -b async.xml

In the sample code you can see this has generated one asynchronous method along with other methods.

 public Future<?> sayHelloAsync()

After the code generation call the methods asynchronously by passing a callback or using lambda functions.

example code

        HelloWorldServerImplService service =  new HelloWorldServerImplService();
    HelloWorldServer server = service.getHelloWorldServerImplPort();
    CompletableFuture<String> future = new CompletableFuture<>();
    server.sayHelloAsync("John",res-> {
        try{
            future.complete(res.get());
        }catch(Exception ex) {
          LOG.error("Exception {}",ex);
        }

    });
    return future.thenApply(res-> {
        return ok(res);
    });

Second step

Now we can start using SOAP services in our application. But there is a problem. Are these SOAP calls truly asynchronous?. We can check that from any jvm monitoring tool. Lets use jvisualvm for now. In the sample code provided, we know we are making SOAP call for every HTTP request. So fire few http requests using ab (apache benchmark)

ab -c 50 -n 1000 localhost:9000/async

In jvisualvm we can see output similar to the following result.

Important points note here are

1. Live thread count : 682
2. Thread state : parked (Color orange indicates parked state)

This shows that we are creating one thread per request. And most of the threads are waiting for the SOAP request to complete. This is exactly opposite to what we are trying to achieve. When we invoke SOAP request java internally uses JAX-WS implementation available to JVM. Unfortunately Oracle JRE’s default implementation uses blocking HTTP.

Solution : Use any other JAX-WS implementation which is not blocking in nature. Here we can use Apache Cxf library which has non blocking methods. To use apache cxf in our play application we need to add two dependencies to our build.sbt.

• cxf-rt-frontend-jaxws will make CXF the JAX-WS implementation in our application

• cxf-rt-transports-http-hc will allow CXF to use the non-blocking HTTP client

Java will pick apache cxf as its SOAP transport mechanism once its added in the class path. We don't need to make any other code change. Now we can run our apache benchmark against this and monitor the thread result.

Woha!!. As you can see from the screen the number of live threads have come down to 50 and most of the threads are in running(green) state.

What's happening here?

Apache cxf uses http client from Apache HttpComponents which uses event driven I/O model based on Java NIO. That means apache cxc threads, which is making the actual SOAP request over HTTP will not be waiting for the response to come back from server. Once the response is ready from the server, java NIO will invoke the correct thread. This means we will be able to server much larger number of request with minimum threads.

Further improvements

While doing performance testing we faces few more issues with SOAP services

1. Creating service port for every request
   Since WSDL files are loaded in service constructor, every SOAP request had to wait for the WSDL file to be loaded. We fixed this issue by making the service request global and making sure the WSDL files are loaded only once.

2. Apache cxf thread pool size was fixed at 50.
   This was fixed by keeping a cxf configuration file (cxf.xml) in conf folder. And keep below parameters.

       <property name="name" value="default" />
       <property name="lowWaterMark" value="200"/>
       <property name="highWaterMark" value="200"/>
       <property name="queueSize" value="512" />          
   

To read cxf.xml configuration file, play framework needs two more dependencies.

    "org.springframework" % "spring-expression" % "4.3.0.RELEASE"
    "org.springframework" % "spring-context" % "4.3.0.RELEASE"

Finally our application is ready for SOAP services. All the SOAP requests are now event driven non-blocking.

Sample code used can be found here