Next: 4 Compiling and Running Up: Ibis Programmer's Manual Previous: 2 Some Ibis concepts
Subsections

- 3.1 Program Preamble
- 3.2 Creating an Ibis Instance
- 3.3 StaticProperties
- 3.4 Setting up Communication
  - 3.4.1 Which Instance Does What?
  - 3.4.2 Creating a Port Type
  - 3.4.3 Creating Send and Receive Ports
  - 3.4.4 Setting Up a Connection
- 3.5 Connection upcalls, connection downcalls
- 3.6 Communicating
- 3.7 Finishing up
- 3.8 Ibis utilities
  - 3.8.1 PoolInfo
  - 3.8.2 Other utilities
- 3.9 Avoiding deadlocks

#
3 An Ibis Application

An Ibis application consists of several parts:

- Creating an Ibis instance in each instance of the application. An Ibis application can run on multiple hosts. On each of these hosts, an Ibis instance must be created.
- Setting up communication. Communication setup in Ibis consists of creating one or more send ports, through which messages can be sent, and creating one or more receive ports, through which messages can be received, and creating connections between them.
- Actually communicating. A send port is used to create a write message, which is sent to the receive ports that this send port is connected to.
- Finishing up. Connections must be closed, and each Ibis instance must be ended.

The next few subsections will discuss each of these steps in turn, illustrating them with parts of an RPC-style Ibis application. This application will have a client and a server. As this is a toy application, the server will have to compute the length of a string. The client will send the string, and receive the result. The server will have to do some other work as well, just to make things a little more interesting.

## 3.1 Program Preamble

Ibis applications need to import classes from the IPL (Ibis Portability Layer) package, which lives in `ibis.ipl`. We recommend that you simply import all `ibis.ipl` classes with one import line:

    import ibis.ipl.*;

Of course it is also possible to import only the needed classes, but this tends to result in a list of 10 or more `import`s.

##
3.2 Creating an Ibis Instance

All instances of a program that want to participate in an Ibis run must create an Ibis instance. To create an Ibis instance, the static `createIbis()` method of the `ibis.ipl.Ibis` class must be used. The specification of this method is:

    static Ibis createIbis(StaticProperties props,
                    ResizeHandler h)
            throws IbisException;

There may be several Ibis implementations available, and the system selects the best one for you, based on some user-specified requirements. These requirements tell the system what features must be supported by the selected Ibis. They are summarized in an object of the `ibis.ipl.StaticProperties` class, as discussed in Section 3.3

The second parameter of `createIbis()` specifies an upcall handler with `joined()` and `left()` upcalls that get called when an Ibis instance joins or leaves the run. In our RPC example, we will not use this, so we specify `null` instead. However, when such a `ResizeHandler` is used, its `joined()` upcall is called for every Ibis that joins the run, including the Ibis being created itself. Upcalls to the `ResizeHandler` must be explicitly enabled by invoking the `enableResizeUpcalls()` method of the Ibis just created. This ensures that the `ResizeHandler` has been able to do the necessary initializations.

The `enableResizeUpcalls()` method blocks until the `joined()` upcall for this Ibis has been invoked. Knowing which Ibises have joined the run, and how many there are, is often useful in dividing the work. See also section 3.4.1.

Now back to our example. In Secion 3.3 we will discuss the creation of a variable `props` describing the static properties required. For the moment we will just assume its existence, and create an Ibis instance as follows:

    Ibis ibis = null;
    try {
        ibis = Ibis.createIbis(props, null);
    } catch(NoMatchingIbisException e) {
        System.err.println("Could not find a matching Ibis");
        ...
    }

Note that the properties can be so specific that no matching Ibis can be found, and therefore `createIbis()` may throw an exception indicating this.

##
3.3 StaticProperties

Below is a snippet from the RPC example, constructing the static properties as required:

    StaticProperties props = new StaticProperties();
    props.add("communication", "OneToOne, Reliable, " +
                               "AutoUpcalls, ExplicitReceipt");
    props.add("serialization", "object");
    props.add("worldmodel", "closed");

This states that the selected Ibis must support reliable one-to-one communication, must support upcalls at the receiver side without the receiver having to poll for messages, and must also support explicit receipt of messages at the receiver side. (We want upcalls so that the server can do other work, and we want explicit receipt for the client side). In addition, the selected Ibis must support some form of object serialization (a string must be sent), and must support the ``closed'' worldmodel, which means that all participating Ibises join at the start of the run, and a synchronization takes place before `createIbis()` returns (in the example we have a client and a server).

A complete list of property values is given below. The possible property values of the `communication` property are (capitals are not significant):

OneToOne
  One-to-one (unicast) communication is supported (if an Ibis implementation does not support this, you may wonder what it does support).
OneToMany
  one-to-many (multicast) communication is supported (in Ibis terms: a sendport may connect to multiple receiveports).
ManyToOne
  many-to-one communication is supported (in Ibis terms: multiple sendports may connect to a single receiveport).
FifoOrdered
  messages from a send port are delivered to the receive ports it is connected to in the order in which they were sent.
Numbered
  all messages originating from any send port of a specific port type have a sequence number. This allows the application to do its own sequencing.
Reliable
  reliable communication is supported, that is, a reliable communication protocol is used. When not specified, an Ibis implementation may be chosen that does not explicitly support reliable communication.
AutoUpcalls
  upcalls are supported and polling for them is not required. This means that when the user creates a receiveport with an upcall handler installed, when a message arrives at that receive port, this upcall handler is invoked automatically.
PollUpcalls
  upcalls are supported but polling for them may be needed. When an Ibis implementation claims that it supports this, it may also do AutoUpcalls, but polling does no harm. When an application asks for this (and not AutoUpcalls), it must poll.
ExplicitReceipt
  explicit receive is supported. This is the alternative to upcalls for receiving messages.
ConnectionDowncalls
  connection downcalls are supported. This means that the user can invoke methods to see which connections were lost or created.
ConnectionUpcalls
  connection upcalls are supported. This means that an upcall handler can be installed that is invoked whenever a new connection arrives or a connection is lost.

The possible `serialization` properties are:

Byte
  Only the methods `readByte()`, `writeByte()`, `readArray(byte[])` and `writeArray(byte[])` are supported.
Data
  Only `read()`/`write()` and `readArray()`/`writeArray()` of primitive types are supported.
Object
  Some sort of object serialization is supported. An Ibis implementation will, of course, specify what kind of object serialization it supports. The ``Object'' property allows a user to just ask for object serialization, and not care if it is Ibis or Sun serialization.
Ibis
  Ibis serialization is supported. This is the fastest object serialization supported in Ibis. Its drawback is that it requires identical Java implementations on sender and receiver side; it also requires user-defined `writeObject()`/`readObject()` methods to be symmetrical, that is, each write in `writeObject()` must have a corresponding read in `readObject()` (and vice versa).
Sun
  Sun serialization (through `java.io.ObjectOutputStream/InputStream`) is supported.

The possible `worldmodel` properties are:

Open
  Ibises can join the run at any time during the run.
Closed
  The number of nodes involved in the run is known in advance and available from `ibis.util.PoolInfo` (see Section 3.8.1).

If a specific implementation of Ibis is required, that can be dealt with too. There is a property called `name`, which can be used to supply a nickname for the Ibis implementation that is required. The currently known nicknames are:

tcp
  This is an Ibis implementation on top of TCP sockets. It is currently probably the most stable Ibis around, and it supports almost all properties.
panda
  This is a message passing Ibis implementation on top of the Panda communication layer. It only supports the ``closed'' worldmodel, but on our system it gives much higher throughput and lower latencies because it runs on Myrinet instead of our (100Mbitps) Ethernet on which the TCP version runs.
net
  This is the most flexible Ibis implementation, and is becoming quite stable now. It can use multiple networks simultaneously.
nio
  This is an Ibis implementation on top of Java NIO.

Alternatively, you can specify the fully qualified name of an implementation of an `ibis.ipl.Ibis` subclass. You can use this to indicate that you want to use your own ibis implementation, for which no nickname exists.

A user running an Ibis application can override a property, or make it more specific. This is done by means of Java system properties, which can be set by means of a command line option or with the `System.setProperty` method. For instance, Sun and IBM JVMs support options of the form -Dproperty=value. The system property name is that of the Ibis static property, prefixed with ```ibis.`''. So, adding `-Dibis.serialization=ibis` to the command line will cause `createIbis()` to look for an Ibis implementation that supports Ibis serialization instead of any object serialization.

## 3.4 Setting up Communication

Setting up communication consists of several steps:

- create a port type;
- create a send and a receive port;
- set up connections between them.

The next few subsections discuss each of these steps in turn, but first we will discuss how to decide which Ibis instance does what.

###
3.4.1 Which Instance Does What?

Up until now, we have discussed only matters that all instances of the Ibis application should do, but now things become different. One instance of the application may want to send messages, while another instance may want to receive them. It may not even be clear which instance is going to do what. This can of course be solved with program parameters, but Ibis also provides a so-called registry (of type `ibis.ipl.Registry`), which is obtained through the `ibis.registry()` method. Ibis also provides the `ibis.ipl.IbisIdentifier` class. The `ibis.ipl.Ibis.identifier()` method returns such an Ibis identifier, which identifies this specific Ibis instance.

Using these methods it is possible to decide, in the RPC example, who is going to be the server by means of an ``election'': the Ibis registry provides a method `elect()` which (globally) selects one of a number of invokers. For our RPC example this could be done as follows:

    IbisIdentifier me = ibis.identifier();
    Registry registry = ibis.registry();
    IbisIdentifier server = registry.elect("Server");
    boolean iAmServer = server.equals(me);

In our example, one instance of the program is the server and all other instances are clients. Of course, the client and the server can also be different programs altogether.

The `ResizeHandler`, as discussed in Section 3.2, can be used to keep track of the number of Ibis instances currently involved in the run.

### 3.4.2 Creating a Port Type

To be able to create send and receive ports, it is first necessary to create one or more port types. A port type is an object of type `ibis.ipl.PortType`. Within an Ibis instance, multiple port types, with different properties, can be created. The properties of a port type are, like the required properties of an Ibis implementation, specified by a `StaticProperties` object. A port type is identified by its name, together with these properties. The `Ibis` class contains a method to create a port type, specified as follows:

    PortType createPortType(String name,
                            StaticProperties portprops)
            throws IbisException, java.io.IOException;

For our RPC example program, we would create a port type with properties as discussed in Section 3.2:

    StaticProperties portprops = new StaticProperties();
    portprops.add("communication",
                  "OneToOne, Reliable, " +
                      "AutoUpcalls, ExplicitReceipt");
    portprops.add("serialization", "object");
    PortType porttype = null;
    try {
        porttype = ibis.createPortType("RPC port", portprops);
    } catch(Exception e) {
        ...
    }

In general, the port properties should be a subset of the properties specified when creating the Ibis instance. If a property is specified that was not specified when creating the Ibis instance, this may result in an `IbisException`, depending on the strictness of the Ibis implementation. An `IbisException` will also result when the same name is already used for a port type with different properties. Port types are registered with a name server (this will be discussed in Section 4). If communication with this name server fails, a `java.io.IOException` is thrown.

### 3.4.3 Creating Send and Receive Ports

The `PortType` class contains several variants of a method `createSendPort()` that creates a send port (of type `ibis.ipl.SendPort`) and also several variants of a method `createReceivePort()` that creates a receive port (of type `ibis.ipl.ReceivePort`). See the API for an exhaustive list of variants.

In Ibis, receive ports usually have specific names, so that a send port can set up a connection to a receive port. In contrast, send ports usually are anonymous, because a receive port cannot initiate a connection.

For our RPC example, the server will have to create a receive port to receive a request and a send port to send an answer. The server is not allowed to block waiting for a request, so it will want a receive port that enables upcalls. To do that, the server must first define a class that implements the `ibis.ipl.Upcall` interface. This interface contains one method:

    void upcall(ReadMessage m) throws java.io.IOException;

We will go into the details of a `ReadMessage` in Section 3.6. For now, we will assume that there is a class `RpcUpcall` that implements this interface, and that the application has a field `rpcUpcall` of this type.

    try {
        SendPort serverSender = porttype.createSendPort();
        ReceivePort serverReceiver =
            porttype.createReceivePort("server", rpcUpcall);
    } catch(java.io.IOException e) {
        ....
    }

The client will have to create a send port to send a request and a receive port to receive an answer. The client is allowed to block waiting for an answer, so it will want a receive port that enables explicit receipt. So, the client will create an anonymous server port, and a named receive port that enables explicit receipt (no upcall handler is supplied):

    try {
        SendPort clientSender = porttype.createSendPort();
        ReceivePort clientReceiver =
            porttype.createReceivePort("client");
    } catch(java.io.IOException e) {
        ....
    }

When a receive port is created, it will not immediately accept connections. This must be explicitly enabled by invoking the `enableConnections()` method. Incoming connection attempts are kept pending until connections are enabled. So, the creator of the receive port can determine when he is ready to accept connections. If the receive port is configured for upcalls, these must explicitly be enabled by invoking the `enableUpcalls()` method. Again, incoming messages are kept pending until upcalls are enabled.

### 3.4.4 Setting Up a Connection

Now that we have send ports and receive ports, it is time to set up connections between them. A connection is initiated by the `connect()` method of `ibis.ipl.SendPort`. Here is its specification:

    void connect (ReceivePortIdentifier r) throws IOException;

This version blocks until an accept or deny is received. An `IOException` is thrown when the connection could not be established. There also is a `connect()` version with a time-out.

So, we need a `ibis.ipl.ReceivePortIdentifier` to set up the connection. This identifier can be obtained through the Ibis registry, by means of the `lookupReceivePort()` method, which has the following specification:

    ReceivePortIdentifier lookupReceivePort(String name)
            throws IOException;

This method blocks until a receive port with the specified name is found. Our RPC server would set up the following connection:

    try {
        ReceivePortIdentifier client
                = registry.lookupReceivePort("client");
        serverSender.connect(client);
    } catch(IOException e) {
        ...
    }

Our RPC client would set up the following connection:

    try {
        ReceivePortIdentifier server
                = registry.lookupReceivePort("server");
        clientSender.connect(server);
    } catch(IOException e) {
        ...
    }

This completes the connection setup.

Note that a send port can set up connections to more than one receive port (if the port type supports the `OneToMany` communication property). Also, multiple send ports can set up connections to the same receive port (if the port type supports the `ManyToOne` communication property).

##
3.5 Connection upcalls, connection downcalls

Sometimes it is useful for an application to know which send ports are connected to a receive port, and vice versa, or which connections are being closed. Ibis implementations may support two different mechanisms for obtaining this type of information: connection upcalls and connection downcalls. See Section 3.3 for the corresponding properties.

When a port type is configured for using connection upcalls, a receive port may be instantiated with a `ReceivePortConnectUpcall` object. This is an interface with two methods:

    boolean gotConnection(ReceivePort me,
                          SendPortIdentifier applicant);
    void lostConnection(ReceivePort me,
                        SendPortIdentifier johndoe,
                        Exception reason);

The `gotConnection()` method gets called when a send port attempts to connect to the receive port at hand. An implementation of this method can decide whether to allow this connection or not by returning `true` or `false`. The `lostConnection()` method gets called when an existing connection to the receive port at hand gets lost for some reason.

A send port can be instantiated with a `SendPortConnectUpcall` object. This is an interface with a single method:

    void lostConnection(SendPort me,
                        ReceivePortIdentifier johndoe,
                        Exception reason);

This method is called when an existing connection from the send port at hand gets lost for some reason. Note that there is no `gotConnection()` counterpart, because it is always the send port that initiates a connection.

When a port type is configured for using connection downcalls, receive ports and send ports of this type maintain connection information, and support methods that allow the user to obtain this information. A receive port has the following methods:

    SendPortIdentifier[] newConnections();
    SendPortIdentifier[] lostConnections();
    SendPortIdentifier[] connectedTo();

`newConnections()` returns the send port identifiers of the connections that are new since the last call or the start of the program. `lostConnections()` returns the send port identifiers of the connections that were lost since the last call or the start of the program. `connectedTo()` returns the send port identifiers of all connections to this receive port. A send port has the following methods:

    ReceivePortIdentifier[] newConnections();
    ReceivePortIdentifier[] lostConnections();
    ReceivePortIdentifier[] connectedTo();

which do exactly the same as their receive port counterparts.

##
3.6 Communicating

Communication in Ibis consists of messages, sent from a send port, and received at a receive port. When a sender wants to send a message, it will first have to obtain one from the send port. Such a message is of type `ibis.ipl.WriteMessage` and is obtained by means of the `newMessage()` method of `SendPort`, which is specified as follows:

    WriteMessage newMessage() throws IOException;

For a given send port, only one message can be alive at any time. When a new message is requested while a message is alive, the request is blocked until the live message is finished.

Once a write message is obtained, data can be written to it. A write message has various methods for the different types of data that can be written to it. For instance, the `writeInt()` method can be used to write an integer value, and the `writeObject()` method can be used to write an object. The kind of data that can be written to the message depends on the serialization property specified when the port type was created. The most general form is object serialization, which supports all write methods in a write message. The `data` serialization property does not allow use of the `writeObject()` method, but does allow the use of all other write methods. The `byte` serialization property only allows use of the `writeByte()` method and the `writeArray(byte[])` method.

Once there is a considerable amount of data in the message, Ibis can be given a hint to start sending, using the `send()` method. This hint is not needed, however. When the message is complete, the message can be sent out by invoking the `finish()` method.

Our client in the RPC example could have the following:

    int obtainLength(String s) throws IOException {
        WriteMessage w = clientSender.newMessage();
        w.writeObject(s);
        w.finish();
        ...
    }

At the receiving side, a message can be received in two ways, depending on how the receive port was created: either by means of an upcall, or by means of an explicit receive. For each write method in the `WriteMessage` type, there is a corresponding read method in the `ReadMessage` type. For a given receive port, only one message can be alive at any time. A read message is alive until it is finished (by a `finish()` call), or the upcall returns.

Now, let us present some more code of our RPC example, this time from the server:

    public void upcall(ReadMessage m) throws IOException {
        String s = (String) m.readObject();
        int len = s.length();
        m.finish();
        WriteMessage w = serverSender.newMessage();
        w.writeInt(len);
        w.finish();
    }

Note that the read message is finished before replying to the request. To prevent deadlocks, upcalls are not allowed to block (call `Thread.wait()`) or access the network (write a message or read another message) as long as a read message is active.

Now, we can also finish the `obtainLength()` method of the client:

        ...
        ReadMessage r = clientReceiver.receive();
        int len = r.readInt();
        r.finish();
        return len;
    }

## 3.7 Finishing up

Closing of a connection is initiated by closing a send port by means of the `close()` method. The `ReceivePort` class also has a `close()` method, but this method blocks until all send ports that have a connection to it are closed. So, send ports have to be closed first.

Our RPC client will do the following:

    clientSender.close();
    clientReceiver.close();

and the code of the server should be clear by now.

Ibis itself must also be ended. Both our client and our server should invoke the `Ibis.end()` method:

    ibis.end();

As of Java 1.3, it is also possible to add a so-called shutdown hook. This could be done right after the Ibis instance is created:

    Runtime.getRuntime().addShutdownHook(new Thread() {
        public void run() {
            try {
                ibis.end();
            } catch (IOException e) {
            }
        }
    });

This shutdown hook gets invoked when the program terminates, and forcibly closes all ports.

##
3.8 Ibis utilities

The `ibis.util` package contains several utilities that may be useful for Ibis applications. We will discuss some of them here.

###
3.8.1 PoolInfo

The `ibis.util.PoolInfo` utility provides methods for finding out information about the nodes involved in a closed-world run, such as:

- the total number of hosts involved in the run.
- the rank number of the current host in the pool.
- the host names of the hosts involved in the run.
- the `InetAddress` of the hosts involved in the run.

A `PoolInfo` instance is created with its static method `createPoolInfo()`. It depends on the following system properties:

ibis.pool.total_hosts
  This system property must be present for closed-world runs. It indicates the total number of hosts involved in the run.
ibis.pool.host_names
  If present, this system property contains the list of host names involved in the run. If not present, `createPoolInfo()` instantiates a `PoolInfoClient` that will collect this information during startup. A `PoolInfoClient` uses a `PoolInfoServer` to collect the required information. This `PoolInfoServer` usually is started by the Ibis nameserver (see Section 4.1).

ibis.pool.host_number
  If present, this system property indicates the rank number of the current host. If not present, the system will provide a rank number.

###
3.8.2 Other utilities

The `ibis.util.Stats` utility contains methods for computing the mean and standard deviation of an array of numbers. A timer utility is provided in `ibis.util.Timer`. See the Ibis API for other utilities. Most of these are used in Ibis implementations, but may have other uses.

##
3.9 Avoiding deadlocks

As with most communication layers, it is quite easy to write code that deadlocks with Ibis. For example, if you have two Ibis instances that are writing large amounts of data to each other, and there are no readers for this data active, this will almost certainly result in a deadlock, because network buffers will fill up, causing the senders to block. Such a deadlock can be avoided by having a separate reader thread, or by installing an upcall handler for the incoming message.

Another common source of deadlocks is if you have a port type that specifies the `ManyToOne` as well as the `OneToMany` communication property. Multiple hosts doing simultaneous multicasts is a well-known source of deadlocks, because most systems do not implement a functioning flow-control for these cases.

Next: 4 Compiling and Running Up: Ibis Programmer's Manual Previous: 2 Some Ibis concepts