Chapter 8. Marshalling Custom Java Objects with ProtoStream

Data Grid uses a ProtoStream API to encode and decode Java objects into Protocol Buffers (Protobuf); a language-neutral, backwards compatible format.

8.1. Protobuf Schemas

Protocol Buffers, Protobuf, schemas provide structured representations of your Java objects.

You define Protobuf message types .proto schema files as in the following example:

package book_sample;

message Book {
    optional string title = 1;
    optional string description = 2;
    optional int32 publicationYear = 3; // no native Date type available in Protobuf

    repeated Author authors = 4;
}

message Author {
    optional string name = 1;
    optional string surname = 2;
}

The preceding .library.proto file defines an entity (Protobuf message type) named Book that is contained in the book_sample package. Book declares several fields of primitive types and an array (Protobuf repeatable field) named authors, which is the Author message type.

Protobuf Messages

  • You can nest messages but the resulting structure is strictly a tree, never a graph.
  • Type inheritance is not possible.
  • Collections are not supported but you can emulate arrays with repeated fields.

8.2. ProtoStream Serialization Contexts

A ProtoStream SerializationContext contains Protobuf type definitions for custom Java objects, loaded from .proto schema files, and the accompanying Marshallers for the objects.

The SerializationContextInitializer interface registers Java objects and marshallers so that the ProtoStream library can encode your custom objects to Protobuf format, which then enables Data Grid to transmit and store your data.

8.3. ProtoStream Types

ProtoStream can handle the following types, as well as the unboxed equivalents in the case of primitive types, without any additional configuration:

  • String
  • Integer
  • Long
  • Double
  • Float
  • Boolean
  • byte[]
  • Byte
  • Short
  • Character
  • java.util.Date
  • java.time.Instant

To marshall any other Java objects, you must generate, or manually create, SerializationContextInitializer implementations that register .proto schemas and marshallers with a SerializationContext.

8.4. Generating Serialization Context Initializers

Data Grid provides an protostream-processor artifact that can generate .proto schemas and SerializationContextInitializer implementations from annotated Java classes.

Procedure

  1. Add the protostream-processor dependency to your pom.xml.

    <dependencyManagement>
      <dependencies>
        <dependency>
          <groupId>org.infinispan</groupId>
          <artifactId>infinispan-bom</artifactId>
          <version>${version.infinispan}</version>
          <type>pom</type>
        </dependency>
        <dependency>
          <groupId>org.infinispan.protostream</groupId>
          <artifactId>protostream-processor</artifactId>
          <scope>provided</scope>
        </dependency>
      </dependencies>
    </dependencyManagement>
  2. Annotate the Java objects that you want to marshall with @ProtoField and @ProtoFactory.

    Book.java

    import org.infinispan.protostream.annotations.ProtoFactory;
    import org.infinispan.protostream.annotations.ProtoField;
    ...
    
    public class Book {
       @ProtoField(number = 1)
       final String title;
    
       @ProtoField(number = 2)
       final String description;
    
       @ProtoField(number = 3, defaultValue = "0")
       final int publicationYear;
    
       @ProtoField(number = 4, collectionImplementation = ArrayList.class)
       final List<Author> authors;
    
       @ProtoFactory
       Book(String title, String description, int publicationYear, List<Author> authors) {
          this.title = title;
          this.description = description;
          this.publicationYear = publicationYear;
          this.authors = authors;
       }
       // public Getter methods omitted for brevity
    }

    Author.java

    import org.infinispan.protostream.annotations.ProtoFactory;
    import org.infinispan.protostream.annotations.ProtoField;
    
    public class Author {
       @ProtoField(number = 1)
       final String name;
    
       @ProtoField(number = 2)
       final String surname;
    
       @ProtoFactory
       Author(String name, String surname) {
          this.name = name;
          this.surname = surname;
       }
       // public Getter methods omitted for brevity
    }

  3. Define an interface that extends SerializationContextInitializer and is annotated with @AutoProtoSchemaBuilder.

    @AutoProtoSchemaBuilder(
          includeClasses = {
                Book.class,
                Author.class,
          },
          schemaFileName = "library.proto", 1
          schemaFilePath = "proto/", 2
          schemaPackageName = "book_sample")
    interface LibraryInitializer extends SerializationContextInitializer {
    }
    1
    names the generated .proto schema file.
    2
    sets the path under target/classes where the schema file is generated.

During compile-time, protostream-processor generates a concrete implementation of the interface that you can use to initialize a ProtoStream SerializationContext. By default, implementation names are the annotated class name with an "Impl" suffix.

Examples

The following are examples of a generated schema file and implementation:

target/classes/proto/library.proto

// File name: library.proto
// Generated from : org.infinispan.commons.marshall.LibraryInitializer

syntax = "proto2";

package book_sample;



message Book {

    optional string title = 1;

    optional string description = 2;

    optional int32 publicationYear = 3 [default = 0];

    repeated Author authors = 4;
}


message Author {

    optional string name = 1;

    optional string surname = 2;
}

LibraryInitializerImpl.java

/*
 Generated by org.infinispan.protostream.annotations.impl.processor.AutoProtoSchemaBuilderAnnotationProcessor
 for class org.infinispan.commons.marshall.LibraryInitializer
 annotated with @org.infinispan.protostream.annotations.AutoProtoSchemaBuilder(dependsOn=, service=false, autoImportClasses=false, excludeClasses=, includeClasses=org.infinispan.commons.marshall.Book,org.infinispan.commons.marshall.Author, basePackages={}, value={}, schemaPackageName="book_sample", schemaFilePath="proto/", schemaFileName="library.proto", className="")
 */

package org.infinispan.commons.marshall;

/**
 * WARNING: Generated code!
 */
@javax.annotation.Generated(value = "org.infinispan.protostream.annotations.impl.processor.AutoProtoSchemaBuilderAnnotationProcessor",
      comments = "Please do not edit this file!")
@org.infinispan.protostream.annotations.impl.OriginatingClasses({
      "org.infinispan.commons.marshall.Author",
      "org.infinispan.commons.marshall.Book"
})
/*@org.infinispan.protostream.annotations.AutoProtoSchemaBuilder(
   className = "LibraryInitializerImpl",
   schemaFileName = "library.proto",
   schemaFilePath = "proto/",
   schemaPackageName = "book_sample",
   service = false,
   autoImportClasses = false,
   classes = {
      org.infinispan.commons.marshall.Author.class,
      org.infinispan.commons.marshall.Book.class
   }
)*/
public class LibraryInitializerImpl implements org.infinispan.commons.marshall.LibraryInitializer {

   @Override
   public String getProtoFileName() { return "library.proto"; }

   @Override
   public String getProtoFile() { return org.infinispan.protostream.FileDescriptorSource.getResourceAsString(getClass(), "/proto/library.proto"); }

   @Override
   public void registerSchema(org.infinispan.protostream.SerializationContext serCtx) {
      serCtx.registerProtoFiles(org.infinispan.protostream.FileDescriptorSource.fromString(getProtoFileName(), getProtoFile()));
   }

   @Override
   public void registerMarshallers(org.infinispan.protostream.SerializationContext serCtx) {
      serCtx.registerMarshaller(new org.infinispan.commons.marshall.Book$___Marshaller_cdc76a682a43643e6e1d7e43ba6d1ef6f794949a45e1a8bc961046cda44c9a85());
      serCtx.registerMarshaller(new org.infinispan.commons.marshall.Author$___Marshaller_9b67e1c1ecea213b4207541b411fb9af2ae6f658610d2a4ca9126484d57786d1());
   }
}

8.5. Manually Implementing Serialization Context Initializers

In some cases you might need to manually define .proto schema files and implement ProtoStream marshallers. For example, if you cannot modify Java object classes to add annotations.

Procedure

  1. Create a .proto schema with Protobuf messages.

    package book_sample;
    
    message Book {
        optional string title = 1;
        optional string description = 2;
        optional int32 publicationYear = 3; // no native Date type available in Protobuf
    
        repeated Author authors = 4;
    }
    
    message Author {
        optional string name = 1;
        optional string surname = 2;
    }
  2. Use the org.infinispan.protostream.MessageMarshaller interface to implement marshallers for your classes.

    BookMarshaller.java

    import org.infinispan.protostream.MessageMarshaller;
    
    public class BookMarshaller implements MessageMarshaller<Book> {
    
       @Override
       public String getTypeName() {
          return "book_sample.Book";
       }
    
       @Override
       public Class<? extends Book> getJavaClass() {
          return Book.class;
       }
    
       @Override
       public void writeTo(MessageMarshaller.ProtoStreamWriter writer, Book book) throws IOException {
          writer.writeString("title", book.getTitle());
          writer.writeString("description", book.getDescription());
          writer.writeInt("publicationYear", book.getPublicationYear());
          writer.writeCollection("authors", book.getAuthors(), Author.class);
       }
    
       @Override
       public Book readFrom(MessageMarshaller.ProtoStreamReader reader) throws IOException {
          String title = reader.readString("title");
          String description = reader.readString("description");
          int publicationYear = reader.readInt("publicationYear");
          List<Author> authors = reader.readCollection("authors", new ArrayList<>(), Author.class);
          return new Book(title, description, publicationYear, authors);
       }
    }

    AuthorMarshaller.java

    import org.infinispan.protostream.MessageMarshaller;
    
    public class AuthorMarshaller implements MessageMarshaller<Author> {
    
       @Override
       public String getTypeName() {
          return "book_sample.Author";
       }
    
       @Override
       public Class<? extends Author> getJavaClass() {
          return Author.class;
       }
    
       @Override
       public void writeTo(MessageMarshaller.ProtoStreamWriter writer, Author author) throws IOException {
          writer.writeString("name", author.getName());
          writer.writeString("surname", author.getSurname());
       }
    
       @Override
       public Author readFrom(MessageMarshaller.ProtoStreamReader reader) throws IOException {
          String name = reader.readString("name");
          String surname = reader.readString("surname");
          return new Author(name, surname);
       }
    }

  3. Create a SerializationContextInitializer implementation that registers the .proto schema and the ProtoStream marshaller implementations with a SerializationContext.

    ManualSerializationContextInitializer.java

    import org.infinispan.protostream.FileDescriptorSource;
    import org.infinispan.protostream.SerializationContext;
    import org.infinispan.protostream.SerializationContextInitializer;
    ...
    
    public class ManualSerializationContextInitializer implements SerializationContextInitializer {
       @Override
       public String getProtoFileName() {
          return "library.proto";
       }
    
       @Override
       public String getProtoFile() throws UncheckedIOException {
          // Assumes that the file is located in a Jar's resources, we must provide the path to the library.proto file
          return FileDescriptorSource.getResourceAsString(getClass(), "/" + getProtoFileName());
       }
    
       @Override
       public void registerSchema(SerializationContext serCtx) {
          serCtx.registerProtoFiles(FileDescriptorSource.fromString(getProtoFileName(), getProtoFile()));
       }
    
       @Override
       public void registerMarshallers(SerializationContext serCtx) {
          serCtx.registerMarshaller(new AuthorMarshaller());
          serCtx.registerMarshaller(new BookMarshaller());
       }
    }