Custom Serialization

You can use your own custom stateful serialization for OBJECT fields creating a serialization factory that implements Serialization.
As an example, you can take a look to AvroFieldSerialization.
Implementing FieldConfigurable your serialization factory will receive the metadata associated with the OBJECT field to be serialized.

Note how the AvroFieldSerialization's state is retrieved from the field's properties avro.schema and avro.reflection

  public class AvroFieldSerialization<T> implements Serialization<T>,FieldConfigurable{
    private Schema schema;
    private boolean isReflect;
    
    public AvroFieldSerialization(){
    }
  
    @Override
    public Deserializer<T> getDeserializer(Class<T> clazz) {
      return new AvroFieldDeserializer<T>(schema, isReflect);
    }
  
    @Override
    public Serializer<T> getSerializer(Class<T> clazz) {
      return new AvroFieldSerializer(schema, isReflect);
    }
  
    @Override
    public void setFieldProperties(Map<String,String> properties) {
      schema = Schema.parse(properties.get("avro.schema"));
      String r = properties.get("avro.reflection");
      isReflect = (r != null) && Boolean.parseBoolean(r);
    }
 

Built-in serialization in Pangool

There exist 4 types of built-in serializations supported in Pangool for OBJECT fields. However, any other custom serialization can be added, as explained above. Let's show how to define fields with them:

Writables

  List<Field> fields = new ArrayList<Field>(); 
  fields.add(Field.createObject("my_writable", TextLongWritable.class);
  Schema schema = new Schema("schema", fields);
  
  Tuple tuple = new Tuple(schema);
  tuple.set("my_writable",new TextLongWritable("eric", 89L));

Avro

  List<Field> fields = new ArrayList<Field>(); 
  fields.add(Fields.createAvroField("my_avro_field", myAvroSchema));
  Schema schema = new Schema("schema", fields);
  
  Tuple tuple = new Tuple(schema);
  Record avroRecord = new Record(myAvroSchema); //Avro Record
  tuple.set("my_avro_field", avroRecord);

Thrift

  List<Field> fields = new ArrayList<Field>(); 
  fields.add(Field.createObject("my_thrift_field", MyThrifObject.class));
  Schema schema = new Schema("schema", fields);
  
  Tuple tuple = new Tuple(schema1);
  tuple.set("my_thrift_field",new MyThriftObject());

Protostuff

  List<Field> fields = new ArrayList<Field>(); 
  fields.add(Field.createObject("my_protostuff_field", MyProtoStuffObject.class));
  Schema schema = new Schema("schema", fields);
  
  Tuple tuple = new Tuple(schema);
  tuple.set("my_protostuff_field",new MyProtostuffObject());

Pangool (Tuple inside Tuple)

(It is possible to serialize Pangool Tuples inside Pangool Tuples.)

  Schema tupleSchema = new Schema(...);
  List<Field> fields = new ArrayList<Field>(); 
  fields.add(Fields.createTupleField("my_tuple_field", tupleSchema));
  Schema metaTupleSchema = new Schema("schema", fields);
  
  Tuple tuple = new Tuple(tupleSchema);
  ...
  Tuple metaTuple = new Tuple(metaTupleSchema);
  metaTuple.set("my_tuple_field", tuple);

Next: Custom Comparators »