From Cereal​

Cereal and the JaiScript archives share a shape: a type describes itself once, in a serialize function or a save/load pair, and an archive drives it in both directions. Three things are more than spelling:

• The files. The archives write their own JSON and binary formats, so a file Cereal wrote is converted once: loaded with Cereal, saved with JaiScript.
• Registration. A polymorphic type gets its identity from a jai::property_owner base or a jai::registrar, in place of CEREAL_REGISTER_TYPE.
• Versions. Both formats name every property in every record, so most of what CEREAL_CLASS_VERSION did is unnecessary.

A Boost.Serialization codebase takes the same path; From Boost.Serialization lists its spellings. The archives are documented in Serialization.

The concept map​

Headers: <jaiscript/serialization/convenience.hpp> for the archives and the one-call entry points, <jaiscript/serialization/construct.hpp> for load_and_construct, <jaiscript/core/registrar.hpp> for registrars, and <jaiscript/properties/macros.hpp> with <jaiscript/properties/property_serialization.hpp> for properties.

Archives and entry points​

Names​

Serialization functions​

Versions and polymorphism​

Pointers and identity​

Containers and standard types​

Construction, access, context and errors​

A worked conversion​

A small save game, five ways: items, an inventory with private fields, and a player that holds the inventory through a shared_ptr and derives hpFraction on load. Every form but JAI_PROPERTY, which is shorter, keeps each type and field at the same line number.

#include <cereal/archives/json.hpp>
#include <cereal/types/memory.hpp>
#include <cereal/types/string.hpp>
#include <cereal/types/vector.hpp>
#include <sstream>

struct Item {
    std::string name;
    int count = 1;

    template<class Archive>
    void serialize(Archive& ar) {
        ar(CEREAL_NVP(name));
        ar(CEREAL_NVP(count));
    }
};

class Inventory {
public:
    void add(const Item& item) { items.push_back(item); }

private:
    friend class cereal::access;
    std::vector<Item> items;
    int capacity = 20;

    template<class Archive>
    void serialize(Archive& ar) {
        ar(CEREAL_NVP(items));
        ar(CEREAL_NVP(capacity));
    }
};

struct Player {
    std::string name;
    int hp = 100;
    std::shared_ptr<Inventory> inventory = std::make_shared<Inventory>();
    float hpFraction = 1.0f;

    template<class Archive>
    void save(Archive& ar) const {
        ar(CEREAL_NVP(name));
        ar(CEREAL_NVP(hp));
        ar(CEREAL_NVP(inventory));
    }

    template<class Archive>
    void load(Archive& ar) {
        ar(CEREAL_NVP(name));
        ar(CEREAL_NVP(hp));
        ar(CEREAL_NVP(inventory));
        hpFraction = hp / 100.0f;
    }
};

int main() {
    Item potion;
    potion.name = "potion";
    potion.count = 3;

    Player player;
    player.name = "Vex";
    player.hp = 40;
    player.inventory->add(potion);

    std::stringstream stream;
    {
        cereal::JSONOutputArchive ar(stream);
        ar(player);
    }
    std::string text = stream.str();

    Player loaded;
    {
        cereal::JSONInputArchive ar(stream);
        ar(loaded);
    }
}

#include <cereal/archives/json.hpp>
#include <cereal/types/memory.hpp>
#include <cereal/types/string.hpp>
#include <cereal/types/vector.hpp>
#include <sstream>

struct Item {
    std::string name;
    int count = 1;
};

template<class Archive>
void serialize(Archive& ar, Item& item) {
    ar(cereal::make_nvp("name", item.name));
    ar(cereal::make_nvp("count", item.count));
}

class Inventory {
public:
    void add(const Item& item) { items.push_back(item); }

private:
    template<class Archive> friend void serialize(Archive& ar, Inventory& inventory);
    std::vector<Item> items;
    int capacity = 20;
};

template<class Archive>
void serialize(Archive& ar, Inventory& inventory) {
    ar(cereal::make_nvp("items", inventory.items));
    ar(cereal::make_nvp("capacity", inventory.capacity));
}

struct Player {
    std::string name;
    int hp = 100;
    std::shared_ptr<Inventory> inventory = std::make_shared<Inventory>();
    float hpFraction = 1.0f;
};

template<class Archive>
void save(Archive& ar, const Player& player) {
    ar(cereal::make_nvp("name", player.name));
    ar(cereal::make_nvp("hp", player.hp));
    ar(cereal::make_nvp("inventory", player.inventory));
}

template<class Archive>
void load(Archive& ar, Player& player) {
    ar(cereal::make_nvp("name", player.name));
    ar(cereal::make_nvp("hp", player.hp));
    ar(cereal::make_nvp("inventory", player.inventory));
    player.hpFraction = player.hp / 100.0f;
}

int main() {
    Item potion;
    potion.name = "potion";
    potion.count = 3;

    Player player;
    player.name = "Vex";
    player.hp = 40;
    player.inventory->add(potion);

    std::stringstream stream;
    {
        cereal::JSONOutputArchive ar(stream);
        ar(player);
    }
    std::string text = stream.str();

    Player loaded;
    {
        cereal::JSONInputArchive ar(stream);
        ar(loaded);
    }
}

#include <jaiscript/core/engine.hpp>
#include <jaiscript/serialization/convenience.hpp>
#include <memory>
#include <string>
#include <vector>

struct Item {
    std::string name;
    int count = 1;

    template<class Archive>
    void serialize(Archive& ar) {
        ar(JAI_NVP(name));
        ar(JAI_NVP(count));
    }
};

class Inventory {
public:
    void add(const Item& item) { items.push_back(item); }

private:
    friend jai::access;
    std::vector<Item> items;
    int capacity = 20;

    template<class Archive>
    void serialize(Archive& ar) {
        ar(JAI_NVP(items));
        ar(JAI_NVP(capacity));
    }
};

struct Player {
    std::string name;
    int hp = 100;
    std::shared_ptr<Inventory> inventory = std::make_shared<Inventory>();
    float hpFraction = 1.0f;

    template<class Archive>
    void save(Archive& ar) const {
        ar(JAI_NVP(name));
        ar(JAI_NVP(hp));
        ar(JAI_NVP(inventory));
    }

    template<class Archive>
    void load(Archive& ar) {
        ar(JAI_NVP(name));
        ar(JAI_NVP(hp));
        ar(JAI_NVP(inventory));
        hpFraction = hp / 100.0f;
    }
};

int main() {
    Item potion;
    potion.name = "potion";
    potion.count = 3;

    Player player;
    player.name = "Vex";
    player.hp = 40;
    player.inventory->add(potion);

    auto eng = jai::engine::make();
    std::string text = jai::to_json(*eng, player);

    Player loaded = jai::from_json<Player>(*eng, text);
}

#include <jaiscript/core/engine.hpp>
#include <jaiscript/properties/macros.hpp>
#include <jaiscript/properties/property_serialization.hpp>
#include <jaiscript/serialization/convenience.hpp>
#include <memory>
#include <string>
#include <vector>

struct Item : jai::property_owner<Item> {
    JAI_PROPERTY((std::string), name);
    JAI_PROPERTY((int), count, 1);
};

class Inventory : public jai::property_owner<Inventory> {
public:
    void add(const Item& item) { items->push_back(item); }

private:
    JAI_PROPERTY((std::vector<Item>), items);
    JAI_PROPERTY((int), capacity, 20);
};

struct Player : jai::property_owner<Player> {
    JAI_PROPERTY((std::string), name);
    JAI_PROPERTY((int), hp, 100);
    JAI_PROPERTY((std::shared_ptr<Inventory>), inventory, std::make_shared<Inventory>());
    JAI_TRANSIENT_PROPERTY((float), hpFraction, 1.0f);

    template<class Archive>
    void load(Archive& ar) {
        load_owned_properties(ar);
        hpFraction = hp / 100.0f;
    }
};

int main() {
    Item potion;
    potion.name = "potion";
    potion.count = 3;

    Player player;
    player.name = "Vex";
    player.hp = 40;
    player.inventory->add(potion);

    auto eng = jai::engine::make();
    std::string text = jai::to_json(*eng, player);

    Player loaded = jai::from_json<Player>(*eng, text);
}

#include <jaiscript/core/engine.hpp>
#include <jaiscript/serialization/convenience.hpp>
#include <memory>
#include <string>
#include <vector>

struct Item {
    std::string name;
    int count = 1;
};

template<class Archive>
void serialize(Archive& ar, Item& item) {
    ar("name", item.name);
    ar("count", item.count);
}

class Inventory {
public:
    void add(const Item& item) { items.push_back(item); }

private:
    template<class Archive> friend void serialize(Archive& ar, Inventory& inventory);
    std::vector<Item> items;
    int capacity = 20;
};

template<class Archive>
void serialize(Archive& ar, Inventory& inventory) {
    ar("items", inventory.items);
    ar("capacity", inventory.capacity);
}

struct Player {
    std::string name;
    int hp = 100;
    std::shared_ptr<Inventory> inventory = std::make_shared<Inventory>();
    float hpFraction = 1.0f;
};

template<class Archive>
void save(Archive& ar, const Player& player) {
    ar("name", player.name);
    ar("hp", player.hp);
    ar("inventory", player.inventory);
}

template<class Archive>
void load(Archive& ar, Player& player) {
    ar("name", player.name);
    ar("hp", player.hp);
    ar("inventory", player.inventory);
    player.hpFraction = player.hp / 100.0f;
}

int main() {
    Item potion;
    potion.name = "potion";
    potion.count = 3;

    Player player;
    player.name = "Vex";
    player.hp = 40;
    player.inventory->add(potion);

    auto eng = jai::engine::make();
    std::string text = jai::to_json(*eng, player);

    Player loaded = jai::from_json<Player>(*eng, text);
}

Inventory's fields are private, so the non-intrusive forms declare the free function a friend (or make the fields public); JAI_PROPERTY declarations need no friend.

Both Cereal forms write:

{
    "value0": {
        "name": "Vex",
        "hp": 40,
        "inventory": {
            "ptr_wrapper": {
                "id": 2147483649,
                "data": {
                    "items": [
                        {
                            "name": "potion",
                            "count": 3
                        }
                    ],
                    "capacity": 20
                }
            }
        }
    }
}

The JaiScript member functions and free functions write:

{
  "name": "Vex",
  "hp": 40,
  "inventory": {
    "$id": 1,
    "$val": {
      "items": [
        {
          "name": "potion",
          "count": 3
        }
      ],
      "capacity": 20
    }
  }
}

The JAI_PROPERTY form writes the same record with each level in name order:

{
  "hp": 40,
  "inventory": {
    "$id": 1,
    "$val": {
      "capacity": 20,
      "items": [
        {
          "count": 3,
          "name": "potion"
        }
      ]
    }
  },
  "name": "Vex"
}

A second shared_ptr to the inventory in the same save writes {"$id": 1} and loads as the same object. A class with a CEREAL_CLASS_VERSION gate ports without it (Versions).

A polymorphic hierarchy​

#include <cereal/types/polymorphic.hpp>
#include <cereal/types/base_class.hpp>

struct Effect {
    virtual ~Effect() = default;
    double remaining = 0;

    template<class Archive>
    void serialize(Archive& ar) { ar(CEREAL_NVP(remaining)); }
};

struct Burn : Effect {
    int damagePerTick = 1;

    template<class Archive>
    void serialize(Archive& ar) { ar(cereal::base_class<Effect>(this), CEREAL_NVP(damagePerTick)); }
};

CEREAL_REGISTER_TYPE(Burn)
CEREAL_REGISTER_POLYMORPHIC_RELATION(Effect, Burn)

#include <jaiscript/core/registrar.hpp>

struct Burn : Effect {
    int damagePerTick = 1;

    template<class Archive>
    void serialize(Archive& ar) { Effect::serialize(ar); ar(JAI_NVP(damagePerTick)); }
};

static jai::registrar<Burn, void, jai::bases<Effect>> burnRegistrar("Burn");

Effect changes only its NVP. The registrar, a static object, gives the class the name the archives write as $type and records the edge to Effect, which a load checks before it hands a Burn back through an Effect pointer. An element of a std::vector<std::shared_ptr<Effect>> is written as {"$id": N, "$type": "Burn", "$val": {"remaining": 3.5, "damagePerTick": 2}}. The base's fields land in the derived class's record, where Cereal nests them in a record of their own, so keep field names distinct across a hierarchy. The same registrar binds the class into an engine through jai::bind_registrar (Serialization).

Root objects and files​

jai::to_json(*eng, value) writes one root value. Cereal code that writes several (ar(settings, world)) gathers them into one struct or opens the root record by hand:

jai::serialization::json_archive_writer out(2, eng.get());     // indent, engine
out.begin_object("", 1);
out("settings", settings);
out("world", world);
out.end_object();
std::string text = out.str();

jai::serialization::json_archive_reader in(text, eng.get());
std::string typeName;
std::uint32_t version = 0;
in.begin_object(typeName, version);
in("settings", settings);
in("world", world);
in.end_object();

The binary pair takes the same calls: binary_archive_writer out(eng.get()) with out.data(), and binary_archive_reader in(bytes, eng.get()).

The property route: no serialize functions​

A class that derives jai::property_owner and declares its fields with JAI_PROPERTY needs no function: the archives save and load every property by name, the same declaration makes the field reflected and script-visible (dynamic_binder), and the class carries its own polymorphic identity. It suits types whose functions only list their fields.

#include <jaiscript/properties/macros.hpp>
#include <jaiscript/properties/property_serialization.hpp>

class Effect : public jai::property_owner<Effect> {
public:
    JAI_PROPERTY((double), remaining, 0.0);
};

class Burn : public jai::property_owner<Burn, Effect> {
public:
    JAI_PROPERTY((int), damagePerTick, 1);
};

class Player : public jai::property_owner<Player> {
public:
    JAI_PROPERTY((std::string), name, "hero");
    JAI_PROPERTY((int), hp, 100);
    JAI_PROPERTY((std::shared_ptr<Inventory>), inventory, std::make_shared<Inventory>());
    JAI_PROPERTY((std::vector<std::shared_ptr<Effect>>), effects);
    JAI_TRANSIENT_PROPERTY((float), hpFraction, 1.0f);
};

Player player;
std::string json = jai::to_json(*eng, player);
Player loaded = jai::from_json<Player>(*eng, json);

• A load reads the properties the record and the class share. A property the record lacks keeps its initializer, and one the class no longer declares is skipped, so adding or removing a JAI_PROPERTY needs no migration code.
• JAI_NAMED_PROPERTY((type), "saved_name", member) keeps a key stable through a rename.
• property_owner<Burn, Effect> records Burn's name and its edge to Effect, so no registrar is needed. The name is the bare class name; JAI_PROPERTY_OWNER_AS(Burn, "Burn") pins it.
• A property_owner copies and moves as its members do, so it is held by value, in a container or through a pointer. By value it is written as its static type; a derived object behind a base keeps its $type through a std::shared_ptr, as the effects do.
• Members that are not properties are not saved. A class that needs more keeps a save / load pair, which takes precedence, and calls save_owned_properties(ar) and load_owned_properties(ar) in it for every level's properties (property_mgr.save(ar) covers only the declaring level). A property reads as hp.get() where an expression needs the plain value.

Versions in a self-describing format​

Cereal's binary archives are positional: a reader asks for what the writer wrote, in order, and CEREAL_CLASS_VERSION tells it which layout it holds. Both JaiScript formats name every property in every record (the binary one through a table of names and sizes ahead of the values), so a reader finds a property by name and skips the rest:

• Reordering fields changes nothing, except among fields that hold the same shared object (Pitfalls).
• A removed field is skipped.
• An added field is read under if (ar.has_property("capacity")) { ar(JAI_NVP(capacity)); }, and keeps its initializer for an older record (a property_owner does this for every property).

What is left for a version is a field whose meaning changes: a unit, an encoding, one field split into two. The archives record no class version for a C++ type, so such a type writes one as a property:

template<class Archive>
void save(Archive& ar) const {
    ar("format", std::uint32_t(2));
    ar(JAI_NVP(metersPerSecond));
}

template<class Archive>
void load(Archive& ar) {
    std::uint32_t format = 1;
    if (ar.has_property("format")) { ar("format", format); }
    if (format >= 2) {
        ar(JAI_NVP(metersPerSecond));
    } else {
        double kilometersPerHour = 0;
        ar("speed", kilometersPerHour);
        metersPerSecond = kilometersPerHour / 3.6;
    }
}

A Cereal version history does not carry over. Had the worked conversion's inventory gained capacity in its second version, its Cereal function would gate the field:

    template<class Archive>
    void serialize(Archive& ar, std::uint32_t const version) {
        ar(CEREAL_NVP(items));
        if (version >= 2) { ar(CEREAL_NVP(capacity)); }
    }
};
CEREAL_CLASS_VERSION(Inventory, 2)

The gate serves Cereal files written before capacity was. Those files reach JaiScript through the converter, which reads them with Cereal and its version branch (below), so the first JaiScript layout holds every current field and the JaiScript function is the ungated one the worked conversion shows. A versioned member function still compiles and is called, but the number it receives is not a record version, so leave it unused on the JaiScript side.

A migration path​

1. Add JaiScript beside Cereal. Link the library and make one engine the save code can reach. Nothing changes on disk.
2. Make the leaf types dual (below), starting with types that hold only primitives, strings and containers of those. Cereal's output does not change.
3. Move up through the owners: a type is ready when every type it holds is.
4. Register the polymorphic hierarchies with a registrar or a property_owner base beside each CEREAL_REGISTER_TYPE, which the converter still needs.
5. Prove each type round-trips through both formats: jai::from_json<T>(*eng, jai::to_json(*eng, value)) and jai::from_binary<T>(*eng, jai::to_binary(*eng, value)), compared with the original. The two formats run different code, and a field one drops can hide behind the other.
6. Switch the writers. New files go out through JaiScript under a new name (an extension, a folder); the loader falls back to Cereal for the old name.
7. Convert the existing data once (below).
8. Remove Cereal when no old file remains: its branches, registrations, includes and friend lines. Types whose functions only list fields can then take the property route.

Both libraries in one function​

jai::serialization::jai_archive<Archive> is true for the JaiScript archives and false for every Cereal archive, so an if constexpr on it keeps each library's spelling on its own side, and the branch not taken is never instantiated:

struct Burn : Effect {
    int damagePerTick = 1;

    template<class Archive>
    void serialize(Archive& ar) {
        if constexpr (jai::serialization::jai_archive<Archive>) {
            Effect::serialize(ar);
            ar(JAI_NVP(damagePerTick));
        } else {
            ar(cereal::base_class<Effect>(this), CEREAL_NVP(damagePerTick));
        }
    }
};

CEREAL_REGISTER_TYPE(Burn)
CEREAL_REGISTER_POLYMORPHIC_RELATION(Effect, Burn)
static jai::registrar<Burn, void, jai::bases<Effect>> burnRegistrar("Burn");

• Where the two bodies share nothing, write two overloads: one constrained with requires jai::serialization::jai_archive<Archive>, the other with requires jai::serialization::not_jai_archive<Archive>.
• A Cereal function with a version parameter keeps it. The JaiScript archives call the versioned member form too, so the version branches stay inside the Cereal side.
• A type with load_and_construct keeps one per library. They take different construct types, so they overload without constraints.
• Code that already holds a cereal::NameValuePair<T> hands a JaiScript archive its two halves: ar(pair.name, pair.value).

Converting existing files​

Existing data crosses once, through a program that links both libraries and the dual types. Cereal loads each file exactly as the old code wrote it (the same archive type, the same root shape, its registrations compiled in), and JaiScript saves it:

#include <cereal/archives/binary.hpp>
#include <jaiscript/serialization/convenience.hpp>

void convertSave(jai::engine& eng, const std::filesystem::path& from, const std::filesystem::path& to) {
    SaveGame save;
    {
        std::ifstream is(from, std::ios::binary);
        cereal::BinaryInputArchive in(is);
        in(save);
    }
    std::string text = jai::to_json(eng, save);

    SaveGame check = jai::from_json<SaveGame>(eng, text);
    if (jai::to_json(eng, check) != text) {
        throw std::runtime_error("lossy conversion: " + from.string());
    }
    std::ofstream(to) << text;
}

The check loads the new text back and saves it again, so a field the JaiScript load misses shows up as a difference. (A type holding an unordered_map or unordered_set can list the same entries in another order after a reload; compare those by content.) Content that ships with the program converts in one batch.

Files you cannot convert in advance​

Saves on players' machines convert on first load: ship releases that read the old name through Cereal, write the new name through JaiScript, and read the new name from then on. Tell the formats apart by name, never by trying one reader and catching its error (first pitfall).

The JaiScript formats belong to the library's 0.x surface: data saved by one 0.x release is not promised to load in the next (Stability). Keep the converter and the original files until the format your data depends on has settled.

From Boost.Serialization​

Everything above applies to a Boost codebase. Boost's member functions keep their signatures, because serialize(Archive&, const unsigned int) is the versioned member form the JaiScript archives call. What changes is inside them: Boost streams fields with &, << and >>, and the JaiScript archives take named fields through their call operator.

During the transition one function serves both libraries, as it does for Cereal (above): jai_archive<Archive> is false for every Boost archive, so the else branch keeps ar << name << hp; or ar >> name >> hp;, the class carries both friend lines, and a save/load pair keeps its BOOST_SERIALIZATION_SPLIT_MEMBER() until Boost leaves the build.

A Boost text or binary archive is positional and reads only through the code and class versions that wrote it, so the converter loads each old file with Boost and saves it with JaiScript (Converting existing files).

Pitfalls​

• Reading an old file with the new loader. A property_owner load reads the properties it recognizes and keeps the initializers of the rest, so a Cereal file handed to it can load as an object of defaults without an error. Keep the formats under different names.
• Keys from JAI_NVP. It takes its argument's text as the key: JAI_NVP(m_hp) saves "m_hp" and JAI_NVP(this->hp) saves "this->hp". ar("hp", m_hp) spells the key you want.
• A field the record lacks. ar("name", v) on a missing property throws jai::serialization_error (Property 'name' not found during deserialization). A field a Cereal function wrote conditionally is read under ar.has_property("name").
• A polymorphic unique_ptr. A std::unique_ptr<Effect> holding a Burn saves the Effect fields and loads an Effect, with no error. A polymorphic owner is a std::shared_ptr.
• An unregistered derived class. Saving one through a base pointer throws Cannot serialize unregistered polymorphic type '...' through pointer-to-base '...'; loading a $type the program does not know throws Unknown polymorphic type: Name, and a $type naming a registered class that does not derive from the requested base is rejected.
• The name is in the file. An automatic $type name follows the C++ class name, so renaming the class breaks old saves. Pin a name that must outlive a rename (the registrar's argument, JAI_PROPERTY_OWNER_AS) before the first save.
• A registration a binary never sees. A property_owner registers its name when the program constructs or binds the type somewhere; a tool that only loads a type registers it with a jai::registrar. A registrar in a static library's object file that nothing references is dropped by the linker, as Cereal's registrations are.
• A versioned free function. A free serialize(Archive&, T&, std::uint32_t) is not found: the type has no route, and handing it to an archive is a compile error that lists the routes.
• A forgotten friend jai::access;. A compile error that names the function, as in Cereal, wherever an archive is handed the type. A property_owner level with the function and no friend is that error too. A JAI_PROPERTY whose type forgot it compiles (its mode is a run-time value) and its save throws jai::serialization_error (dispatch).
• Base and derived fields share a record. A derived field named like a base field writes the same key twice. (In a property_owner chain the derived level wins, as C++ name hiding does.)
• A property saved twice. A hand-written save that calls save_owned_properties(ar) and also writes a JAI_PROPERTY field by name writes that key twice.
• Shared objects read out of order. A shared_ptr writes its object at the first reference the save reaches (a weak_ptr that comes first writes it as {"$ref": N, "$val": {...}}) and {"$id": N} at every later one, and the load resolves ids in the order load reads its fields. When the save writes inventory then equipped, both holding one object, a load that reads equipped first throws jai::serialization_error (Dangling shared_ptr reference: {"$id": N} names an object this archive never defined (...)); a weak_ptr read before the field that carries its object loads empty. Read such fields in the order the save writes them. A property_owner's automatic save and load walk its properties in one order (each level by name, base levels first), so this concerns a hand-written pair.
• One graph per call. A shared_ptr saved by two separate to_json calls loads as two objects, as with two Cereal archives.
• Exception types. catch (const cereal::Exception&) becomes catch (const jai::serialization_error&); std::exception covers both while both libraries run.
• Enums are numbers. Both libraries write the underlying integer, so an enumerator inserted mid-list renumbers every value after it in old files. Give enumerators explicit values, or save a name.