Why is it so difficult to add nlohmann's json to Urho3D?

[ghost]

I have a scene that I'm trying to serialize to XML. The scene contains Components that hold onto references to other components. I've made them hold onto Component IDs (unsigned int), thus I need to serialize a set. I figured the easiest way would be to use JSON, and not being able to find any way to extend the type conversions on Variant without editing Variant.h/Variant.cpp I set out to edit them.

I have these simple converter files that allow nlohmann's json to work with EASTL's string and vector. If I put them into my own new project, then they work great. If I put them into the existing Urho3D project, then they fail with a slew of template errors. The errors are the same as if the converter functions weren't found, so that's probably what's happening.

JSON.h

#pragma once
#include <nlohmann/json.hpp>
#include <EASTL/string.h>
#include <EASTL/vector.h>

using nlohmann::json;
namespace nlohmann {
    void to_json(json& j, const ea::string& s);
    void from_json(const json& j, ea::string& s);
    void to_json(json& j, const ea::vector<ea::string>& s);
    void from_json(const json& j, ea::vector<ea::string>& s);
}

JSON.cpp

#include "JSON.h"

namespace nlohmann {

using namespace ea;

void to_json(json& j, const string& s) {
    j = json{ s.c_str() };
}

void from_json(const json& j, string& s) {
    std::string t;
    j.get_to(t);
    s = t.c_str();
}

void to_json(json& j, const vector<string>& s) {
    std::vector<std::string> t;
    for (auto& ss : s)
        t.push_back(ss.c_str());
    j = t;
}

void from_json(const json& j, vector<string>& s) {
    s.clear();
    std::vector<std::string> t;
    j.get_to(t);
    for (auto& tt : t)
        s.push_back(tt.c_str());
}

}

Within the Urho3D project, the fact that I have to_json and from_json in the nlohmann namespace means that the library can't find any other converters, so something as simple as nlohman::json j = std::string(); fails because it can't convert to std::string, but it suddenly can again if I change the namespace name. This didn't happen in my project - my converters being in the nlohmann namespace was perfectly fine and had no effect on the compiler's ability to find other converters.

It makes no sense to me that it works in one project but not another, unless there's something set in the project settings for Urho3D that is making this fail? I can't being to imagine what that is.

[eugeneko]

Can you provide full minimal example that fails to compile?

[rokups]

But why would you do this? I suppose you are familiar with this library.. But what you are doing will inevitable be slow because each little thing you move to json would be copied. I do not have high hopes for general performance of this library either. Urho3D already has JSONValue + JSONFile and we have rapidjson in third-party dependencies if wrapped classes do not cut it.

[ghost]

The point of the files here were to simply make the nlohmann json library automatically convert ea::string to std::string and really only amounts to something cosmetic. The reason why I'm using this json library is because I'm (relatively) familiar with it unlike Urho3D/rbfx, and I usually pluck it into all of my projects that have file I/O. The reason why I would do this at all is because I was having issues figuring out how to serialize my Component attributes (ea::vector<unsigned int> for instance), and expanding the serializing capabilities of Variant seemed to be a solution. The reason why this is acceptable to me is because I plan to profile and refactor/rewrite critical things later on, especially as I learn more about what tools already exist within the library to accomplish these tasks.

Ultimately I just didn't use the automatic type conversion functions and simply bypassed the errors that brought me here. Presumably the issue was namespaces and the fault was my own, especially if this isn't an issue anyone has brought up before.

[rokups]

It is hard to tell what the issue was exactly since you did not post any errors. No matter. Right course of action would be figuring out how to achieve what you need using facilities of the engine. What would be the point in using engine if you go your own way and implement your own solutions each time you have a problem? You would essentially be writing another engine in parallel, wasting lots of your time and missing out on likely more convenient ways of doing things.

---

So on to your problem. You need to serialize ea::vector<unsigned int>, Variant does not support such type. However Variant supports unsigned int and ea::vector<Varaiant>. If you decare your attribute to use VAR_VARIANTVECTOR you could store integer array that way. Would be OK for small arrays, but if your data is big - we need to look further. You may use VAR_BUFFER (ea::vector<unsigned char>). Not ideal mapping, but integer array is merely 4x larger char array. If all else fails you could implement a subclass of Serializable, implement LoadJSON()/SaveJSON() (and maybe others if you need them) methods and declare your attribute as VAR_CUSTOM storing Serializable pointer.

[eugeneko]

JSON serialization is quite handy and readable in the output, unline binary buffers.

What would be the point in using engine if you go your own way and implement your own solutions each time you have a problem?

I guess that nlohmann json is just easier to integrate. You copy string there, you read stuff. JSONFile/JSONValue cannot do that.

[rokups]

JSONFile::FromString() can do that. I do agree using another object to deserialize json string is awkward. Still beats copying entire dataset or even side-stepping built-in serialization facilities.

[ghost]

It is hard to tell what the issue was exactly since you did not post any errors.

Yeah, in hindsight (and after sleep) I would have not posted this question the way I had. I agree with your views on using the engine facilities.

Ultimately I'm trying to make a game where I load into a large world hosted on a server, and I can interact with NPCs, transfer items between inventories, have interconnected networks of 'computers' in-game, and the ability to facilitate a large amount of user content. I'm only just starting to poke around, but so far I just love this engine. I wish I knew exactly what to do read to accomplish this, but trial and error has always been a big part of online learning for me.

I like modifying Variant to accept more types regardless of the internals (it should just be whatever is fastest) as it makes both attribute serialization and using event data cleaner.

[rokups]

Err, i did not suggest to modify Variant, but merely use it to serialize custom data ;)

Regarding your goal - networking is a weak spot of urho in general. I would not depend too much on it. I also have no advice on how to approach this problem. We have been talking about networking upgrades, but you know how it is. Time is scare, lots of stuff to do, so that has been only talk and no code so far.