POST

Explorations in Lua as a Data File Format

Fairly recently I came across http://prideout.net/blog/?p=1 covering using Lua as an effects file format. It made me realize some things that sort of made me feel like an ass; chief among them that JSON as a format is actually just valid Javascript that results in a hierarchical structure of arrays/tables/whatever. So yeah, apparently I’m sort of slow on the uptake as I’m now aware that that’s the whole point of JSON.

TLDR You can omit the surrounding parenthesis to a function in Lua when you are passing in a table as a parameter. So:

Texture
{
  name = "thing_color";
  path = "textures/thing_color.png";
};

Material
{
  name = "MyMaterial";
  texture = "thing_color";
};

MyGameObject
{
  name = "Thing";
  position = "0,3,0";
  material = "MyMaterial";
};

When processed by Lua would attempt to call the functions Texture, Material, MyGameObject from the global scope. This allows you to define these functions, then load any of these data files and they will automatically result in corresponding function calls instead of having to do any parsing whatsoever.

I took an experimental pass at making my personal project use this approach for levels and objects and really the results were mixed. Specifically because:

  1. The structure doesn’t lend itself particularly well to being able to write back to a file (so tools are more difficult to write)

  2. When handling hierarchical structures, such as a game object with components on it, the evaluation order of the function parameters is backwards from what you actually need (i.e. parent created first)

  3. This leads to the issue of needing to treat your structure as a stream of sorts, which then requires “Closing” tags, generally making everything less convenient

  4. General Lua syntax and structure complications

#1: Difficulties Writing Back to the Structure and File(s)

To get around this, and also to be able to store data which can be reused for multiple object instances, storing everything as tables and then instantiating objects passing in those tables works, but, again, starts to invalidate the very reason why this approach was compelling (which is convenience and the leveraging of an existing layer of technology in your game… assuming that you have Lua in there at all and that’s why you’re even trying this in the first place).

Here I’m not going to bother with an example, as I’m sure there are a million ways to slice it, but the end result is strange at best. You end up needing to create something of a prefab format where you build up tables that can be referenced in some way by another “Command” that uses that data to create an instance. There you also need a mechanism for allowing override data to be specified for that instance (e.g. starting off different game objects in different positions and states). There is the added difficulty here of needing to be able to handle deep hierarchies of objects/components when specifying override data. Although honestly this edge case can be a pain without facilitating overrides at all; it’s just generally creates a lot of headaches.

#2 & 3: Deep Hierarchies and “Prefabs”

Changing everything to treat things as a stream works fairly well, but causes the structure to get far less concise. For example:

MyGameObject;
{
  name = "Thing";
  position = "0,3,0";
  material = "MyMaterial";
};

SomeComponent;
{
  health = 20;
  defense = 4;
};

EndComponent;

EndMyGameObject;

That’s not horrible, but certainly ventures into “Not exactly awesome” territory.

#4: General Complications with Lua Syntax/Processing

At the end of this you have files with a list of commands in them, but you need to have helper functions for initiating processing, dealing with a stack of created objects, and still quite a few tweaky little hoops to jump through in order to cleanly write back any data.

In conclusion

I’d say that using Lua as an definition file for things like materials, rendering pipelines, effects/shaders, and anything else that doesn’t necessitate hierarchies or a prefab approach is very nice and clean. Similarly writing tools to process those files wouldn’t be too much extra work and you could likely get away with just writing out the format “Manually” without much issue.

For other needs a concise and widely supported format such as JSON is really going to be more straightforward and have far better support in other environments when it comes time to write tools. BitSquid has taken an approach similar to this in their engine and it seems very nice. Their engine is extremely data driven however, so it’s unclear to me how a, um, how to describe my own engine… a less theoretically sound engine might fair with such a format, but I’d venture to say that it’s entirely within the realm of possibility that it’s better than using XML.

Not featured here are my thoughts on Google Protocol Buffers, which I’ve researched but not used yet. In that case the issue becomes the fact that everything is stored in binary blobs, but otherwise it’s a very compelling solution.