Explorations in Lua as a Data File Format

Here I’m not going to bother with an example, as I’m sure there are a million ways to slice it, but the end result is strange at best. You end up needing to create something of a prefab format where you build up tables that can be referenced in some way by another “Command” that uses that data to create an instance. There you also need a mechanism for allowing override data to be specified for that instance (e.g. starting off different game objects in different positions and states). There is the added difficulty here of needing to be able to handle deep hierarchies of objects/components when specifying override data. Although honestly this edge case can be a pain without facilitating overrides at all; it’s just generally creates a lot of headaches.

#2 & 3: Deep Hierarchies and “Prefabs”

Changing everything to treat things as a stream works fairly well, but causes the structure to get far less concise. For example:

  name = "Thing";
  position = "0,3,0";
  material = "MyMaterial";

  health = 20;
  defense = 4;



That’s not horrible, but certainly ventures into “Not exactly awesome” territory.

#4: General Complications with Lua Syntax/Processing

At the end of this you have files with a list of commands in them, but you need to have helper functions for initiating processing, dealing with a stack of created objects, and still quite a few tweaky little hoops to jump through in order to cleanly write back any data.

In conclusion

I’d say that using Lua as an definition file for things like materials, rendering pipelines, effects/shaders, and anything else that doesn’t necessitate hierarchies or a prefab approach is very nice and clean. Similarly writing tools to process those files wouldn’t be too much extra work and you could likely get away with just writing out the format “Manually” without much issue.

For other needs a concise and widely supported format such as JSON is really going to be more straightforward and have far better support in other environments when it comes time to write tools. BitSquid has taken an approach similar to this in their engine and it seems very nice. Their engine is extremely data driven however, so it’s unclear to me how a, um, how to describe my own engine… a less theoretically sound engine might fair with such a format, but I’d venture to say that it’s entirely within the realm of possibility that it’s better than using XML.

Not featured here are my thoughts on Google Protocol Buffers, which I’ve researched but not used yet. In that case the issue becomes the fact that everything is stored in binary blobs, but otherwise it’s a very compelling solution.