ToDo:
Different program: Batch test all of the scenes, follow all of the conditional paths, etc., to make sure that things are as we would expect. This would go a long ways to making sure that the scenes are coded properly.
First, I redefined the problem slightly. Rather than following all conditional paths, I just wanted to run every response of every scene. (Conditionals depend not only on dice rolls, but also on things like having a feud, or having run the same scene before and having made a certain decision.) This turned out to be very useful to catch script functions that weren’t actually implemented yet. It’s much faster running the tests than going through a scene manually. Plus being automated, any issue was going to stay visible until fixed, instead of possibly falling through the cracks.
Except that’s not strictly true, due to what I call “lucky dice.” There were scenes that would sometimes work and sometimes fail. Usually this was due to having a specific person on the clan ring. One answer to this would be to make dice non-random. They could generate the same sequence of numbers every time through. I chose not to do this for a couple reasons. First, it would be even further from the "all of the conditional paths” idea, since it would be the same path every time. In other words, it focused on making the tests work, and not making the game work. Second, due to the nature of what was being tested, it didn’t really guarantee repeatability. The dice rolls would be the same, but their contexts would be different as we added scenes or revised them. (In other words, the 57th roll might always be a 16, but that roll might determine a clan attack in one run, and random text in the next after we fixed a bug.)
Another factor is that running every scene is unnatural. Most scenes have some sort of condition (there is no point in having someone ask for a treasure if you have no treasure), or are explicitly designed to follow another scene. The unit tests didn’t test for this sort of continuity, but forcing a scene to run when its precondition wasn’t met often caused problems that would never show up in a real game.
For a while, I just lived with the false positives (a scene failing if there no friendly neighbors, or after your advisors died in an apocalypse) that came and went. We’re not using build automation, so it was annoying but not really gumming up the works.
It seemed like a more serious limitation that the test would run every response, but couldn’t handle other interactivity, like picking a clan to petition, the number of warriors to hunt down a monster with, or how large a gift to offer. OSL is designed to pause execution and wait for user response, then resume running the script. This is all implicit — the interpreter doesn’t literally sit there waiting (which would be a great way to run down your batteries), it sends a message and quits. It took me a while to figure out how to hook into this asynchronous process in a way that would work with unit tests (which are essentially synchronous, running a script until it is complete). Once I did, the tests suddenly got a lot more thorough. (A typical response might ask you for a clan, a diplomat, and a gift, all before resolving the negotiation. So a typical response had only been very partially tested before.) And this meant that there were more false positives, since there was more code being run, but depending on a particular story context.
I tried to ensure a proper context (when testing, hard-code a friendly neighbor), but false positives would still come and go depending on chance. I finally came up with an approach that allowed me to get broader coverage by having free-rolling dice, but also avoid testing scenes in a way that was almost guaranteed to fail.
The goal of this test was to exercise each scene as fully as practical. Randomness helped give broader test coverage within a scene (at least over multiple test runs). But randomness cascading through the test run was causing problems. Ergo, I needed to eliminate the cascade. The breakthrough here was to save the game at the beginning of the test, and restore before running each response. This minimized randomness in a scene’s context, so it would never be run in a known impossible situation.
The final step was to capture the clean setup in a script, which is run at the start of the unit test (before the game is saved).
So once again, we can run the test when coding up a scene, and the errors it reports represent real problems.
We’re still not going through every part of a script — the player decision maker doesn’t handle combat. (It also gets stuck in certain negotiations by refusing to up its offer.) And while saving and loading is fairly fast, doing so for multiple responses for over a hundred scenes does add up. But exercising every script still takes only 32 seconds on my development machine, and running all the unit tests is a single command, so it’s not really a big deal.
Everything this test catches would eventually be found by QA, but only much later. It’s always better to shorten the time between creating a bug and finding it, so things are still fresh in the coder’s mind. And test automation means QA can spend more time on bugs that only a human can catch. So it’s now part of our scene coding process.
