On March 7, Annie Swafford posted an interesting critique of the transformation method implemented in Syuzhet.  Her basic argument is that setting the low-pass filter too low may result in misleading ringing artifacts.[1]  This post takes up the issue of ringing artifacts more directly and explains how Annie’s clever method of neutralizing values actually demonstrates just how effective the Syuzhet tool is in doing what it was designed to do!   But lest we begin chasing any red herring, let me be very clear about the objectives of the software.

1. The tool is meant to reveal the simple (and latent) shape of stories, not the complex shape of stories, not the perfect shape of stories, not the absolute shape of stories, just the simple foundational shapes.[2]  This was the challenge that Vonnegut put forth when he said “There is no reason why the simple shape of stories cannot be fed into computers.”
2. The tool uses sentiment, as detected by four possible methods, as a proxy for “plot.”  This is in keeping with Vonnegut’s conception of “plot” as a movement between what he called “good fortune” and “ill fortune.”  The gamble Syuzhet makes is that the sentiment detection methods are both “good enough” and also may serve as a satisfying proxy for the “good” and “ill” fortune Vonnegut describes in his essay and lecture.
3. Despite some complex mathematics, there is an interpretive dimension to this work. I suppose this is why folks call it “digital humanities” instead of physics. Syuzhet was designed to estimate and smooth the emotional highs and lows of a narrative; it was not designed to provide a perfect mapping of emotional valence. I don’t think such perfect mapping is computationally possible; if you want/need that kind of experience, then you need to read the book (some of ’em are even worth it).  I’m interested in detecting/revealing the simple shape of stories by approximating the fundamental highs and lows of emotional valence. I believe that this is what Vonnegut had in mind.
4. Finally, when examining the shapes produced by graphing the Syuzhet values, we must remember what Vonnegut said: “This is an exercise in relativity, really. It is the shape of the curves that matters and not their origins.”  When Vonnegut speaks of the shapes, he speaks of them as “simple” shapes.

In her critique of the software, Annie expresses concern over the potential for ringing artifacts when using a Fourier transformation and a very low, low-pass filter.  She introduces an innovative method for detecting this possible ringing.  To demonstrate the efficacy of her method, she “neutralizes” one third of the sentiment values in Joyce’s Portrait of the Artist as a Young Man and then retransforms and graphs the new neutralized shape against the original foundation shape of the story.

Annie posits that if the Syuzhet tool is working as she thinks it should, then the last third of the foundational shape should change in reaction to this neutralization.  In Annie’s example, however, no significant change is observed, and she concludes that this must be due to a ringing artifact.  Figure 1 (below) is the evidence she presents on her blog.

Figure 1: last third neutralized

For what it is worth, we do see some minor differences between the blue and the orange lines, but really, these look like the same “Man in Hole” plot shapes.  Ouch, this does look like a bad ringing artifact.  But could there be another explanation?

There may, indeed, be some ringing here, but it’s not nearly so extreme as Figure 1 suggests.  An alternative conclusion is that the similarity we observe in the two lines is due to a similarity between the actual values and the neutralized values.  As it happens, the last third of the novel is already pretty neutral compared to the rest of the novel.  In fact, the mean valence for the entire last third of the novel is -0.05.  So all we have really achieved in this test is to replace a section of relatively neutral valence with another segment of totally neutral valence.

This is not, therefore, a very good book in which to test for the presence of a ringing artifacts using this particular method of neutralization.  What we see here is a case of the right result but the wrong conclusion.  Which is not to say that there is not some ringing present; I’ll get to that in a moment.  But first another experiment.

If, instead of resetting those values to zero, we set them to 3 (making Portrait end on a very happy note indeed), we get a much different shape (blue line in figure 3).  The earlier parts of the novel are now represented as comparatively less negative and the end of the novel is mucho positive.

And, naturally, we can also set those values very negative and produce the graph seen in figure 4.  Oh, poor Stephen.

“But wait, Jockers, you can’t deny that there is still an artificial “hump” there at the end of figure 3 and an artificial trough at the end of figure 4.”   Nope, I won’t deny it, there really can be ringing artifacts.  Let’s see if we can find some that actually matter . . .

First let’s test the beginning of the novel using the Swafford method.  We neutralize the beginning third of the novel and graph it against the original shape (figure 5).  Hmm, again it seems that the foundation shape is pretty close to the original.  Is this a ringing artifact?

Could be, but in this case it is probably just another false ringer.  Guess what, the beginning of Joyce’s novel is also comparatively neutral.  This is why the Swafford method results in something similar when we neutralize the first third of the book.  Do note that the first third is a little bit less neutral than the last third.  This is why we see a slightly larger difference between the blue and orange lines in figure 5 compared to figure 1.

But what about the middle section?

If we set the middle third of the novel to neutral, what we get is a very different shape (and a very different novel)!  Figure 6 uses the Swafford method to remove the central crisis of the novel. This is no longer a “man in hole” story, and the resulting shape is precisely what we would expect.  Make no mistake, that hump of happiness is not a ringing artifact.  That hump in the middle is now the most sustained non-negative moment in the book.  We have replaced hell with limbo (not heaven because these are neutral values), and in comparison to the other parts of the book, limbo looks pretty good!  Keep in mind Vonnegut’s message from #4 above: “This is an exercise in relativity.”  Also keep in mind that there is some scaling going on over the y-axis; in other words, we should not get too hung up on the precise position on the y-axis at the expense of seeing the simple shape.

In the new graph, the deepest trough has now shifted to the early part of the novel, which is now the location of the greatest negative valence in the story (it’s the section where Stephen gets sick and is then beaten by father Dolan). The end of the book now looks relatively darker since we no longer have the depths of hell from the midsection for comparison, but the end third of Portrait is definitely not as negative as the beginning third and this is reflected nicely in figure 6.  (This more positive ending is also evident, by the way, in the original shape–orange line–where the hump in the last third is slightly higher than the early hump.)

So, the Swafford method proves to be a very useful tool for testing and confirming our expectations.  If we remove the most negative section of the novel, then we should see the nadir of the simple shape shift to the next most negative section.  That is precisely what we see.  I have tested this over a series of other novels, and the effect is the same (see figure 9 below, for example).  This is a great method for validating that the tool is working as expected. Thanks Annie Swafford!

“But wait a second Jockers, what about those rascally ringing artifacts you promised.”

Yes, yes, there can indeed be ringing artifacts.  Let’s go get some. . . .

Annie follows her previous analysis with what seems like an even more extreme example.  She neutralizes everything in Joyce’s Portrait except for the middle 20 sentences of the novel.[3] When the resulting graph looks a lot like the original man-in-hole graph, she says, in essence: “Busted! there is your ringing artifact Dr. J!”  Figure 7 is the graphic from her blog.

Busted indeed!  Those positive valence humps, peaking at 25 and 75 on the x-axis are dead ringers for ringers.  We know from constructing the experiment in this manner, that everything from 0 to ~49 and everything from ~51 to 100 on the x-axis is perfectly neutral, and yet the tool, the visualization, is revealing two positive humps before and after the middle section: horrible, happy, phantom humps that do not exist in the story!

But wait. . .

With all smoothing methods some degree of imprecision is to be expected.  Remember what Vonnegut points out: this is “an exercise in relativity.”  Relatively speaking, even the extreme example in figure 7 is, in my opinion, not too bad.  Just imagine a hypothetical protagonist cruising along in a hypothetical novel such as the one Annie has written with her neutral values.  This protagonist is feeling pretty good about all that neutrality; she ain’t feeling great, but she’s better than bad.  Then she hits that negative section . . . as Vonnegut would say, “oh, God damn it.”[4]  But then things get better, or more precisely, things get comparatively better.  So, the blue line is not a great representation of the narrative, but it’s not a bad approximation either.

But look, I understand my colleague’s concern for more precision, and I don’t want it to appear that I’m taking this ringing business too lightly.  Figure 8 (below) was produced using precisely the same data that Annie used in her two-sentence example; everything is neutralized except for those two sentences from the exact middle of the novel.  This time, however,  I have used a low pass filter set at 100.  Voila!  The new shape (blue) is nothing at all like the original (orange), and the new shape also provides the deep level of detail–and lack of ringing–that some users may desire.[5]  Unfortunately, using such a high, low-pass filter does not usually produce easily interpretable graphs such as seen in figure 8.

In this very simple example, turning the low-pass filter up to 100 produces a graph that is very easy to read/interpret.   When we begin looking at real novels, however, a low-pass of 100 does not result in shapes that are very easy to visually interpret, and it becomes necessary to smooth them.  I think that is what visualization is all about, which is to say, simplifying the complex so that we can get the gist.  One way to simplify these emotional trajectories is to use a low, low pass filter.  Given that going low may cause more ringing, you need to decide just how low you can go.  Another option, that I demonstrated in my previous post, is to use a high value for the low pass filter (to avoid potential ringing) and then apply a lowess smoother (or your own favorite smoother) in order to reveal the “simple shape” (see figure 1 of http://www.matthewjockers.net/2015/03/09/is-that-your-syuzhet-ringing/).

In a future post, I’ll explore something I mentioned to Annie in our email exchange (prior to her public critique): an ad hoc method I’ve been working on that seeks to identify an “ideal” number of components for the low pass filter.

FOOTNOTES:

[1] Annie does not actually explain that the low-pass filter is a user controlled parameter or that what she is actually testing is the efficacy of the default value.  Users of the tool are welcome to experiment with different values for the low pass filter as I have done here: Is that your Syuzhet Ringing.

[2] I’ve been calling these simple shapes “emotional trajectories” and “plot.” Plot is potentially controversial here, so if folks would like to argue that point, I’m sympathetic.  For the first year of this research, I never used the word “plot,” choosing instead “emotional trajectory” or “simple shape,” which is Vonnegut’s term.  I realize plot is a loaded and nuanced word, but “emotional trajectory” and “simple shape” are just not really part of our nomenclature, so plot is my default term.

[3] There is a small discrepancy between Annie’s blog and her code.  Correction: Annie writes about and includes a graph showing the middle “20” sentences, but then provides code for retaining both the middle 2 and the middle 20 sentences.  Either way the point is the same.

[4] The two negative valence sentences from the middle of Portrait are as follows: “Nay, things which are good in themselves become evil in hell. Company, elsewhere a source of comfort to the afflicted, will be there a continual torment: knowledge, so much longed for as the chief good of the intellect, will there be hated worse than ignorance: light, so much coveted by all creatures from the lord of creation down to the humblest plant in the forest, will be loathed intensely.”

[5]  Annie has written that “Syuzhet computes foundation shapes by discarding all but the lowest terms of the Fourier transform.” That is a rather misleading comment. The low-pass-filter is set to 3 by default, but it is a user tunable parameter.  I explained my reasons for choosing 3 as the default in my email exchange with Annie prior to her critique.   It is unclear to me why Annie does not to mention my explanation, so here it is from our email exchange:

“. . . The short and perhaps unsatisfying answer is that I selected 3 based on a good deal of trial and error and several attempts to employ some standard filters that seek to identify a cutoff / threshold by examining the frequencies (ideal, butterworth, and several others that I don’t remember any more).  The trouble with these, and why I selected 3 as the default, is that once you go higher than 3 the resulting plots get rather more complicated, and the goal, of course, is to do the opposite, which is to say that I seek to reduce the plot to a simple base form (along the lines of what Vonnegut is suggesting).  Three isn’t magic, but it does seem to work well at rooting out the foundational shape of the story.  Does it miss some of the subtitles, yep, but again, that is the point, in part.  The longer answer is that is that this is something I’m still experimenting with.  I have one idea that I’m working with now…”