Markets that outsmart themselves

I believe financial markets outsmart themselves. I'm not talking about the Efficient Market Hypothesis, but something more meta. This idea probably has a proper name, but I don't know what it is.

Here's the gist of it: There is money to be made predicting financial markets, so people are motivated to predict them. However, once someone acts on a prediction, they alter the market, causing it to become an additional degree more complex. The end result is that the market, which had previously fit a model, no longer fits any model.

Being the geek that I am, of course I have always wanted to simulate this behavior. Tonight I finally sat down and did it.

In my simulation, I have a pool of market actors, each of which has its own algorithm for predicting a market with a single stock. Every day the actors use their models to predict the future price of the stock, and buy and sell shares according to their predictions.

Learning behavior is simulated in two steps. First, random, sporadic mutations occur in the pool of models. If the resulting model is unfit to survive in the market, the other models will naturally eat its lunch. In addition, a certain number of actors are randomly selected to copy the most successful algorithm, so winning algorithms prosper.

Here is a sample of the resulting stock price history. In this case, I had 200 market actors, and the graph shows days 100-200 of a simulation.

Interestingly, while the stock price remains generally in a consistent range, there are periods of irrational exuberance, including a five-day period in which the stock price leaps to 1000, then returns to normal.

In the lead-up to the period of irrational exuberance, the winning algorithms grew progressively simpler, going from six-term polynomials down to a single-term polynomial. If this were to occur in the real world, it would be something like a single wealthy individual getting a crazy idea, and everyone following suit until the market crashes.

Note that I don't assume the stock has any kind of real value here. My market actors are trying to outguess each other. Each actor in the market is trying to predict
how the aggregate of actors in the market will predict
how the aggregate of actors in the market will predict
how the aggregate of actors in the market will predict
...
how the market will behave.

Structure of political events

I've off and on been trying to find ways to create computational models of political events in order to predict their outcomes. I've started with an intuitive sense that events evolve in a structured way, and if we can predict the probability of the smaller steps in that evolution, we can predict the total outcome of the event.

For example, consider the question of whether Afghanistan will approve the bilateral security agreement offered by the United States. The Loya Jirga (assembly of elders) has just approved the agreement, and urged Karzai to sign it before the end of the year. In order for Karzai to sign it, the following internal events must occur:

E1: The Afghan National Assembly must approve the BSA. This will no doubt be influenced by the fact that the Loya Jirga has already approved it, since the Afghan constitution declares the Loya Jirga to be the highest expression of the will of the Afghan people. However, there is some question of whether the Loya Jirga was properly assembled according to the constitution, and some feeling that Karzai convened the Loya Jirga because he thought he could influence its decision more than he could influence the National Assembly directly. Let's say the odds of this are something like 0.9.

E2: Karzai must choose to sign the BSA before the end of the year. Currently he is calling for it to be signed next year.

Intuitively, the outcome of the total event should be something like:

P_E1 * P_E2

Where P_E is the probability that event E will occur.

I do not know specifically why Karzai is holding out on signing the BSA this year. In the absence of knowledge, it is tempting to assign a 0.5 probability to P_E2, giving final odds of 0.9 * 0.5 = 0.45. However, my recent experience is showing that there is another way to look at these questions.

In the most general description, event E2 is an irreversible state change. Once Karzai chooses to approve the BSA, he cannot (in a practical sense) reverse the approval and return to his original state. He could make his decision on any day, so P_E2 should really decay towards zero according to a formula like the following:

P_E2 = 1 - (1 - p_E2)^d

Where p_E2 represents the odds on any given day that Karzai will change his mind, and d is the number of days remaining before the end of the year.

If delaying the BSA is good in itself for Karzai, then he will never change his mind, so we could say that p_E2 is zero and so likewise is P_E2. That might be the case if Karzai believes he could be charged with treason, assassinated, or otherwise subject to persecution/prosecution for signing the BSA.

On the other hand, Karzai may be threatening to delay signature of the BSA in order to extract some concession from the United States. In that case, for him to fail to sign the BSA by the end of the year signifies the failure of his gambit. The odds of his signature in that case are calculated very differently, so we should probably think about this as two separate events:

E2a: Karzai signs the BSA even though he believes he could be persecuted as a result.

E2b: Karzai signs the BSA after receiving a concession from the United States.

In order to calculate E2 from these two values, we need to decide the odds that either or both are true. Let's say there is are 0.1 odds that Karzai fears serious persecution and 0.9 odds that he is trying to wring out a concession:

P_E2 = 0.1 P_E2a + 0.9 P_E2b

As the end of the year approaches, the value of Karzai's signature drops, so the value of what he expects in return should decrease.  Meanwhile, the value of what the US offers should gradually increase until it meets the value of what Karzai has to offer. If both players know each other well, then they have already calculated that they will reach an agreement before the end of the year, and the only question is whether they are wrong. In that case, P_E2b should depend on the odds that the two players have correctly estimated each other's thresholds for negotiation.

P_E2b = 1 - (1 - C_E2b)^d

Where C_E2b represents the odds that the two parties have correctly assessed each other's positions and can reach an agreement on any given day.

So, if we estimate 0.9 odds that the National Assembly will concur with the decision of the Loya Jirga, and 0.8 odds that the two players have correctly estimated each other's positions, then the total likelihood of a timely signature is:

P_Etotal = P_E1 * (0.1 P_E2a + 0.9 P_E2b)

P_Etotal = 0.9 * (0.1 * 0.0 + 0.9 * (1 - (1 - 0.8))^d)

Under that model, the odds of Karzai signing the BSA hover at about 0.81 till the last three days of December, when they suddenly plummet to zero.

One of the things that bedevils me here, though, is the unknown intermediate steps. If I have time, I think I would like to see what kind of behavior emerges if I simulate situations where there are thousands of dependent steps and thousands of possible routes to a particular outcome. Do complex networks of interrelated events conform to a different set of rules en masse?

(Note added 12/17/2013: The Karzai example here does not work if, for example Karzai is engaged in negotiations with Iran. The reason is that the settled state of negotiations with the US gives Karzai the opportunity to develop other options and choose between them. So maybe we should expect that a less-than-satisfactory settled negotiation will usually stall while other options are developed.)

The genome of a narrative

One of my hobbies is political forecasting. It's an interesting pursuit with many fascinating challenges, and one of them is the challenge of getting good information from unreliable sources.

The internet can be seen as a vast set of assertions of varying validity, produced and consumed by the "mindspace" of the networked world. Some forecasters try to get good information by aggregating many different assertions, on the principle that the process of aggregation will reduce the influence of errors. That is a good way to reduce noise, but it doesn't help when there are widespread misconceptions.

People don't like to change their minds, so very often the first idea they take up is the one they will stick with in the long run. That means that ideas which travel quickly can occupy territory in the mindspace ahead of ideas that travel more slowly. An idea travels quickly if it is easily passed on, so all it needs is to be simple, easy to explain, and make sense. Slower, more complex ideas lose the race.

I have also found that ideas carrying a strong emotional payload can effectively defend their territory in the mindspace against competitors. For example, Stars and Stripes recently published an article about a false story alleging that Obama wants to emasculate the US Marines by asking them to wear female covers. In this case, the falsehood triggers stronger emotions than the truth, so it gains and holds ground.

The end result is that the viability of an idea on the internet is not necessarily correlated with its truth, and a false idea may easily replicate enough to influence the results of aggregation.

To address this, I have adopted an approach that is similar to the narrative analysis used in the study of Folkloristics.  I try to identify the main narratives relating to a subject and trace the genealogy of each back to its original source (if possible). Then I attempt to explain why the original source released the narrative into the wild.

(The original version of this post had an example of a narrative here, but I took it out because it made the post too long.  Now I wish I had it back, because it was interesting.)

I am interested in the question of how (or whether) computational linguistics and other tools can be used to trace the genealogy of narratives on the internet. Among other things, I imagine this could lead to identifying large currents of thought--channels by which ideas spread from a small number of sources to a large audience.