My Obsession With Predicting The World

Introduction

Recent conversations have left me thinking about my motivations, and the driving force behind the hours I put into my work at the expense of nearly everything else.

I don’t think I’ve ever written about this publicly before. For as long as I’ve had ambition, I’ve been driven by a desire to build a world model so adept at predicting the world that it can drive out inefficiencies in all markets, public or private.

If you’ve ever watched Westworld, yes, I’m talking about something like Rehoboam: a superintelligence that could map human behavior and predict the future. Interestingly enough, Rehoboam started out as a model for predicting stock prices (really).

It’s a large part of why being a “quant” has interested me. The markets are the ultimate testing field for your ability to predict the future across various timescales, and it’s an immensely scalable endeavor with a positive flywheel: the better you are at predictions, the more resources you get to become better at predictions, and so on.

I Have A Dream

One of my fears is that I’ll die before reaching my grand ambition of predicting all the useful things society needs predictions about, and be forever immortalized as just another human obsessed with making money for the sake of making money.

My reason for being is that I believe it’s possible to build an entity so adept at predictions that it can minimize inefficiencies across all human systems. That we can build a world model that guides humanity toward decisions better than heuristics or random guesses.

This means we can fund the startup ideas with the greatest predicted impact, issue scholarships to children with nascent genius. For example: which president should be elected to increase the probability of global peace and prosperity?

I think humans are in a race against time. As a species, we have finite time to expand beyond the limits of our current prison. When facing finite resources, we need to maximize efficiency to improve our odds of reaching the next bound before we run out and perish. If we never figure out how to escape Earth before resources are consumed, or before conditions make leaving impossible, we’ll inevitably die as a species in a prison of our own making.

Oh me! Oh life! of the questions of these recurring,

Of the endless trains of the faithless, of cities fill’d with the foolish,

Of myself forever reproaching myself, (for who more foolish than I, and who more faithless?)

Of eyes that vainly crave the light, of the objects mean, of the struggle ever renew’d,

Of the poor results of all, of the plodding and sordid crowds I see around me,

Of the empty and useless years of the rest, with the rest me intertwined,

The question, O me! so sad, recurring—What good amid these, O me, O life?

Answer.

That you are here—that life exists and identity,

That the powerful play goes on, and you may contribute a verse.

Walt Whitman, O Me! O Life!

I hope my verse will be an entity that makes humanity more efficient, that nudges our probability of reaching the next bound up a little.

Predictions

I’ve been obsessed with predictions since I was a teen. I remember being endlessly fascinated by how accurately physics could predict the future. Simple models had profound implications. They showed us that prediction was possible, that we could know where an object would end up if we knew enough about its initial conditions and the system it was operating in.

We model systems as random when our predictions of them are no better than guessing, i.e., when we don’t have enough information about the system to generate a prediction better than chance.

For example, if we were asked to predict (assign probabilities to) whether a coin lands heads or tails, our initial instinct would be to assign equal probability to both. That would be our best prediction with no further information about the coin flip. But if we knew the initial face of the coin, the height of the flip, the force being applied, the dimensions and mass of the coin, and the material it would land on, we could construct a model that predicts the landing face with significantly greater accuracy than random.

While I believe there’s a limit to determinism, I also deeply believe that when it comes to human systems, we haven’t even scratched the surface of what’s possible. The data we’ve needed has traditionally been either highly sparse or highly unstructured, and the models and compute we needed that were capable of finding generalizable patterns in such high-dimensional data were previously unavailable.

The principle that more relevant data yields better predictions given a sufficiently powerful model holds true even as systems grow more complex. So an entity that endeavored to predict the world would need to accumulate as much data as possible. We’ve learned in recent years, with the advent of LLMs, that with sufficient scale and complexity in data and model, a form of intelligence emerges.

Machine Learning And Higher Dimensions

Data is inert on its own. You need a model to transform data into actual predictions. Further, which model is best for producing useful predictions is a non-trivial question, thanks to the no-free-lunch theorem. Fortunately, advances in machine learning give us a rather satisfactory solution: letting the data dictate the form of the model, rather than having a theory dictate it.

Really large and powerful machine learning models can pick up high-dimensional patterns that tend to generalize, because the underlying behavior driving them is reliably repeating.

This means there’s a world where we might use the Substacks an entrepreneur follows or the podcasts he’s subscribed to as features to predict his probability of success, long before he’s even incorporated his company. The patterns we’re interested in are more about his desires, drives, and views of the world, and less about how he spends 20 minutes of his time each day. We can glimpse those beliefs from tangential data.

To push this further, we might learn about a person’s potential from their childhood experiences, their belief systems, attributes and character, long before they have become an adult. Wouldn’t it be nice to allocate capital to genius in a part of a world where it would be bound to be wasted?

Conclusion

So we know we need vast amounts of data, and vast amounts of computing power to train enormous models to learn high-dimensional patterns from that data.

That translates to a very expensive endeavor but the good news is this: data and compute is getting cheaper every passing day, and unstructured data is becoming less and less of an issue with the parsing powers of LLMs. We are also getting scarily good at training enormous god models that can learn patterns in dimensions so large its unfathomable to the human mind.

Yet, to build something of such scale, we would need a business that could sequentially finance its scale and ambitions. Starting with predictions in liquid markets seems like the obvious choice, before moving to progressively harder markets and systems. You can see how one might be motivated to pursue the path I have pursued if you think along these lines...

So, if I die while in the midst of running something that resembles a simple cash grab, know that my shame is immeasurable, and that it was never supposed to be the end destination!

Source

I just know someone is going to say I retconned this for openforage, but I had actually written this in 2023, and while AI means that screenshots now mean very little...