An Economic Analysis Of Numerai
Introduction
This article contains an analysis of Numerai’s data scientists and some thoughts on Numerai’s various products.
I am especially interested in the differences in performance between “Numerai Classic” – their flagship product – and “Numerai Signals” – a relatively new product designed to capture alpha orthogonal to Classic.
I draw several conclusions that should be interesting to the Numerai team, Numerai users, as well as career quants in general, because the analysis reveals something about our industry and the problems faced by crowdsourcing platforms.
Data
All of the data were scrapped from Numerai’s public API for both Classic and Signals.
The Landscape
Broad overview
Users
One of the first interesting things to me is how sparse Numerai’s user base (~10,000 users and ~4,000 “active” users) is, and I say this NOT to throw shade at Numerai. You wouldn’t expect to see this in a company trying to get users for the past 10 years or so, but that’s the truth about the industry.
In the X fintwit bubble, it might feel like everyone and their mothers are quants and are going to eat you alive – but whenever we look at data on crowdsourcing platforms, the number of users tells a very different story. This is a VERY niche industry.
I have broad user metrics for most crowdsourcing platforms, and penetrating >10,000 ACTIVE users is hard for everyone. It is a technical field and an UNFRIENDLY learning environment with a STEEP and LONG learning curve that exceeds the patience of most people.
Gamifying and reducing the slope of this learning curve is arguably the most important “operational alpha” of a crowdsourcing platform, but that is going to be out of scope for this article.
Stakes & Fat Tails
The next thing I want you to realize is that looking at averages in any crowdsourcing platform is a moot exercise. The stories are told in the tails.
The median stake is, unsurprisingly, 0, and the top 10% stake is STILL ONLY 10 NMR (~$100), and on the far end you have users staking up to ~$100,000. This “fat-tail” property is essentially the only thing consistent between Numerai Classic and Numerai Signals.
If you look at the stake distributions between Classic and Signals, it is clear that Signals is a significantly “smaller” product by both users AND the amount of NMR staked.
Returns
If you are a Numerai user, or an aspiring one, and you want to figure out how much you are likely to earn, check this out!
This is where you can tell that Classic and Signals are completely different beasts.
Ex-ante Chances Of Making Money
Firstly, the distribution of returns around Classic is much tighter, and the average 52-week return for Classic is actually positive. On the other hand, the distribution of Signals returns is wide with an average around 0.
What does this mean?
It means that if you are a beginner and you are hoping to make money with Signals, you should absolutely be ready for disappointment and a steep learning curve. Your Bayesian prior of making money should be regularized to 0 UNLESS you know what you are doing.
On the other hand, if you are patient and are willing to slog through the initial activation energy to submit a model, and have reached a level of confidence where you are actually staking some NMR, your Bayesian prior of making money on Classic should be significantly non-zero, which is great.
Why?
We will cover how Classic and Signals are different animals in a later section of the article, and why this leads to different statistics that cannot be trivially explained by an “immature” product.
Correlation (User Performance)
The correlation here is the canonical “correlation to returns”.
The returns of NMR and correlation SHOULD tell a somewhat similar story, because IF Numerai has done a good job aligning performance with incentives, then performance (correlation) and incentives (returns) should be roughly aligned.
We Are Playing A Hard Game
The first thing we should contend with is that correlations are “low”. Does this hold across the industry, or is this a Numerai artifact? Sadly, unlike other fields with higher signal-to-noise ratios, correlations of about ~0.01 already belong to the best quantitative researchers – both in Numerai and at brand-name firms.
This should make you EXTREMELY suspicious of papers and course-sellers touting high Sharpe ratios with implied correlations > 0.05. It is effectively impossible unless they have identified an EXTREMELY inefficient spot in the market, and IF they have truly done so, marketing it openly for $200 is NOT the most optimal way to monetize that inefficiency. Hence, you are either reading material written by a charlatan, or you are reading material written by an idiot – neither of which I would recommend. Instead, you should be reading material by SysLS.
Correlation vs Returns
In Classic, there is a very clear relationship between correlation and returns on NMR. This shouldn’t be surprising to anyone actually using Numerai, but for the uninitiated, this is in part because Numerai explicitly rewards correlation in Classic as part of the payout (hence this shows up in returns), but NOT for Signals.
In Signals, the relationship is much looser because, in place of correlation, Numerai rewards Signals with “Alpha”, which for all intents and purposes can be thought of as a target that is orthogonal to Classic’s target.
This means correlation is not a great metric for Signals, and you see this show up in the averages. The average correlation for Signals is actually non-zero, BUT, as you can see above, the average return for Signals is approximately zero.
Hence, I will iterate ONE MORE TIME:
In Numerai Signals, if you just reproduce standard value/momentum/country/sector signals OR are too close to Classic, they will be neutralized away and score near zero, regardless of WHETHER OR NOT they actually have predictive power.
Good Signals models are those where the orthogonal component after neutralization still predicts the neutralized target. You are not trying to predict raw returns; you’re trying to add something Numerai doesn’t already have.
A Mental Model: Classic Vs Signals
A Quantitative Investment Process
In every investment process, you can roughly split it up into the following:
Data: Sourcing, cleaning, processing data for ingestion.
Forecasts: Creating forecasts where corr(forecast_t, returns_t+N) > 0
Portfolio: Combining forecasts to maximize IDIO_IR(forecasts) while minimizing EXPOSURE(forecasts)
Execution: Get the portfolio that maximizes IDIO_IR(forecasts) while minimizing transaction costs
Going into detail for all of the above is out of scope for this article, but for each component, the above is sufficient to make my primary arguments.
For EACH component, there are two kinds of competitive edge that the market is paying you for.
The first kind is a RESEARCH competitive edge, where the market is paying you for having done the research to do each component “well”. This is the part that most people talk about in books, articles, etc. When people say, “I have an edge from learning how to use MODEL A to predict TARGET B”, they are almost always exclusively talking about RESEARCH competitive edge. This is the part where you can “study”, “discover”, or “research” your way into.
The second kind is an OPERATIONAL competitive edge, where the market is paying you for being able to manage the complex operations of maintaining each component of the investment process. People rarely talk about this, and the concept is almost non-existent for retail/hobbyist quants. But there is a very real structural moat from being able to handle petabytes of data, manage and combine millions of forecasts like clockwork, and connect to hundreds of exchanges.
This infrastructural complexity is an operational moat that is not easily replicated by smaller players. They neither have the cultural DNA nor the economics for it.
Classic
You should think of Classic as Numerai is providing users with:
[DATA] The OPERATIONAL competitive edge of sourcing, maintaining, processing, and delivering like clockwork many expensive institutional datasets that are not even within reach of the average investor.
[DATA] The RESEARCH competitive edge of cleaning, transforming, normalizing, and enriching raw data into features that are themselves already predictive of the targets.
[FORECASTS] The OPERATIONAL competitive edge of backtesting, producing, scheduling and submitting your forecasts.
[PORTFOLIO] The OPERATIONAL competitive edge of collecting, storing, managing, and combining thousands of forecasts.
[PORTFOLIO] The RESEARCH competitive edge of doing good portfolio management. For Numerai to attract institutional investors, I can safely assume that they are delivering idiosyncratic returns expected of good hedge funds. They have to create many meta-models that are as orthogonal to one another as possible, whilst ensuring that the firm portfolio respects all factor, risk, and sizing constraints AND is optimal for scale (capacity).
[EXECUTION] The OPERATIONAL and RESEARCH competitive edge of executing well. I won’t elaborate too much on this, but the operational aspects of execution include propagating turnover and churn constraints upwards to the forecast level and communicating, incentivizing, and enforcing these constraints well.
What does this mean?
This means that as a Numerai user, you are essentially being given Numerai’s competitive edge in the DATA, FORECASTS, PORTFOLIO, and EXECUTION components of the investment process. In all fairness, Numerai should ONLY pay for your additive RESEARCH forecasting skills, and the marginal competitive edge you provide by bringing research and operational competitive edge in the FORECASTS component.
This means that IF you believe Numerai has done a good job on the DATA front, then as long as your RESEARCH FORECAST skills are not subtractive, you should be able to extract positive forecasts out of Numerai’s data.
The data above also supports this: the AVERAGE CLASSIC user already has a quite POSITIVE correlation, suggesting that Numerai rents out enough competitive edge in Classic that any non-subtractive modelling skills will result in a non-zero positive correlation.
From Numerai’s perspective, IF they were purely economical and calculative, they should actually remove the “rent component” of their investment process by de-meaning a user’s correlation, because you can argue that any success from predictions by the average user is NOT because of excess modelling skills, but because of the economic rent of Numerai’s competitive edge.
FURTHER (and this is conjecture), Numerai is also more lenient with Classic. Whilst the target implicitly bakes in factor neutralization, I don’t think Classic forecasts are explicitly neutralized to factors. Classic forecasts are also not subjected to the very strict churn and turnover requirements of Signals.
My belief is that Numerai further subsidizes users by handling neutralization to factors and turnover optimization at the portfolio level.
All in all, this makes Classic the quintessential “alpha crowdsourcing” product, where a platform is paying users for the competitive edge they bring to the table in RESEARCH competitive edge in FORECASTS.
Signals
Signals is an entirely different beast, and it is no wonder that users find Signals SIGNIFICANTLY harder. In Signals, Numerai is providing users with:
[EXECUTION] The OPERATIONAL and RESEARCH competitive edge of executing well.
The user is expected to:
Develop and maintain an OPERATIONAL and RESEARCH competitive edge in DATA, because they are expected to source, set up, and maintain the data ETL; clean, process, and enrich the underlying data; and prepare it for ingestion by their forecasts.
DDevelop and maintain a RESEARCH competitive edge in FORECASTS; no different from in Classic. ON TOP OF THAT, they are now responsible for the OPERATIONAL competitive edge to maintain their SIGNALs generation and submission.
Develop and maintain an OPERATIONAL and RESEARCH competitive edge in PORTFOLIO. This is where, in my opinion, Numerai users will STRUGGLE the most. By allowing for arbitrary forecasts across any subset of the universe WHILST needing to be factor neutral AND orthogonal to Classic AND respecting turnover and churn, Numerai has actually passed along a large segment of the responsibility of the PORTFOLIO to its users.
There are immediately two issues with Signals that I foresee making it a very challenging product for Numerai:
Requiring users to maintain an OPERATIONAL competitive edge in DATA and PORTFOLIO is structurally incompatible with hobbyists and part-timers. Classic is able to succeed because Numerai has figured out how to obfuscate as much of the operational complexities as possible, which makes it compatible for part-timers to focus on research.
Consider this: if you handle all the complexities of operations for me, a part-time data scientist, I can work on developing a RESEARCH competitive edge in FORECASTS after work, and start and stop anytime; my skills will compound and eventually get good enough to earn incentives. IF I have to handle the operational complexities of maintaining a 24-hour pipeline, I am no longer able to start and stop whenever I want. This is the primary structural incompatibility of Signals.
The paradigm of needing to do good portfolio management to deliver good Signals forecasts is a non-trivial paradigm for Numerai users to grok, and it was not necessary to succeed in Classic. In Signals, you don’t actually want to submit many small models. You want to submit a portfolio like a good systematic portfolio manager. For obvious competitive reasons, I won’t elaborate beyond this – but if I were to participate in Signals, I would NOT be submitting more than one “model”. Every submission would be my best possible portfolio that is neutralized to factors and Classic and is already turnover-optimized.
How To “Fix” Signals
I’m not SURE Signals is “broken” as a product; I’d need more data and an “inside look”. But two things stand out to me in terms of possible iterations.
Redistribute Economic Rent Of Data From Classic To Signals
If Numerai wanted to capture and incentivize hobbyists and part-timers to actually develop an operational edge in DATA and PORTFOLIO, then they need to pay users accordingly, because the Signals payout does not represent any other components but the RESEARCH competitive edge in FORECASTS. A simple change would be to de-mean Classic returns and pass this mean over to Signals, shared on a stake-weighted basis for positive Signals contributors.
You are essentially saying: “I am collecting my economic rent of providing an OPERATIONAL and RESEARCH edge in data” from Classic, and “I am paying you for the OPERATIONAL and RESEARCH edge in data” to Signals.
Change The Participation Pool Of Signals
There is a very real possibility that the conclusion is that Signals is not compatible with the current (mostly hobbyists and part-timers) user pool of Numerai.
After all, we need to think clearly about why a product like Signals exists in the first place. Numerai, being rational economic agents, must have looked at all possible metas produced by Numerai Classic and come to the conclusion that it does not even remotely span the breadth or depth of all possible portfolios.
The investment process being hierarchical also means that “alpha” flows from data all the way to execution. If you cannot differentiate at the DATA level, it becomes hard to differentiate at the FORECAST level. This MUST be the reason why Numerai structured Signals the way they did.
The raison d’être of Signals is to fill in the gaps that Numerai Classic cannot. Perhaps with very well-designed incentives and a few more tweaks to increase the operational competitive edge of Signals, it can actually get there, but I remain skeptical that this will be done trivially with the current user base.
I think Signals needs to serve a completely different community orthogonal to Classic if it is to succeed as a standalone product and ACTUALLY FULFILL its raison d’être.
It is my STRONG opinion that for Signals to have the highest chance of succeeding, Numerai should change its approach for Signals and actually try to capture the institutional members of the quant community, by rethinking its incentives and program to instead look for industry researchers on the cusp of being PMs and offer them a chance to deliver a portfolio via Signals.
Do not elbow against the largest hedge funds for established PMs; it is a losing proposition because it will result in a race to zero. Instead, create a team to exercise PM selection and PM grooming & management. This team is then responsible for developing a competitive edge in finding and managing PMs to submit Signals. This makes full use of the aligned incentives of Signals to bet on budding PMs on a first-loss basis.
This allows Signals to look for talent that is actually structurally aligned with the requirements to maintain an OPERATIONAL competitive edge in the investment process AND gives Signals a real fighting shot at truly creating a diversified portfolio that doesn’t stem from the same edge as Classic.
I could write another 10,000-word article on thinking about hiring PMs that are a good fit for Signals, but my fingers are sore and I have spent too much time on this article, so...
Parting Words
This was a blast to write and already reflects most of my existing stance on crowdsourcing platforms and hiring PMs.
Numerai Classic and Signals are, in practice, two fundamentally different products that demand very different kinds of edge from their users. Classic effectively rents out Numerai’s data, infrastructure, and portfolio management, so even modest but non-subtractive research skill can translate into positive expected returns.
Signals, by contrast, pushes a large chunk of the operational and portfolio burden onto users, which makes it structurally hostile to hobbyists and part-timers and better suited to genuinely institutional-grade processes.
If Numerai wants Signals to fulfill its intended role as an orthogonal alpha source to Classic, it will likely need to both reallocate economic rent and rethink its target user base toward aspiring PMs with real operational capabilities, rather than treating Signals as just “Classic but with your own data.”