Variant Investment Partners: The Dilemma and Breakthrough of Open Source AI, Why Cryptography is the Last Piece of the Puzzle?

Author: Daniel Barabander

Compiled by: Deep Tide TechFlow

Summary

• The development of foundational AI is currently dominated by a few tech companies, characterized by a closed nature and lack of competition.

• While open-source software development is a potential solution, foundational AI cannot operate like traditional open-source projects (e.g., Linux) due to a "resource problem": open-source contributors not only need to invest time but also bear computational and data costs that exceed individual capabilities.

• Cryptographic technology has the potential to solve this resource problem by incentivizing resource providers to participate in foundational open-source AI projects.

• Combining open-source AI with cryptographic technology can support the development of larger-scale models and drive more innovation, leading to the creation of more advanced AI systems.

Introduction

According to a survey conducted by the Pew Research Center in 2024, 64% of Americans believe that social media has a more negative than positive impact on the country; 78% say that social media companies have too much power and influence in politics; and 83% think that these platforms are likely to intentionally censor political views they disagree with. Discontent with social media has almost become one of the few consensus views in American society.

Looking back over the past 20 years of social media development, this situation seems to have been predetermined. The story is not complicated: a few large tech companies captured users' attention and, more importantly, controlled user data. Although there was initial hope for data openness, these companies quickly changed their strategies, leveraging data to establish unbreakable network effects and closing off access to outsiders. This ultimately led to the current situation: fewer than 10 large tech companies dominate the social media industry, creating an "oligopoly." Given that the status quo is extremely beneficial to them, these companies have little incentive to change. This model is closed and lacks competition.

Today, the trajectory of AI technology development seems to be replaying this scene, but the implications are even more profound. A few tech companies have built foundational AI models by controlling GPU and data resources, while shutting off access to these models. For new entrants without billions of dollars in funding, developing a competitive model is nearly impossible. The computational costs of training a foundational model alone can reach billions of dollars, and those social media companies that benefited from the last wave of technology are using their control over proprietary user data to develop models that are difficult for competitors to reach. We are repeating the mistakes of social media, heading towards a closed and competitive-free AI world. If this trend continues, a few tech companies will have unrestricted control over access to information and opportunities.

Open-source AI and the "Resource Problem"

If we do not want to see a closed AI world, what are our options? The obvious answer is to develop foundational models as open-source software projects. Historically, countless open-source projects have successfully built the foundational software we rely on daily. For example, the success of Linux proves that even core software like operating systems can be developed through open-source means. So why can't LLMs (large language models) do the same?

However, the unique constraints faced by foundational AI models set them apart from traditional software, significantly undermining their viability as conventional open-source projects. Specifically, foundational AI models require enormous computational and data resources, far beyond individual capabilities. Unlike traditional open-source projects that rely solely on people's donated time, open-source AI also requires individuals to contribute computational power and data resources, which is known as the "resource problem."

Taking Meta's LLaMa model as an example helps us better understand this resource problem. Unlike competitors like OpenAI and Google, Meta did not hide the model behind a paid API but instead publicly provided the weights of LLaMa for anyone to use for free ( with certain restrictions ). These weights contain the knowledge the model learned during Meta's training process and are essential for running the model. With these weights, users can fine-tune the model or use the model's output as input for new models.

While Meta's release of LLaMa's weights is commendable, it cannot be considered a true open-source software project. Meta controls the training process behind the scenes, relying on its own computational resources, data, and decisions, unilaterally deciding when to make the model available to the public. Meta has not invited independent researchers or developers to participate in community collaboration because the resources required to train or retrain the model far exceed what an ordinary individual can provide. These resources include tens of thousands of high-performance GPUs, data centers to store these GPUs, complex cooling facilities, and trillions of tokens (text data units required for model training). As noted in the 2024 AI Index Report from Stanford University , "the sharp rise in training costs effectively excludes universities, which have traditionally been strongholds of AI research, from developing top foundational models." For example, Sam Altman mentioned that the cost of training GPT-4 was as high as $100 million , and this does not even include capital expenditures for hardware facilities. Additionally, Meta's capital expenditures increased by $2.1 billion in the second quarter of 2024 compared to the same period in 2023 , primarily for servers, data centers, and network infrastructure related to AI model training. Therefore, while community contributors to LLaMa may have the technical ability to improve the model architecture, they lack sufficient resources to implement those improvements.

In summary, unlike traditional open-source software projects, open-source AI projects require contributors not only to invest time but also to bear high computational and data costs. Relying solely on goodwill and volunteer spirit to incentivize enough resource providers is unrealistic. They need further incentive mechanisms. For example, the open-source large language model BLOOM , which has 176 billion parameters, brought together the efforts of 1,000 volunteer researchers from over 250 institutions across more than 70 countries. While BLOOM's success is admirable (and I fully support it), it took a year to coordinate a single training session and relied on a €3 million grant from a French research institution (not including capital expenditures for the supercomputer used to train the model). Relying on a new round of funding to coordinate and iterate on BLOOM's process is too cumbersome to compete with the development speed of large tech labs. More than two years have passed since BLOOM's release, and there have been no reports of the team developing any follow-up models.

To make open-source AI possible, we need to find a way to incentivize resource providers to contribute their computational power and data resources, rather than letting open-source contributors bear these costs themselves.

Why Cryptographic Technology Can Solve the "Resource Problem" of Foundational Open-source AI

The core breakthrough of cryptographic technology lies in making high-resource-cost open-source software projects feasible through an "ownership" mechanism. It addresses the resource problem of open-source AI by incentivizing potential resource providers to participate in the network rather than requiring open-source contributors to bear these resource costs upfront.

Bitcoin is a great example. As one of the earliest cryptographic projects, Bitcoin is a completely open-source software project, with its code publicly available from the start. However, the code itself is not the key to Bitcoin. Simply downloading and running the Bitcoin node software to create a blockchain locally has no practical significance. The true value of the software is realized only when the computational effort to mine blocks far exceeds that of any single contributor: maintaining a decentralized, uncontrolled ledger. Similar to foundational open-source AI, Bitcoin is also an open-source project that requires resources beyond individual capabilities. While the reasons for the computational resource demands differ—Bitcoin requires computational resources to ensure the network's immutability, while foundational AI needs computational resources to optimize and iterate models—the commonality lies in the need for resources that exceed individual capabilities.

The "secret" that allows Bitcoin and any other cryptographic network to incentivize participants to provide resources for open-source software projects is the provision of network ownership through tokens. As Jesse outlined in the founding principles he wrote for Variant in 2020 , ownership provides a strong incentive for resource providers to contribute resources in exchange for potential returns within the network. This mechanism is similar to how startups address early funding shortages through "sweat equity"—by compensating early employees (e.g., founders) primarily in the form of company ownership, startups can attract labor they otherwise could not afford. Cryptographic technology extends the concept of "sweat equity" from focusing on time contributors to resource providers. Thus, Variant focuses on investing in projects that leverage ownership mechanisms to build network effects, such as Uniswap, Morpho, and World.

If we want open-source AI to become a reality, then the ownership mechanism enabled by cryptographic technology is the key solution to the resource problem. This mechanism allows researchers to freely contribute their model design ideas to open-source projects because the computational and data resources required to realize these ideas will be borne by resource providers, who will receive partial ownership of the project in return, rather than requiring researchers to bear the high upfront costs themselves. In open-source AI, ownership can take various forms, but one of the most anticipated is ownership of the model itself, which is also the solution proposed by Pluralis .

The approach proposed by Pluralis is known as Protocol Models . In this model, computational resource providers can train specific open-source models by contributing computational power and thus gain partial ownership of the future inference revenue of that model. Since this ownership is tied to a specific model and its value is based on the inference revenue of the model, computational resource providers are incentivized to choose the optimal model for training and are unlikely to fabricate training data (because providing useless training would directly reduce the expected value of future inference revenue). However, a key question is: how does Pluralis ensure the security of ownership if the training process requires sending the model's weights to the computational providers? The answer lies in using "model parallelism" techniques to shard the model and distribute it to different workers. An important characteristic of neural networks is that even if a contributor only understands a tiny portion of the model weights, they can still participate in training, ensuring that the complete set of weights cannot be extracted. Moreover, since many different models will be trained simultaneously on the Pluralis platform, trainers will face a multitude of different weight sets, making it extremely difficult to reconstruct the complete model.

The core idea of Protocol Models is that these models can be trained and used, but cannot be fully extracted from the protocol (unless the computational power used exceeds the resources required to train the model from scratch). This mechanism addresses a common criticism of open-source AI, which is that closed AI competitors may steal the labor of open-source projects.

Why Cryptographic Technology + Open-source = Better AI

At the beginning of the article, I highlighted the ethical and regulatory issues of closed AI by analyzing the control that large tech companies have over AI. However, in an era of pervasive powerlessness, I worry that such arguments may resonate less with most readers. Therefore, I would like to present two reasons, based on practical effects, explaining why cryptographically supported open-source AI can truly lead to better AI.

First, the combination of cryptographic technology and open-source AI can coordinate more resources, thereby driving the development of the next generation of foundational models. Research shows that increases in both computational power and data resources contribute to improved model performance, which is why the scale of foundational models has been continuously expanding. Bitcoin demonstrates the potential of combining open-source software with cryptographic technology in terms of computational power. It has become the largest and most powerful computing network globally, with a scale far exceeding the cloud computing resources owned by large tech companies . The uniqueness of cryptographic technology lies in its ability to transform isolated competition into collaborative competition. By incentivizing resource providers to contribute resources to solve common problems rather than fighting alone and duplicating efforts, cryptographic networks achieve efficient resource utilization. Open-source AI powered by cryptographic technology will be able to leverage global computational and data resources to build models that far surpass those of closed AI. For example, Hyperbolic has already demonstrated the potential of this model. They have created an open market that allows anyone to rent GPUs at a lower cost, thereby fully utilizing distributed computing resources.

Second, the combination of cryptographic technology and open-source AI will accelerate innovation. This is because once the resource problem is solved, machine learning research can return to its highly iterative and innovative open-source nature. Before the emergence of foundational large language models (LLMs), researchers in the field of machine learning typically publicly released their models and their replicable design blueprints. These models often used open-source datasets and had relatively low computational demands, allowing researchers to continuously optimize and innovate based on them. It was this open iterative process that led to numerous breakthroughs in the field of sequence modeling, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and attention mechanisms, ultimately making the Transformer model architecture possible. However, this open research approach changed after the launch of GPT-3. OpenAI demonstrated through the success of GPT-3 and ChatGPT that with sufficient computational resources and data, it is possible to train large language models with language understanding capabilities. This trend has led to a sharp rise in resource thresholds, gradually excluding academia, while large tech companies, in order to maintain competitive advantages, no longer publicly disclose their model architectures. This situation limits our ability to push the frontiers of AI technology.

Open-source AI enabled by cryptographic technology can change this status quo. It can allow researchers to iterate on cutting-edge models again, discovering the "next Transformer." This combination not only addresses the resource problem but also revitalizes the innovative vitality of the machine learning field, paving the way for broader future developments in AI.