I’d like to highlight a conversation that took place on crypto twitter this week regarding EOS.

It’s no secret to anyone deep in the EOS world that the validator infrastructure requirements are quite different for EOS than they are for a network like Bitcoin. Given the sheer performance of EOS, there’s far more data being generated daily than there is on other networks. That requires a different approach to storing, indexing, and serving all of that data.

The conversation started with this tweet by Corey Miller of BlockTower Capital. I believe that Corey’s intention was to call attention to a situation he thought unsustainable, so I don’t fault him for bringing it up. But he got some information wrong, especially about the network security implications of this issue. As with most things in this space, there’s a lot more nuance to this issue than can fit in a series of tweets. But crypto twitter ran with it, and most people just assumed it to be fact. Here I’ll attempt to offer a more in-depth explanation of the situation at hand.

When it comes to full nodes in EOS, we need to distinguish between a consensus node and an API history node. A consensus node requires a block log, which is a log of all of the blocks and actions that have occurred on EOS, ever. The size of the EOS block log is currently about 195GB. The block log contains all the data that has ever passed through the EOS blockchain, but it is stored in a compact format that is not easy or efficient to access. A full history node (sometimes referred to as an API history node), on the other hand, contains the same data as a block log, but in an easier and more efficient to read format. This is done by loading the data and receipts for actions, blocks, and transactions into a database like MongoDB or Postgres . A single data point can be stored as often 4-5 times so that it can be queried in different forms. This is the dataset that Corey was referring to. As Kedar Iyer from LibertyBlock points out, full history data is “computed data to give people convenient API access.” This currently comes out to about 4TB and is much more expensive and technically complex to maintain than a simple block log. That said, these full history nodes are not actually necessary for securing the network, as Corey implied. Rather, they’re provided as a service to wallets, block explorers, and other dApps and tools that need to conveniently access historical state. Most dApps don’t require access to a full queryable history of every action on EOS, so they will maintain just the subset of data they need in an easily queryable format.

That’s not to say that there aren’t issues with full history API nodes on EOS. When Block.one first published the EOSIO software, they included a native history plugin. However, this plugin had some major issues, and Block.one deprecated it in August 2018 without a plan in place to replace it. Although the plugin was imperfect, it was the closest thing to a “standard” that existed, so a number of BPs (including Greymass and EOS Sweden) continued to run full history nodes based on that setup. On the other hand, a number of BPs decided to create alternatives to the B1 plugin. These include solutions by EOS Canada, Attic Lab, EOS Tribe, EOS Rio, CryptoLions and others. These are all full history nodes, but they function in different ways.

The main issue here is that these nodes aren’t on the same standard. They store, index, and serve the blockchain data differently, so it’s not very simple for dApps to plug into a number of different ones for redundancy. What the network needs right now is a new standard– one that is open-source, addresses the issues from the original deprecated plugin, and can built upon by anyone else. The good news? EOS Rio is working on just that, and it should go live within a few weeks. Rio’s solution will not only create a new open-source standard, but it removes much of the inefficiency from the original history plugin, resulting in much lower storage requirements for full history nodes.

In the meantime, there are a number of other options that are promising. Greymass has released a light history spec, which allows BPs to spin up partial history nodes that cover more than 90% of requests in most cases, at less than 25% of the cost of running a full history node. The EOS Canada team has built dfuse, which is an incredibly powerful blockchain data API that many dApps are using. And then there’s LiquidApps from Liquid EOS, a new dApp layer that can provide additional monetary incentive for BPs to run full history nodes.

In other words, this situation is not quite as dire as it seems, and there are very real solutions on the horizon. If you’re interested in learning more about them, we recorded a podcast with the Greymass team that we’ll release on Monday. We’re also going to record a podcast with the EOS Rio team to discuss their new standard as soon as they are ready to go live. Make sure you check out those episodes!

Myles Snider, CEO