From: thepipeline_xyz
In the crypto industry, accurate and reliable information about performance gains is often lacking amidst marketing hype, particularly concerning concepts like the parallel EVM [00:00:34]. The absence of standardized benchmarks makes it difficult to ascertain true performance and can lead to misleading claims [00:44:56].
The Need for Standardization
Unlike other computer-related fields such as graphics cards or CPUs, which have standard benchmarks [00:44:26], the crypto space currently lacks consistent metrics [00:44:54]. This allows projects to make broad claims without accountability, as there’s no standard to hold them to [00:44:58].
For instance, Transaction Per Second (TPS) figures can be easily misunderstood. A client not highly optimized can still achieve 50,000 TPS for simple token transfers [00:45:19]. Therefore, stating “200,000 TPS” without specifying the type of transactions (e.g., Uniswap vs. simple token transfers) provides no meaningful context and can sound deceptively impressive [00:45:45]. It’s akin to comparing someone doing 20 simple math problems a minute to someone doing one complex calculus problem a minute, without specifying the problem difficulty [00:52:07].
Challenges and Pitfalls in Benchmarking
Superficial Optimizations
Some claims about performance, like simply implementing a parallel EVM, might sound impactful but yield little true performance gain on their own [00:03:18]. The actual bottleneck is often state access, which involves reading data from SSDs [00:03:33]. Relying solely on parallel EVM without addressing underlying inefficiencies in the state database is like putting a Lamborghini body on a Toyota Corolla engine – it looks good but doesn’t deliver significant performance [00:08:10].
Cheating Benchmarks with Hardware
A common shortcut to appear performant is to use excessive amounts of RAM [00:50:24]. While RAM is faster, it is two orders of magnitude more expensive than SSDs (e.g., 2 terabytes of RAM costs 200 for a 2TB NVMe SSD) [00:50:32]. This approach does not scale well and undermines decentralization by increasing hardware requirements for participants [00:51:29].
Invalid Assumptions
Assumptions about certain technologies, like access lists inherently improving parallel execution, are not always correct [00:31:46]. Rigorous measurement and experimentation are necessary to validate hypotheses, and often, intuitive ideas prove to be inaccurate [00:34:21].
Monad’s Approach to Benchmarks
Monad Labs advocates for and will contribute to establishing industry-wide standardized benchmarks.
- Ethereum History as a Benchmark: Monad uses recent Ethereum history as its internal benchmark to accurately measure performance [00:46:20]. This involves replaying real-world transaction data, which can take hours or even days for a full history [00:48:00].
- Publicly Available Benchmarks: Monad plans to build a publicly available GitHub repository where others can download and replicate their benchmarks for various chains [00:46:24].
- Open and Reproducible: The goal is to provide transparent and reproducible benchmarks, allowing the wider crypto community to contribute, suggest improvements, or verify results [00:46:38]. This includes standardizing hardware specifications (e.g., 32 GB RAM vs. 256 GB RAM) for fair comparison [00:47:15].
Benefits of Standardization
Standardized benchmarks will bring much-needed rigor to the blockchain industry [00:49:00]. They will:
- Increase Accountability: Hold projects to their performance claims and prevent misleading marketing [00:45:00].
- Provide Context: Allow for a clear understanding of what performance numbers actually represent [00:46:07].
- Drive Progress: Enable engineers to make informed, scientific decisions by identifying where optimizations are truly effective across different clients [00:48:16].
- Foster Transparency: Promote a culture of open experimentation and shared knowledge, moving away from intuition-based marketing claims [00:48:56].
Ultimately, establishing robust benchmarks will improve the overall engineering standards of the industry and contribute to the space’s advancement [00:48:43].