Blockchain indexes are a critical process that makes distributed application (DAPP) data accessible and available. Despite the success of graphs, the challenges remain with data portability and fragmentation, particularly across a variety of blockchains.
Blockchain index: Enables DAPP data
The rise of distributed applications (DAPPS) has lifted a new paradigm of interaction with technology. However, building these applications presents the fundamental challenge of accessing the vast, ever-growing data stored on the blockchain. Unlike traditional databases that can be easily searched in languages such as structured query languages (SQL), blockchain data is stored in a sequence of linear and temporal expressions of blocks.
To find a single transaction or user wallet history, millions of blocks need to be painstakingly scanned. This is a slow, inefficient, and even too expensive process for most Dapps. This is where blockchain indexes become essential. Essentially, indexing is the process of retrieving all raw, unstructured data from the blockchain and organizing it into a searchable query database. This is accomplished by a multi-step process often referred to as extraction, transformation, and loading (ETL).
The indexer begins the process by connecting to a blockchain node to monitor new activity in real time. It then extracts important data points such as transaction details, smart contract events, and state changes. This is followed by a critical step in which the raw extracted data is decoded and converted into a more meaningful, structured form. Finally, this transformed data is loaded into a structured database, allowing immediate access for Dapps and developers to use with queries.
The dangers of concentrated indexes
Without this ETL, valuable blockchain data is practically inaccessible and DAPP cannot be used effectively. Indexing is a fundamental component of the Web3 ecosystem, but not all solutions are created equally. As pointed out by Nick Hansen, team lead at the Graph Foundation, many Dapps are currently relying on centralized vendors or building their own in-house index infrastructure.
This can provide a quick solution, but poses serious risks, such as providing centralized vendors with the power to change terms of service or shut down services. Reliance on a single point of failure also undermines the central tenet of decentralization in which Web3 is built. Furthermore, centralized solutions are said to lack the transparent, unreliable mechanisms needed to ensure the accuracy of the data. This can lead to many issues with DAPP.
A distributed approach to graphs
To overcome this problem, a graph called “Google on Blockchain” offers a decentralized alternative to this problem. It is a protocol that coordinates a global network of independent participants to provide fast, reliable, verifiable blockchain data.
The network operates through a system of incentives and punishments equipped with native token GRT. Participants in the system essentially include indexers, which are node operators who bet GRT and compete to provide data queries. In return for providing the service, they earn both a query fee and an indexing reward.
Curators are other participants who show what application programming interfaces (APIs) or subgraphs index because they need to be indexed because they are the most valuable.
To ensure data integrity, the protocol includes a robust thrashing mechanism. If the indexer is found to be malicious or provide incorrect data, you can “reduce” or confiscate some of the stained GRT. Therefore, this creates a strong economic incentive for the indexer to be honest and perform their obligations accurately. This variance model eliminates the “lagpur” risk associated with central providers, unauthorized and resistant to sudden changes in censorship or service conditions.
Meanwhile, some observers argue that the central issue of data portability is not just moving data between blockchains. It is rooted in the way Dapps are built fundamentally. They often force developers to use different index solutions, with today’s ecosystems being fragmented and often within different chains or even within a single application.
Hansen admits that while graph indexing solutions set the bar high, there are still lingering issues that challenge developers. He adds:
Developers need data workflows tailored to the nuance and complexity of blockchain data while adhering to decentralized standards.
However, the Graph Foundation team lead noted that the Graph team is addressing user challenges and that current product development focuses on solving many of these issues.
Looking ahead, Hansen hopes that graphs will continue to evolve with the growing needs of Web3 and the industry. He added that the infrastructure being built is “designed to support builders, users and institutions alike without compromising transparency or control.”