Sui Indexing Framework Enhances Onchain Data Access

The Sui Indexing Framework offers customizable access to Sui’s onchain data through a powerful data ingestion framework, according to The Sui Blog. This framework allows the collection of both raw onchain data and derived datasets by various software, whether operating onchain or offchain.

Leveraging this framework enables developers to create customizable data feeds that can build software and products responsive to onchain events. The Sui Indexing Framework addresses the limitations of traditional blockchain data structures, which are often not optimized for random data access across their entire history. Customizable data feeds built with this framework empower developers to harness onchain data more effectively for real-time analytics and responsive applications.

The Power of Onchain Data Feeds

For instance, a musician could use NFTs to distribute music to fans. By creating a non-transferrable NFT collection, each NFT could grant automatic access to an audio file stored in an offchain database upon minting. A custom indexer created with the Sui Indexing Framework could track the minting transactions associated with these specific NFTs, enabling a separate offchain service to perform actions like transferring audio files, triggered by events monitored through the custom indexer.

The framework is particularly useful for those seeking a leaner Full node setup. Without an indexing solution, Full nodes typically retain the history of every transaction. A custom indexer can be created using the Sui Indexing Framework, which feeds checkpoint data to be stored separately from the Full node, allowing for more efficient infrastructure setups as Full nodes can be aggressively pruned.

Additionally, the Sui Indexing Framework is crucial for the development of onchain data dashboards. While a data analytics platform requires many elements, this framework is foundational for data ingestion that these apps rely on.

How It Works

Data ingestion with the Sui Indexing Framework begins with subscribing to the checkpoint stream from Sui to receive the most recent data. The simplest approach is to subscribe to a remote store of checkpoint data, such as those provided by Mysten Labs:

• Testnet - https://checkpoints.testnet.sui.io
• Mainnet - https://checkpoints.mainnet.sui.io

A worker function must be created to process the checkpoint data. The main application then calls the worker function whenever it detects an event in the remote store.

use async_trait::async_trait;
use sui_data_ingestion_core::{setup_single_workflow, Worker};
use sui_types::full_checkpoint_content::CheckpointData;

struct CustomWorker;

#[async_trait]
impl Worker for CustomWorker {
    async fn process_checkpoint(&self, checkpoint: CheckpointData) -> Result<()> {
        println!("processing checkpoint {}", checkpoint.checkpoint_summary.sequence_number);
        // custom processing logic
        ...
        Ok(())
    }
}

#[tokio::main]
async fn main() -> Result<()> {
    let (executor, term_sender) = setup_single_workflow(
        CustomWorker,
        "https://checkpoints.mainnet.sui.io".to_string(),
        0, /* initial checkpoint number */
        5, /* concurrency */
        None, /* extra reader options */
    ).await?;
    executor.await?;
    Ok(())
}

For those operating their own Full node, they can create their own checkpoint stream by adding the following checkpoint-executor-config information to the Full node configuration file:

checkpoint-executor-config:
  data-ingestion-dir: <path to a local directory>

Once configured, the Full node dumps checkpoint data into a local directory, and the indexer daemon listens for checkpoint events and processes the data as new checkpoints arrive. The checkpoint data returned is a CheckpointData struct, familiar to current apps. The indexer can then process the data in the same manner as hosted subscriptions.

The Sui Indexing Framework supports both pull-based and push-based processing methods, offering developers the flexibility to choose between straightforward implementation or reduced latency. This versatility is crucial for applications prioritizing real-time data access and responsiveness.

Dive Deeper

Whether creating apps that respond to real-time blockchain events or managing general data and infrastructure, the Sui Indexing Framework offers the flexibility and reliability needed. For detailed implementation guidance, explore the Sui Custom Indexer documentation. To see the framework in action, explore the specialized indexing pipelines used by Mysten Labs, SuiNS, and the Sui Bridge.