NEAR Lake Framework
NEAR Lake Framework is an ecosystem of library companions that read data from the NEAR Data Lake. They allow you to build your own indexer by simply subscribing to the stream of blocks that is being constantly pushed to the NEAR Lake
Example of a simple indexer built on top of NEAR Lake Framework
You can create your own logic to process the NEAR Protocol data in the programming languages of your choice:
Data Lake
The NEAR Lake is a set of AWS S3 buckets which are constantly receiving new blocks from a running indexer maintained by Aurora
Events such as FT Events and NFT Events are constantly being written as JSON files in two AWS S3 buckets:
near-lake-data-testnet(eu-central-1region)near-lake-data-mainnet(eu-central-1region)
Both buckets are set up the way the requester pays for the access. Anyone can read from these buckets by connecting to them with their own AWS credentials to be charged by Amazon.
Latency
Indexers based on the Lake Framework inherit the latency characteristics of a NEAR Indexer plus an extra 0.1-2.1s latency of dumping to and reading from AWS S3.
Most of the latency is there due to the finalization delay and both Indexer Framework and Lake Framework add quite a minimum overhead.
Cost
Indexers based on NEAR Lake consume 100-500MB of RAM depending on the size of the preloading queue, it does not require any storage, and it can potentially run even on Raspberry PI.
Getting the blockchain data from S3 will cost around $30,16 per month as NEAR Lake configured S3 buckets in such a way that the reader is paying the costs.
AWS S3 Cost Breakdown
Assuming NEAR Protocol produces 1 block every 600ms, on a full day the network can create up to 144000 blocks (86400s / 600ms per block).
According to the Amazon S3 prices list requests are charged for $0.005 per 1000 requests and get is charged for $0.0004 per 1000 requests.
Calculations (assuming we are following the tip of the network all the time):
144000 blocks per day * 10 requests for each block / 1000 requests * $0.0004 per 1k requests = $0.576 * 30 days = $17.20
Note: 10 requests for each block means we have 9 shards (1 file for common block data and 9 separate files for each shard)
And a number of list requests we need to perform for 30 days:
144000 blocks per day / 1000 requests * $0.005 per 1k list requests = $0.72 * 30 days = $21.60
$17,20 + $21,60 = $30,16
Comparison with NEAR Indexer Framework
NEAR Lake Framework is reading data from AWS S3, while the NEAR Indexer is running a full node and reading data from the blockchain directly in real time.
| Feature | Indexer Framework | Lake Framework |
|---|---|---|
| Allows to follow the blocks and transactions in the NEAR Protocol | Yes | Yes (but only mainnet and testnet networks) |
| Decentralized | Yes | No (Pagoda Inc dumps the blocks to AWS S3) |
| Reaction time (end-to-end) | minimum 3.8s (estimated average 5-7s) | minimum 3.9s (estimated average 6-8s) |
| Reaction time (framework overhead only) | 0.1s | 0.2-2.2s |
| Estimated cost of infrastructure | $500+/mo | $20/mo |
| Ease of maintenance | Advanced (need to follow every nearcore upgrade, and sync state) | Easy (deploy once and forget) |
| How long will it take to start? | days (on mainnet/testnet) | seconds |
| Ease of local development | Advanced (localnet is a good option, but testing on testnet/mainnet is too heavy) | Easy (see tutorials) |
| Programming languages that a custom indexer can be implemented with | Rust only | Any (currently, helper packages are released in Python, JavaScript, and Rust) |
Examples & Tutorials
near-examples/near-lake-raw-printer: simple example of a data printer built on top of NEAR Lake Frameworknear-examples/near-lake-accounts-watcher: source code for a video tutorial on how to use the NEAR Lake Frameworknear-examples/indexer-tx-watcher-example-lakeindexer example that watches for transaction for specified accounts/contracts build on top of NEAR Lake Framework
See Tutorials page