Hive Nodes

Applications that interface directly with the Hive blockchain will need to connect to a Hive node. Developers may choose to use one of the public API nodes that are available, or run their own instance of a node.

Public Nodes

All nodes listed use HTTPS (https://). If you require WebSockets for your solutions, please consider setting up your own hived node or proxy WebSockets to HTTPS using lineman.

Private Nodes

The simplest way to get started is by deploying a pre-built dockerized container.

System Requirements

We assume the base system will be running at least Ubuntu 22.04 (jammy). Everything will likely work with later versions of Ubuntu. IMPORTANT UPDATE: experiments have shown 20% better API performance when running U23.10, so this latter version is recommended over Ubuntu 22 as a hosting OS.

For a mainnet API node, we recommend:

• At least 32GB of memory. If you have 64GB, it will improve the time it takes to sync from scratch, but it should make less of a difference if you’re starting from a mostly-synced HAF node (i.e., restoring a recent ZFS snapshot)
• 4TB of NVMe storage
  • Hive block log & shared memory: 500GB
  • Base HAF database: 3.5T (before 2x lz4 compression)
  • Hivemind database: 0.65T (before 2x lz4 compression)
  • base HAF + Hivemind: 2.14T (compressed)
  • HAF Block Explorer: ~0.2T

Running Hive node with Docker

Install ZFS support

We strongly recommend running your HAF instance on a ZFS filesystem, and this documentation assumes you will be running ZFS. Its compression and snapshot features are particularly useful when running a HAF node. We intend to publish ZFS snapshots of fully-synced HAF nodes that can downloaded to get a HAF node up & running quickly, avoiding multi-day replay times.

sudo apt install zfsutils-linux

Install Docker

Follow official guide https://docs.docker.com/engine/install/.

Create a ZFS pool

Create your ZFS pool if necessary. HAF requires at least 4TB of space, and 2TB NVMe drives are readily available, so we typically construct a pool striping data across several 2TB drives. If you have three or four drives, you will get somewhat better read/write performance, and the extra space can come in handy. To create a pool named “haf-pool” using the first two NVMe drives in your system, use a command like:

sudo zpool create haf-pool /dev/nvme0n1 /dev/nvme1n1

If you name your ZFS pool something else, configure the name in the environment file, as described in the next section. Note: By default, ZFS tries to detect your disk’s actual sector size, but it often gets it wrong for modern NVMe drives, which will degrade performance due to having to write the same sector multiple times. If you don’t know the actual sector size, we recommend forcing the sector size to 8k by specifying setting ashift=13 (e.g., zfs create -o ashift=13 haf-pool /dev….)

Configure your environment

Clone HAF API Node repository from here https://github.com/openhive-network/haf_api_node Make a copy of the file .env.example and customize it for your system. This file contains configurable parameters for things like directories versions of hived, HAF, and associated tools The docker compose command will automatically read the file named .env. If you want to keep multiple configurations, you can give your environment files different names like .env.dev and .env.prod, then explicitly specify the filename when running docker compose: docker compose --env-file=.env.dev ...

Set up ZFS filesystems

The HAF installation is spread across multiple ZFS datasets, which allows us to set different ZFS options for different portions of the data. We recommend that most nodes keep the default datasets in order to enable easy sharing of snapshots.

Initializing from scratch

If you’re starting from scratch, after you’ve created your zpool and configured its name in the .env file as described above, run:

sudo ./create_zfs_datasets.sh

to create and mount the datasets. By default, the dataset holding most of the database storage uses zfs compression. The dataset for the blockchain data directory (which holds the block_log for hived and the shared_memory.bin file) is not compressed because hived directly manages compression of the block_log file. If you have a LOT of nvme storage (e.g. 6TB+), you can get better API performance at the cost of disk storage by disabling ZFS compression on the database dataset, but for most nodes this isn’t recommended.

Assisted startup

./assisted_startup.sh

Depending on your environment variables, assisted start up script will quickly bootstrap the process.

Building Without Docker

Full non-docker steps can be reviewed here:

Build Eclipse by @gtg

Syncing blockchain

Initializing from a snapshot

If you’re starting with one of our snapshots, the process of restoring the snapshots will create the correct datasets with the correct options set. First, download the snapshot file from: https://gtg.openhive.network/get/snapshot/ Since these snapshots are huge, it’s best to download the snapshot file to a different disk (a magnetic HDD will be fine for this) that has enough free space for the snapshot first, then restore it to the ZFS pool. This lets you easily resume the download if your transfer is interrupted. If you download directly to the ZFS pool, any interruption would require you to start the download from the beginning.

wget -c https://whatever.net/snapshot_filename

If the transfer gets interrupted, run the same command again to resume. Then, to restore the snapshot, run:

sudo zfs recv -d -v haf-pool < snapshot_filename

Replay with blocklog

Normally syncing blockchain starts from very first, 0 genesis block. It might take long time to catch up with live network, because it connects to various p2p nodes in the Hive network and requests blocks from 0 to head block.
It stores blocks in block log file and builds up the current state in the shared memory file. But there is a way to bootstrap syncing by using trusted block_log file. The block log is an external append only log of the blocks.
It contains blocks that are only added to the log after they are irreversible because the log is append only.

Trusted block log file helps to download blocks faster. Various operators provide public block log file which can be downloaded from:

• https://files.privex.io/hive/
• https://gtg.openhive.network/get/blockchain/block_log

Both block_log files updated periodically, as of March 2021 uncompressed block_log file size ~350 GB. (Docker container on stable branch of Hive source code has option to use USE_PUBLIC_BLOCKLOG=1 to download latest block log and start Hive node with replay.)

Block log should be place in blockchain directory below data_dir and node should be started with --replay-blockchain to ensure block log is valid and continue to sync from the point of snapshot. Replay uses the downloaded block log file to build up the shared memory file up to the highest block stored in that snapshot and then continues with sync up to the head block.

Replay helps to sync blockchain in much faster rate, but as blockchain grows in size replay might also take some time to verify blocks.

There is another trick which might help with faster sync/replay on smaller equipped servers:

while :
do
   dd if=blockchain/block_log iflag=nocache count=0
   sleep 60
done

Above bash script drops block_log from the OS cache, leaving more memory free for backing the blockchain database. It might also help while running live, but measurement would be needed to determine this.

Few other tricks that might help:

For Linux users, virtual memory writes dirty pages of the shared file out to disk more often than is optimal which results in hived being slowed down by redundant IO operations. These settings are recommended to optimize reindex time.

echo    75 | sudo tee /proc/sys/vm/dirty_background_ratio
echo  1000 | sudo tee /proc/sys/vm/dirty_expire_centisecs
echo    80 | sudo tee /proc/sys/vm/dirty_ratio
echo 30000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs

Another settings that can be changed in config.ini is flush-state-interval - it is to specify a target number of blocks to process before flushing the chain database to disk. This is needed on Linux machines and a value of 100000 is recommended. It is not needed on OS X, but can be used if desired.