Hive Developer Portal
Setup HAF API node
Setup full or light Hive nodes with HAF API
Using docker compose to install and maintain a HAF server and HAF apps
System Requirements
We assume the base system will be running at least Ubuntu 22.04 (jammy). Everything will likely work with later versions of Ubuntu. IMPORTANT UPDATE: experiments have shown 20% better API performance when running U23.10, so this latter version is recommended over Ubuntu 22 as a hosting OS.
For a mainnet API node, we recommend:
- at least 32GB of memory. If you have 64GB, it will improve the time it takes to sync from scratch, but it should make less of a difference if you’re starting from a mostly-synced HAF node (i.e., restoring a recent ZFS snapshot) (TODO: quantify this?)
- 4TB of NVMe storage
- Hive block log & shared memory: 500GB
- Base HAF database: 3.5T (before 2x lz4 compression)
- Hivemind database: 0.65T (before 2x lz4 compression)
- base HAF + Hivemind: 2.14T (compressed)
- HAF Block Explorer: xxx
Install prerequisites
Install ZFS support
We strongly recommend running your HAF instance on a ZFS filesystem, and this documentation assumes you will be running ZFS. Its compression and snapshot features are particularly useful when running a HAF node.
We intend to publish ZFS snapshots of fully-synced HAF nodes that can downloaded to get a HAF node up & running quickly, avoiding multi-day replay times.
sudo apt install zfsutils-linux
Install Docker
Install the latest docker. If you’re running Ubuntu 22.04, the version provided by the native docker.io package is too old to work with the compose scripts. Install the latest version from docker.com, following the instructions here:
https://docs.docker.com/engine/install/ubuntu/
Which are:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Create a ZFS pool
Create your ZFS pool if necessary. HAF requires at least 4TB of space, and 2TB NVMe drives are readily available, so we typically construct a pool striping data across several 2TB drives. If you have three or four drives, you will get somewhat better read/write performance, and the extra space can come in handy.
To create a pool named “haf-pool” using the first two NVMe drives in your system, use a command like:
sudo zpool create haf-pool /dev/nvme0n1 /dev/nvme1n1
If you name your ZFS pool something else, configure the name in the environment file, as described in the next section.
Note: By default, ZFS tries to detect your disk’s actual sector size, but it often gets it wrong
for modern NVMe drives, which will degrade performance due to having to write the same sector multiple
times. If you don’t know the actual sector size, we recommend forcing the sector size to 8k by
specifying setting ashift=13 (e.g., zfs create -o ashift=13 haf-pool /dev....
)
Configure your environment
Make a copy of the file .env.example
and customize it for your system. This file contains
configurable paramters for things like
- directories
- versions of hived, HAF, and associated tools
The docker compose
command will automatically read the file named .env
. If you want to
keep multiple configurations, you can give your environment files different names like
.env.dev
and .env.prod
, then explicitly specify the filename when running docker compose
:
docker compose --env-file=.env.dev ...
Set up ZFS filesystems
The HAF installation is spread across multiple ZFS datasets, which allows us to set different ZFS options for different portions of the data. We recommend that most nodes keep the default datasets in order to enable easy sharing of snapshots.
Initializing from scratch
If you’re starting from scratch, after you’ve created your zpool and configured its name in the .env file as described above, run:
sudo ./create_zfs_datasets.sh
to create and mount the datasets.
By default, the dataset holding most of the database storage uses zfs compression. The dataset for the blockchain data directory (which holds the block_log for hived and the shared_memory.bin file) is not compressed because hived directly manages compression of the block_log file.
If you have a LOT of nvme storage (e.g. 6TB+), you can get better API performance at the cost of disk storage by disabling ZFS compression on the database dataset, but for most nodes this isn’t recommended.
Speeding up the initial sync
Following the instructions above will get you a working HAF node, but there are some things you can do to speed up the initial sync.
Replaying
If you already have a recent block_log file (e.g., you’re already running another instance of hived
somewhere else on your local network), you can copy the block_log and block_log.artifacts files
from that node into your /haf-pool/haf-datadir/blockchain directory. After copying the files,
make sure the ownership is set to the same owner as the /haf-pool/haf-datadir/blockchain directory
so hived can read/write them: chown 1000:100 block_log block_log.artifacts
Before brining up the haf service, you will also need to add the --replay-blockchain
argument to
hived to tell it you want to replay. Edit the .env
file’s ARGUMENTS
line like so:
ARGUMENTS="--replay-blockchain"
Once the replay has finished, you can revert the ARGUMENTS
line to the empty string
Shared Memory on Ramdisk
If you have enough spare memory on your system, you can speed up the initial replay by placing the
shared_memory.bin
file on a ramdisk.
The current default shared memory filesize is 24G, so this will only work if you have 24G free (that’s in addition to the memory you expect to be used by hived and HAF’s integrated PostgreSQL instance).
If you have a 64GB system, ensure you have a big enough swapfile (32GB is recommended and 8GB is known to not be sufficient) to handle peak memory usage needs during the replay. Peak memory usage currently occurs when haf table indexes are being built during the final stage of replay.
To do this, first create a ramdisk:
sudo mkdir /mnt/haf_shared_mem
# then
sudo mount -t tmpfs -o size=25g tmpfs /mnt/haf_shared_mem
# - or -
sudo mount -t ramfs ramfs /mnt/haf_shared_mem
# then
sudo chown 1000:100 /mnt/haf_shared_mem
Then, edit your .env
file to tell it where to put the shared memory file:
HAF_SHM_DIRECTORY="/mnt/haf_shared_mem"
Now, when you resync / replay, your shared memory file will actually be in memory.
Moving Shared Memory back to disk
Once your replay is finished, we suggest moving the shared_memory.bin file back to NVMe storage, because:
- it doesn’t make much performance difference once hived is in sync
- you’ll be able to have your zfs snapshots include your shared memory file
- you won’t be forced to replay if the power goes out
To do this:
- take down the stack (
docker compose down
). - copy the shared memory:
sudo cp /mnt/haf_shared_mem/shared_memory.bin /haf-pool/haf-datadir/blockchain
- destroy the ramdisk:
sudo umount /mnt/haf_shared_mem
- update the
.env
file’s location:HAF_SHM_DIRECTORY="${TOP_LEVEL_DATASET_MOUNTPOINT}/blockchain"
- bring the stack back up (
docker compose up -d
)
Initializing from a snapshot
If you’re starting with one of our snapshots, the process of restoring the snapshots will create the correct datasets with the correct options set.
First, download the snapshot file from: TODO: http://xxxxxx
Since these snapshots are huge, it’s best to download the snapshot file to a different disk (a magnetic HDD will be fine for this) that has enough free space for the snapshot first, then restore it to the ZFS pool. This lets you easily resume the download if your transfer is interrupted. If you download directly to the ZFS pool, any interruption would require you to start the download from the beginning.
wget -c https://whatever.net/snapshot_filename
If the transfer gets interrupted, run the same command again to resume.
Then, to restore the snapshot, run:
sudo zfs recv -d -v haf-pool < snapshot_filename
Launch procedure
start/stop HAF instance based on profiles enabled in your .env
file
docker compose up -d
docker compose logs -f hivemind-block-processing # tail the hivemind sync logs to the console
docker compose down hivemind-block-processing # shut down just the hivemind sync process
docker compose up -d hivemind-block-processing # bring hivemind sync process back up
docker compose down # shut down all containers
This will start or stop all services selected by the profiles you have
enabled in the .env
file’s COMPOSE_PROFILES
variable.
Currently available profiles are:
core
: the minimal HAF system of a database and hivedadmin
: useful tools for administrating HAF: pgadmin, pgheroapps
: core HAF apps: hivemind, HAfAH, haf-block-explorerservers
: services for routing/caching API calls: haproxy, jussi,varnish
Observing node startup
After you start your HAF instance, hived will need some time to catch up to the head block
of the Hive blockchain (typically a few minutes or less if you started from a snapshot,
otherwise it will take many hours or even days depending on your hardware). You can monitor
this process using: docker compose logs -f haf
If syncing or replaying for the first time, HAF will delay creating indexes on its tables until the blockchain data has mostly been added to the database. This means there will be a noticeable delay near the end of the catchup period while these indexes get created. Even on a fast machine this post-sync/replay process currently takes over 2 hours to create the indexes, and another two hours to cluster the account_operations table, so be patient. Do not interrupt the process or your database will be left in an invalid state and might require another full replay.
If you enabled the “admin” profile, you can use pghero’s “Live Queries” view to monitor this process (e.g https://your_server/admin/pghero/live_queries). If not, you can still observe the cpu and disk io usage by postgresql during this process if you run a tool like htop.
After startup: Monitoring services and troubleshooting failures on your API node
Haproxy can be used to monitor the state of the various services on your HAF server:
https://your_server_name/admin/haproxy/
If you see a service is down, you can use an appropriate docker compose log
command to
diagnose the issue. When diagnosing issues, keep in mind that several services depend on other services
(for example, all haf apps depend on the hived service) so start by checking the health of the lowest level
services.
You can diagnose API performance problems using pgAdmin and PgHero. pgAdmin is best for diagnosing severe problems (e.g. locked tables, etc) whereas PgHero is typically best for profiling to determine what queries are loading down your server and can potentially be optimized.
https://your_server_name/admin/
Creating a ZFS snapshot to backup your node
Creating snapshots is fast and easy:
docker compose down #shut down haf
./snapshot_zfs_datasets.sh 20231023T1831Z-haf-only # where 20231023T1831Z-haf-only is an example snapshot name
docker compose up -d
Note: snapshot_zfs_datasets.sh unmounts the HAF datasets, takes a snapshot, and remounts them. Since it unmounts the datasets, the script will fail if you have anything accessing the datasets. In particular, be sure you don’t have any terminals open with a current working directory set to those datasets. In theory, the script shouldn’t have to unmount the datasets before taking the snapshot, but we have occassionally encountered issues where the snapshots didn’t get all needed data.
Deleting Hivemind data from your database (or a similar app’s data)
You may want to remove the Hivemind app’s data from your database – either because you no longer need it and want to free the space, or because you want want to replay your Hivemind app from scratch, which is required for some upgrades.
To delete the data:
- stop Hivemind, but leave the rest of the stack running:
docker compose down hivemind-install hivemind-block-processing hivemind-server
- run the uninstall script:
docker compose --profile=hivemind-uninstall up
- you’ll see the results of a few sql statements scroll by, and it should exit after a few seconds
The Hivemind data is now gone.
If you’re uninstalling Hivemind permanently, then remember to remove the hivemind
profile from your .env
file’s COMPOSE_PROFILES
line so it doesn’t start automatically next time you do a docker compose up -d
.
If you’re upgrading to a new version of hivemind:
- if you’re upgrading to a pre-release version, you’ll need to set
HIVEMIND_INSTANCE_VERSION
in your.env
file to the correct tag for the version you want to run. If you’re just upgrading to a new release version (the ones taggedhaf_api_node
), you can leave this alone. - run
docker compose pull
to grab the new version - run
docker compose up -d
to bring up all services. This should run hivemind’s install, then launch the block processing container. - you can monitor Hivemind’s sync process by watching the logs from
docker compose logs -f hivemind-block-processing
. In a few short days, your Hivemind app should be fully synced and ready to handle API requests.