Security Best Practices
Create a non-root user with sudo
privileges
It is a good idea to log in as a non-root user. Logins with root permissions are often overlooked as a security risk, but this is not the case! The command rm
can wipe your entire server if run incorrectly by a root user. By logging in as a non-root user, you can avoid this issue.
Do NOT routinely use the root account. Please use su
instead.
Connect via SSH to your node
ssh username@server.public.ip.address
# example
# ssh ubuntu@17.23.161.11
Create a new user called haqq_node, for example
sudo useradd -m -s /bin/bash haqq_node
Set the password for haqq_node user
sudo passwd haqq_node
Add haqq_node to the sudo group
sudo usermod -aG sudo haqq_node
Use SSH Keys only connection
The basic rules of hardening SSH are:
- Don't use password for SSH access (use private key)
- Don't allow root to SSH (the appropriate users should SSH in, then
su
orsudo
) - Use
sudo
for users so commands are logged - Log unauthorized login attempts (and consider software to block/ban users who try to access your server too many times, for example fail2ban)
- Lock down SSH to only the ip range your require (optional)
You will need to create a key pair on your local machine, with the public key stored in the keyname
file.
ssh-keygen -t ed25519
Transfer the public key keyname.pub
to your remote node.
ssh-copy-id -i $HOME/.ssh/keyname.pub haqq_node@server.public.ip.address
Login with your new haqq_node user
ssh haqq_node@server.public.ip.address
Disable root login and password based login. Edit the /etc/ssh/sshd_config file
using any text editor, for example nano
sudo nano /etc/ssh/sshd_config
Locate attributtes in sshd_config
:
ChallengeResponseAuthentication
PasswordAuthentication
PermitEmptyPasswords
And modify their with no
parameter
ChallengeResponseAuthentication no
PasswordAuthentication no
PermitEmptyPasswords no
And also find PermitRootLogin
attributte. Edit it with prohibit-password
parameter
PermitRootLogin prohibit-password
Optional: You can also customize SSH Port with your custom numeric value.
Please check for possible conflicts first
Port <port number>
Then validate the syntax of your new SSH configuration using this command
sudo sshd -t
If no errors with the syntax validation, restart the SSH process
sudo systemctl restart sshd
Verify that the ssh login still works
Standard SSH Port is 22
ssh haqq_node@server.public.ip.address
Alternatively, you might need to add the -p <port#>
flag if you used a custom SSH port.
ssh haqq_node@server.public.ip.address -p <custom port number>
Connection command
ssh -i <path to your SSH_key_name.pub> haqq_node@server.public.ip.address
Optional: Make logging in easier by updating your local ssh config.
You can also simplify the ssh command needed to log in to your server, consider updating your local $HOME/.ssh/config
file:
Host ubuntu
User haqq_node
HostName <server.public.ip.address>
Port <custom port number>
This will allow you to log in with ssh haqq_node
rather than needing to pass through all ssh parameters explicitly.
Update your system
It's critically important to keep your system up-to-date with the latest patches to prevent intruders from accessing your system.
sudo apt-get update -y && \
sudo apt dist-upgrade -y && \
sudo apt-get autoremove && \
sudo apt-get autoclean
Enable automatic updates so you don't have to manually install them.
sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
Disable root account
Use sudo execute to run commands as low-level users without requiring their own privileges.
# To disable the root account, simply use the -l option.
sudo passwd -l root
# If for some valid reason you need to re-enable the account, simply use the -u option.
sudo passwd -u root
Setup Two Factor Authentication for SSH (optional)
SSH can be a great tool for connecting to remote Linux systems. What if you could add another layer of security. With 2FA
, you can create a another security layer and it can be implemented in several ways, for example it works well with Google Authenticator.
sudo apt install libpam-google-authenticator -y
To make SSH use the Google Authenticator PAM module, just edit the /etc/pam.d/sshd
file:
sudo nano /etc/pam.d/sshd
Add this line:
auth required pam_google_authenticator.so
Now you need to restart the sshd
daemon using:
sudo systemctl restart sshd.service
Modify /etc/ssh/sshd_config
sudo nano /etc/ssh/sshd_config
Find
ChallengeResponseAuthentication
UsePAM
And update this attributtes with yes
ChallengeResponseAuthentication yes
UsePAM yes
Save the file and exit.
Execute google-authenticator
command.
google-authenticator
It will ask you a series of questions, here is one of recommended configuration:
- Make tokens “time-base”": yes
- Update the
.google_authenticator
file:yes
- Disallow multiple uses:
yes
- Increase the original generation time limit:
no
- Enable rate-limiting:
yes
If you see the QR code and don’t have your phone, it will lead you to a website that has your emergency scratch codes printed on a card. You can keep the card in a safe place, so you won't need to dig around for it during an emergency.
Now, open Google Authenticator
on your phone and add your secret key to make two factor authentication work.
If you are enabling 2FA on a remote machine that you access over SSH you need to follow steps 2 and 3 of this tutorial to make 2FA work.
Secure Shared Memory (optional)
One of the first things you should do is secure the shared memory used on the system. If you're unaware, shared memory can be used in an attack against a running service. Because of this, secure that portion of system memory.
To learn more about secure shared memory, read this techrepublic.com article.
Install Fail2ban (optional)
Fail2ban is an intrusion prevention system that monitors log files and searches for particular patterns that correspond to failed login attempts. If a certain number of failed logins are detected from a specific IP address (within a specified amount of time), fail2ban blocks access from that IP address. Fail2ban works by automatically issuing the following actions against offending clients: iptables or ip6tables firewall rule, Fail2Ban action, efence iptables firewall rule, SSHGuard filter.
sudo apt-get install fail2ban -y
Then edit a config file that monitors SSH logins.
sudo nano /etc/fail2ban/jail.local
Add a following lines to the bottom of the file.
Whitelisting IP address tip: The ignoreip
parameter accepts a list of IP addresses, IP ranges or DNS hosts that you can specify to be allowed to connect. This is where you want to specify your local machine, local IP range or local domain, separated by spaces.
# Example
ignoreip = 192.168.1.0/24 127.0.0.1/8
[sshd]
enabled = true
port = <22 or your random port number>
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
# whitelisted IP addresses
ignoreip = <list of whitelisted IP address, your local daily laptop/pc>
Don't forget to save file. Restart fail2ban to take effect.
sudo systemctl restart fail2ban
Configure your Firewall
What is a Firewall?
Firewall is a network security system that filters and controls the traffic on a predetermined set of rules. This is an intermediary system between the device and the internet.
There are a lot of ways to do this, but the easiest way I find is with iptables-persistent package. You can download the package from Ubuntu’s default repositories:
sudo apt-get update
sudo apt-get install iptables-persistent
Once the installation is complete, you can save your configuration using the command:
sudo invoke-rc.d iptables-persistent save
Simple firewall configuration for validator
After executing this command, we will create rules with allowed ports 22
and 26656
, the rest of the ports will be denied on INPUT
chain:
- Port 22 (or your custom port) TCP for SSH connection
- Port 26656 tcp for p2p for incoming connections
- Deny other
INPUT
connections
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT && \
sudo iptables -A INPUT -p tcp --dport 26656 -j ACCEPT && \
sudo iptables -P INPUT DROP
Now you can check the iptable
configuration using -L
flag.
sudo iptables -L
A more detailed description of used ports can be found below.
Rule for ssh connection
We want to keep our SSH port open from the 192.168.1.3. That is we only want to allow those packets coming from 192.168.1.3 and which wants to go to the port 22.
If you don't have static IP address we don't recommend add to add this IP specific rule. For that case just allow SSH port (by default it use 22
)
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
Execute the below command:
sudo iptables -A INPUT -s 192.168.1.3 -p tcp --dport 22 -j ACCEPT
The above command says looks for the packets originating from the IP address 192.168.1.3, having a TCP protocol and who wants to deliver something at the port 22 of your node. If you find those packets then Accept them.
Now check the iptable
configuration using -L
flag.
sudo iptables -L
Therefore, any packet coming from 192.168.1.3 is first checked if it is going to the port 22 if it isn’t then it is run through the next rule in the chain. Else it is allowed to pass the firewall.
Limit network connections
The key idea is to restrict all connections and allow only used ports for security reasons.
You can find used ports in your node config files and make your own firewall rules and based on your node setup:
~/.haqqd/config/app.toml
~/.haqqd/config/config.toml
~/.haqqd/config/client.toml
Here is the best practice of used services:
port | service | rpc node | validator |
---|---|---|---|
1317 | API server | ✅ | ❌ |
8080 | Rosseta API | ✅ | ❌ |
9090 | gRPC server | ✅ | ❌ |
1317 | gRPC web | ✅ | ❌ |
8545 | EMV RPC HTTP | ✅ | ❌ |
8546 | EMV RPC WebSocket | ✅ | ❌ |
26657 | Tendermint RPC | ✅ | ❌ |
26658 | ABCI application | ❌ | ❌ |
6060 | pprof (Go pkg) | ❌ | ❌ |
26656 | p2p incoming connections | ✅ | ✅ |
26660 | Prometheus | ❌ | ❌ |
For example, you can check your config files for services used:
File app.toml
###############################################################################
### gRPC Configuration ###
###############################################################################
[grpc]
# Enable defines if the gRPC server should be enabled.
enable = false
###############################################################################
### gRPC Web Configuration ###
###############################################################################
[grpc-web]
# GRPCWebEnable defines if the gRPC-web should be enabled.
# NOTE: gRPC must also be enabled, otherwise, this configuration is a no-op.
enable = false
###############################################################################
### JSON RPC Configuration ###
###############################################################################
[json-rpc]
# Enable defines if the gRPC server should be enabled.
enable = false
As you can see there are a few list of services, and we can have limit access to them via creating some specific rules:
Here is an examples of port specific rules:
TCP or UNIX socket address for the RPC server to listen on
sudo iptables -A INPUT -p tcp --dport 26657 -j DROP
TCP or UNIX socket address of the ABCI application, or the name of an ABCI application compiled in with the Tendermint binary
sudo iptables -A INPUT -p tcp --dport 26658 -j DROP
We don't recommend limiting connections to this port
(26656) because it is used in p2p communications between nodes.
# Address to listen for incoming connections
laddr = "tcp://0.0.0.0:26656"
Removing all other traffic
sudo iptables -A INPUT -j DROP
Additional info
You can find additional information and examples here
UFW configuration (alternatively)
The standard UFW
firewall can be used to control network access to your node.
With any new installation, ufw is disabled by default. Enable it with the following settings.
- Port 22 (or your custom port) TCP for SSH connection
- Port 26656 tcp for p2p for incoming connections (necessary)
# By default, deny all incoming and outgoing traffic
sudo ufw default deny incoming
sudo ufw default allow outgoing
# Allow ssh access
sudo ufw allow ssh # port 22 or your custom ssh port number
# Allow necessary port
sudo ufw allow 26656/tcp
# Enable firewall
sudo ufw enable
# Verify status
sudo ufw status numbered
Please don't exposed Grafana and Prometheus ports to the public internet. This introduces a new attack surface and would allow malicious attackers to access your data. You can built a secure solution with Wireguard, for example, and recommend you do so as well!
If you are planning to use Grafana or (and) Prometheus don't forget to allow them too.
# Allow grafana web server port
sudo ufw allow 3000/tcp
# Enable prometheus endpoint port
sudo ufw allow 26660/tcp
Optional but recommended Whitelisting (or permitting connections from a specific IP) can be setup via the following command.
sudo ufw allow from <your local daily laptop/pc>
# Example
# sudo ufw allow from 192.168.50.22
Port Forwarding Tip: You'll need to forward and open ports to your validator. Verify if it's working via using this service for example.
Or you can also use netcat
(or nc in short) is a powerful and easy-to-use utility that can be employed for just about anything in Linux in relation to TCP, UDP, or UNIX-domain sockets.
Verify Listening Ports
If you want to maintain a secure server, you should validate the listening network ports every once in a while. This will provide you essential information about your network.
sudo ss -tulpn
Alternatively you can also use netstat
sudo netstat -tulpn
VPN connection (optional)
Recommended for Advanced Users Only
It is good practice to arrange access to your node using a vpn connection or a secure tunnel and at the same time restrict access to the node from the external network.
You will learn how to set up secure and encrypt network traffic between two machines using WireGuard, greatly minimizing the chance of your local host being attacked by intruders and minimizing the attack surface of a remote host without requiring you to open ports for services like Grafana.
So, for example we can use WireGuard for secure connect with node.
Install WireGuard
sudo apt install linux-headers-generic && \
sudo add-apt-repository ppa:wireguard/wireguard -y && \
sudo apt-get update && \
sudo apt-get install wireguard -y
Setting Up Key Pairs
sudo su
cd /etc/wireguard
umask 077
wg genkey | tee wireguard-privatekey | wg pubkey > wireguard-publickey
Create a wg0.conf configuration file in /etc/wireguard directory. Update your Private and Public Keys accordingly. Change the Endpoint to your remote node public IP or DNS address.
Local machine config
# local node WireGuard Configuration
[Interface]
# local node address
Address = 10.0.0.1/32
# local node private key
PrivateKey = <i.e. SJ6ygM3csa36...+pO4XW1QU0B2M=>
# local node wireguard listening port
ListenPort = 51820
# remote node
[Peer]
# remote node's publickey
PublicKey = <i.e. Rq7QEe2g3qIjDftMu...knBGS9mvJDCa4WQg=>
# remote node's public ip address or dns address
Endpoint = remotenode.mydomainname.com:51820
# remote node's interface address
AllowedIPs = 10.0.0.2/32
PersistentKeepalive = 21
Node side config
# remote node WireGuard Configuration
[Interface]
Address = 10.0.0.2/32
PrivateKey = <i.e. cF3OjVhtKJAY/rQ...Fi7ASWg=>
ListenPort = 51820
# local node
[Peer]
# local node's public key
PublicKey = <i.e. rZLBzslvFtEJ...dfX4XSwk=>
# local node's public ip address or dns address
Endpoint = localnodesIP-or-domain.com:51820
# local node's interface address
AllowedIPs = 10.0.0.1/32
PersistentKeepalive = 21
Configure Firewall
Configure UFW
on both machines
sudo ufw allow 51820/udp
sudo ufw allow from 10.0.0.0/16 to any
# check the firewall rules
sudo ufw verbose
Autostart
Add WireGuard service to systemd
sudo systemctl enable wg-quick@wg0.service
sudo systemctl daemon-reload
Start service
Then you can start and check service
sudo systemctl start wg-quick@wg0
sudo systemctl status wg-quick@wg0
After that you can verify connect between machines
sudo wg
## Example Output
# interface: wg0
# public key: rZLBzslvFtEJ...dfX4XSwk=
# private key: (hidden)
# listening port: 51820
#peer:
# endpoint: 11.34.56.18:51820
# allowed ips: 10.0.0.2/32
# latest handshake: 11 seconds ago
# transfer: 500 KiB received, 900 KiB sent
# persistent keepalive: every 21 seconds
ping 10.0.0.2
To stop and disable WireGuard execute
sudo systemctl stop wg-quick@wg0 && \
sudo systemctl disable wg-quick@wg0.service && \
sudo systemctl daemon-reload
And of course you can also create your own firewall rules depending on the network and node configuration
Additional best practices
Networking | In order to use ufw and Fail2ban's whitelisting feature, you need to assign static internal IPs to your validator node and daily laptop/PC. This is useful in conjunction with ufw and Fail2ban's whitelisting feature. |
Power Outage | Make sure your nodes are backed up by turning on the Uninterruptible Power Supply (UPS), which will allow for power surges if there is a blackout. |
Clear the bash history | When pressing the up-arrow key, you can see prior commands which may contain sensitive data. To clear this, run the following:
|
Additional information
You can find more information about firewall configuration here:
Key security
The test
backend is a password-less variation of the file backend. Keys are stored unencrypted on disk. This keyring is provided for testing purposes only. Use at your own risk!
🚨 DANGER: Never create your mainnet validator keys using a test keying backend. Doing so might result in a loss of funds by making your funds remotely accessible via the eth_sendTransaction
JSON-RPC endpoint.
Security Advisory: Insecurely configured geth can make funds remotely accessible
We recommend using Tendermint KMS that allows separating key management from Tendermint nodes. It is recommended that the KMS service runs in a separate physical hosts.
Sentry Node
Validators are responsible for ensuring that the network can sustain denial of service attacks.
One recommended way to mitigate these risks is for validators to carefully structure their network topology in a so-called sentry node architecture.
More information about sentry node you can find here