==== Docker Cluster HA Setup ====

**1**. Setup a VPN in-between your 3,5,7 or more servers.
This can be done with TincVPN for example but there are many others you can choose from.

https://www.tinc-vpn.org/

**2**. Installing GlusterFS

  apt-get install gpg -y
  curl https://download.gluster.org/pub/gluster/glusterfs/11/rsa.pub | gpg --dearmor > /usr/share/keyrings/glusterfs-archive-keyring.gpg
  DEBID=$(grep 'VERSION_ID=' /etc/os-release | cut -d '=' -f 2 | tr -d '"')
  DEBVER=$(grep 'VERSION=' /etc/os-release | grep -Eo '[a-z]+')
  DEBARCH=$(dpkg --print-architecture)
  echo "deb [signed-by=/usr/share/keyrings/glusterfs-archive-keyring.gpg] https://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/${DEBID}/${DEBARCH}/apt ${DEBVER} main" | sudo tee /etc/apt/sources.list.d/gluster.list
  apt-get update && apt-get install glusterfs-server -y

**3**. Edit /etc/glusterfs/glusterd.vol and add \\
This will prevent glusterfs from getting exposed to the dangerous interwebs.
  option transport.socket.bind-address 10.0.X.1
  
rpcbind does listen on the network and we don't need it, so lets get rid of it.
  apt-get remove rpcbind -y

**4**. Enable GlusterFS

  systemctl start glusterd && systemctl enable glusterd

**5**. Peer with your GlusterFS nodes

  gluster peer probe 10.0.2.1
  gluster peer probe 10.0.3.1

**6**. Check the peering status
  gluster peer status

**7.** Folders for the mount
  mkdir -p /mnt/bricks/docker &&   mkdir -p /mnt/data/docker

**8.** Create your first volume for Docker
  gluster volume create docker replica 3 10.0.1.1:/mnt/bricks/docker 10.0.2.1:/mnt/bricks/docker 10.0.3.1:/mnt/bricks/docker force
  gluster volume start docker

**9**. Mount your first volume
  mount.glusterfs 10.0.X.1:/docker /mnt/data/docker
  
**10**. Make the mount boot ready

  [Unit]
  Description=mounts service service
  Wants=network-online.target glusterd.service
  After=network-online.target glusterd.service
  [Service]
  User=root
  Group=root
  ExecStartPre=sleep 5
  ExecStart=mount.glusterfs 10.0.X.1:/docker /mnt/data/docker
  RemainAfterExit=true
  Type=oneshot
  [Install]
  WantedBy=multi-user.target
Copy this to /etc/systemd/system/mounts.service

**11**. Enable the mount service
  systemctl enable mounts
  
**12**. You may have to edit the GlusterFS systemd file to prevent a race condition with your VPN. \\
GlusterFS will fail to start if your VPN isn't running already.

You can do this with 
  systemctl edit glusterd --full
Added one line

  ExecStartPre=/bin/sh -c 'until ping -c1 10.0.X.1; do sleep 1; done;'
Profit! Next reboot GlusterFS should start up fine.

**13**. Install Docker

  # Add Docker's official GPG key:
  apt-get update
  apt-get install ca-certificates curl -y
  install -m 0755 -d /etc/apt/keyrings
  curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
  chmod a+r /etc/apt/keyrings/docker.asc
  # Add the repository to Apt sources:
  echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
  # Install Docker
  apt-get update && apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

**14**. Init the Swarm on the first Node
  docker swarm init --advertise-addr 10.0.1.1 --listen-addr=10.0.1.1
advertise-addr will only advertise the swarm inside our VPN network  

**15**. Join other Nodes
  docker swarm join --token whateverthattokenis 10.0.1.1:2377 --listen-addr=10.0.2.1
  docker swarm join --token whateverthattokenis 10.0.1.1:2377 --listen-addr=10.0.3.1
listen-addr will force swarm to bind to your local VPN

**16**. Check the Cluster
  docker node ls

**17**. Promote the other Nodes to archive 100% True HA
  docker node promote node2
  docker node promote node3
  
**18**. Deploy your first service \\
In my case it was a ZNC bouncer. \\
Had to run the docker container normally to generate the config files. \\

  docker run -it -v /mnt/data/docker/znc/:/znc-data znc --makeconf
  
Lets deploy the service.
  docker service create --mount type=bind,src=/mnt/data/docker/znc/,dst=/znc-data --publish published=1025,target=1025 --name bouncer znc
The service will get exposed on port 1025 on all nodes.
  
**19**. If you run this, on any node.
  docker node ps $(docker node ls -q)
  
You should be able to check your container status.

**20**. When you reboot the node with your container, the service should be restored in about 60s.