Copy this file to /etc/nginx/sites-enabled/proxmox

server {
  listen 443;
  server_name yourserver.yourdomain.com;
  
  ssl on;
  ssl_certificate /etc/nginx/ssl/ssl.crt;
  ssl_certificate_key /etc/nginx/ssl/ssl.key;
  
  proxy_redirect off;
  location / {
      proxy_set_header X-Forwarded-Proto https;
      proxy_pass https://127.0.0.1:8006;
      
      proxy_http_version 1.1;
      proxy_set_header Connection $http_connection;
      proxy_set_header Origin http://$host;
      proxy_set_header Upgrade $http_upgrade;
  }
}

Make sure you replace ssl_certificate and ssl_certificate_key

Restart Nginx:

service nginx restart

Second step, Turn the pveproxy to localhost only. Copy this file to: /etc/default/pveproxy

ALLOW_FROM="127.0.0.1"
DENY_FROM="all"
POLICY="allow"

You can also block it over iptables, since it does not fully work anymore on 5.x.

post-up iptables -A INPUT -p tcp --dport 8006 -s 127.0.0.0/8 -j ACCEPT #allow localhost for reverse proxy
post-up iptables -A INPUT -p tcp --dport 8006 -j DROP #webinterface
post-up iptables -A INPUT -p tcp --dport 3128 -j DROP #spiceproxy

Restart pveproxy:

service pveproxy restart

https://YOURIP:8006 should be not more reachable.