Copy this file to /etc/nginx/sites-enabled/proxmox

  server {
    listen 443;
    server_name yourserver.yourdomain.com;
    
    ssl on;
    ssl_certificate /etc/nginx/ssl/ssl.crt;
    ssl_certificate_key /etc/nginx/ssl/ssl.key;
    
    proxy_redirect off;
    location / {
        proxy_set_header X-Forwarded-Proto https;
        proxy_pass https://127.0.0.1:8006;
        
        proxy_http_version 1.1;
        proxy_set_header Connection $http_connection;
	      proxy_set_header Origin http://$host;
	      proxy_set_header Upgrade $http_upgrade;
    }
  }
  
Make sure you replace ssl_certificate and ssl_certificate_key

Restart Nginx:
  service nginx restart

Second step, Turn the pveproxy to localhost only. Copy this file to: /etc/default/pveproxy

  ALLOW_FROM="127.0.0.1"
  DENY_FROM="all"
  POLICY="allow"
  
You can also block it over iptables, since it does not fully work anymore on 5.x.
  post-up iptables -A INPUT -p tcp --dport 8006 -s 127.0.0.0/8 -j ACCEPT #allow localhost for reverse proxy
  post-up iptables -A INPUT -p tcp --dport 8006 -j DROP #webinterface
  post-up iptables -A INPUT -p tcp --dport 3128 -j DROP #spiceproxy

  
Restart pveproxy:
  service pveproxy restart

https://YOURIP:8006 should be not more reachable.