Self-hosted VPN on a VPS: which protocol to choose in 2026

Which protocol to run on your own VPN in 2026: WireGuard, AmneziaWG, VLESS Reality, Hysteria2, Outline or MTProto — how DPI detects each one and what to pick.

Every modern VPN protocol encrypts traffic well enough, so encryption is not what sets them apart. What matters is how a connection looks from the outside. Russian ISPs are required to run TSPU, a deep packet inspection system, and Russia has the most detailed public record of what gets blocked, so the examples below come from there. Iran and China rely on similar techniques. The protocol that survives is the one a censor cannot tell apart from ordinary traffic, or does not dare to block. Below: how DPI finds a VPN, what that means for each protocol, and which setup fits which job. Each section links to a full guide.

The short answer

Personal VPN for you and your family — AmneziaWG. The AmneziaVPN app installs the server itself from an IP address and root password, and there are clients for every platform.
Mobile networks and strict ISPs — VLESS + Reality over TCP 443 on your own domain: to DPI it looks like TLS 1.3 to an ordinary website.
Dozens or hundreds of users, subscriptions, several locations — the Remnawave panel with nodes. Keep Marzban for installations that already run it.
Lossy, high-latency links — Hysteria2 as a second profile, as long as the network lets UDP through.
Telegram only — an MTProto proxy: users install nothing, the proxy connects from a link.
Handing out keys fast, without a panel — Outline, though in Russia it is unreliable.
Plain WireGuard and OpenVPN do not connect from Russia: TSPU recognizes them by their first packets. They are still a good fit for linking your own servers and for access from countries without this kind of filtering.

How DPI finds a VPN

Blocking works in layers, and each layer needs its own countermeasure:

Protocol signatures. The first packets of WireGuard and OpenVPN have fixed sizes and fixed header fields, so one or two packets are enough to identify them. Russian ISPs have been blocking both at scale since August 2023.
Behaviour inside the tunnel. Protocols that look like random noise from the first byte, such as Shadowsocks and VMess, have been detected in Russia since spring 2024 by packet sizes and ordering: the TLS handshake of the website the user opens shows through the tunnel. China has blocked this kind of fully encrypted traffic since November 2021.
The site name. The SNI in a TLS ClientHello and in the first QUIC packet is checked against a registry of banned domains. TSPU decrypts that first QUIC packet: its keys are derived from fields sent in the clear.
Limits on hosting networks. Since June 2025, many Russian ISPs stall TCP connections to large hosting and CDN networks after 15–20 KB of data, whatever protocol runs inside. Since November 2025, on some networks a burst of near-simultaneous TLS handshakes to one server freezes new connections for a couple of minutes.
IP bans. Addresses of VPN servers are banned in waves — in 2026, for example, on 4 August and 21 September. The only fix is a new address.
Allowlists. Since autumn 2025 in some Russian regions, and since March 2026 in Moscow too, mobile internet is periodically switched to a mode where only approved addresses work. A server abroad is then unreachable over any protocol.

Tip: ForestsNet Cloud IP addresses are checked from Russia every day for TSPU blocking over TCP and TLS, and blocked ones are not handed out. That does not help against protocol-level blocking: plain WireGuard from Russia does not connect to our servers either, so use a protocol with obfuscation.

Important: on ForestsNet Cloud servers you may not pose as someone else's website, that is, put a domain you do not own in the SNI, in forwarding settings (dest, fallback) or in a TLS certificate. Your own domain and your own web server, including one on the same server, are fine. These settings are checked automatically, and if something is wrong, support will ask you to fix it. Details are in the Third-Party Domains section of our Terms of Service.

WireGuard and OpenVPN: recognized by the handshake

WireGuard opens a connection with a packet of exactly 148 bytes: the first byte is message type 1, followed by three zero bytes. The server replies with 92 bytes of type 2. Seeing that pair is enough for DPI, and encryption does not help: what gives it away is the shape of the packets, not their contents. OpenVPN is recognizable by the opcode at the start of each packet and the sizes of its control exchange. Running it over TCP 443 does not hide it: it is OpenVPN's own protocol, not the TLS a censor expects to see on that port.

Both are fine wherever there is no DPI: tunnels between your own servers, access to a home or office network, staff working from Europe. For connections from Russia, use AmneziaWG — the same speed and design with a different packet shape.

Guides: WireGuard VPN: server and clients and OpenVPN server.

AmneziaWG: WireGuard in disguise

AmneziaWG is a WireGuard fork that changes exactly what WireGuard is recognized by:

Junk before the handshake. The client sends a burst of UDP packets of random size (the Jc, Jmin and Jmax parameters), so the stream does not start where DPI expects it to.
Different sizes and headers. Handshake messages get random padding (S1–S4) and the message type numbers are replaced (H1–H4), so a 148-byte packet of type 1 never appears.
Imitation packets. Newer protocol versions can send packets of a chosen shape before the handshake (I1–I5), for example one that looks like a DNS response.
Header protection. From version 3, the header fields are encrypted with a server key, and rekey and keepalive intervals are picked from ranges, which removes WireGuard's timing signature as well.

Junk and imitation packets are needed on the client only, while padding, headers and the header protection key must match on both sides. A stock WireGuard client will not work: use the AmneziaVPN or AmneziaWG apps, or another client that supports the same protocol version.

The easiest way to set up the server is from the AmneziaVPN app. You enter the IP address and root password, and the app connects over SSH, installs Docker, starts an AmneziaWG container on a random UDP port between 30000 and 49999 and generates the obfuscation parameters itself. Along the way it stops the server from answering ping and changes a few kernel settings — that is by design, not a fault.

The weak spot is UDP. Some mobile carriers cut UDP to foreign addresses, or QUIC altogether, and then no amount of obfuscation helps. Keep a TCP-based profile next to it.

Guide: Amnezia VPN on your own server.

VLESS + Reality: a tunnel disguised as a website

VLESS is a lightweight protocol of the Xray core, and Reality hides it inside TLS 1.3. The server answers the handshake on behalf of a target website and recognizes its own clients by a key hidden inside that same handshake. Everyone else, active probes included, is forwarded to the target and sees an ordinary site. XTLS Vision removes one more VPN giveaway: a TLS handshake nested inside the tunnel.

According to reports on ntc.party and write-ups on Habr, TSPU did not recognize the protocol itself in 2025–2026, which is why Reality is the main option for mobile networks. It is blocked indirectly: by the server's address, by an implausible "hosting IP plus popular site's SNI" pair, by traffic volume and by the number of connections. Hence the key setup rule: the target is your own website on the same server. A domain with an A record pointing at the server, a web server on 127.0.0.1 with a Let's Encrypt certificate, and that domain in the SNI: the name honestly points at the address the site runs on. Using someone else's site as the target is not allowed on our servers under the Terms of Service: popular SNIs like www.microsoft.com on hosting addresses lead to IP bans, and a ban hits every customer on the network. Against freezes triggered by connection bursts, add a second inbound on the XHTTP transport, which packs an app's connections into a few TLS connections.

Important: starting with Xray v26.9.8, Reality rejects clients that do not offer the X25519MLKEM768 key exchange. As of late September 2026, apps built on the sing-box core (Hiddify, Karing, NekoBox) and Shadowrocket cannot connect because of this. Before updating the core on a server or in a panel, check which apps your users have.

Guide: 3x-ui install and VLESS Reality setup.

Hysteria2: speed over QUIC

Hysteria2 runs on QUIC, the transport behind HTTP/3, and answers outsiders like an ordinary website. Its strength is speed on bad links: in Brutal mode it holds the configured rate and resends lost packets instead of slowing down on loss the way TCP does. On mobile networks, congested peering links and long routes it clearly outpaces TCP-based protocols, at the cost of more CPU.

Its weakness is again UDP. Measurements presented at FOCI 2026 show TSPU reading the domain name from the first QUIC packet and dropping connections to domains on the registry. Between March and June 2022, all QUIC to foreign addresses on port 443 was dropped outright. On mobile networks the outcome depends on the carrier. Salamander obfuscation removes the QUIC fingerprint and port hopping moves the connection across ports, but there are no guarantees.

So Hysteria2 makes a good second profile next to VLESS Reality, not the only one.

Guide: Hysteria2 server setup and config.

Shadowsocks and Outline: simple, with no guarantees

Outline is the simplest way to share access: the server installs with one command, a key is a link, and data limits are set in the manager app. Under the hood it is Shadowsocks, whose stream looks like random noise from the first byte.

That is exactly what gives it away. In Russia, Shadowsocks has been detected by indirect signs since spring 2024, and in summer 2025 Outline stopped working for many customers of the major carriers. Prefix disguises only change the first bytes of a connection and help in some places only. Outline makes sense for travel, for countries with light filtering and as a backup channel.

Guide: Outline VPN server setup on your own VPS.

MTProto proxy: Telegram only

An MTProto proxy carries the traffic of one app, Telegram, but users install nothing: the official apps connect to it from a tg://proxy link. In fake-TLS mode the connection looks like TLS to a website whose name is encoded in the secret. That should be your own domain pointing at the server, with a real website behind the proxy: the mtg author recommends it, and it is the only setup allowed on our servers.

2026 has been a hard year for MTProto in Russia. From 1 April, TSPU learned to tell the TLS handshake of Telegram apps apart; an app update fixed it. From June, proxies are blocked for a while when a client opens several connections at once. In June the author of mtg, one of the popular implementations, openly advised users in Russia to switch to a VPN. A proxy is still a handy way to give Telegram to people who will not install a VPN, but keep a proper VPN for yourself.

Guide: MTProto proxy for Telegram on your own server.

Panels: 3x-ui, Remnawave or Marzban

You can configure the Xray core with a single JSON file, but users, limits and subscriptions are easier to manage in a panel:

3x-ui — one server with a web UI: clients with data limits and expiry dates, subscriptions, several inbounds. Enough for a personal VPN or a small team.
Remnawave — a panel with separate nodes, auto-updating subscriptions, an API and integrations. It is the pick for many users across several locations, including paid VPN services.
Marzban — long the standard for multi-node setups, but its last release came out on 9 January 2025, and marzban-node has been unmaintained since August 2025. Start new projects on Remnawave; there are migration guides for existing Marzban installs.

Guides: 3x-ui, Remnawave and Marzban.

What server you need

Resources. A personal VPN for a few devices runs on 1 vCPU and 1 GB of RAM. A panel with a database needs 2 GB or more, and at high speeds Hysteria2 hits the CPU limit before TCP-based protocols do.
Traffic. ForestsNet Cloud plans count outgoing traffic only. On a VPN server almost everything is outgoing: what a user downloads is sent to them, and what they upload is sent on to the internet. Size the plan by your users' total traffic in both directions.
A separate address. An IP ban hits everything on that address. Keep websites and bots on a different server from a VPN that many people use.
Two locations. A second server in another country is a ready backup entry point: if one address is blocked, users switch to the other.

Keep a backup profile

No protocol works everywhere all the time: what one carrier lets through, another blocks, and the rules change every few months. It is safer to give users two profiles with different fingerprints:

Primary — VLESS + Reality on TCP 443.
Backup — over UDP: AmneziaWG or Hysteria2.

If you have a second server, point the backup profile at a different address, so that one IP ban does not leave users offline. Subscriptions are the easiest way to hand out both profiles: the app picks up changes by itself.

How to tell what is being blocked

Nothing works from anywhere, not even from abroad — the problem is the server: check the service, the port and the log.
Works over Wi-Fi but not over mobile data. The carrier is cutting UDP or has switched on allowlists. Switch to a TCP-based profile.
Connects, but pages do not load. Either the freeze after a few kilobytes or a banned address. To test an ISP in Russia, use the TCP 16-20 web check from hyperion-cs.
Works, then drops after a couple of minutes — a limit on the number of connections. For Reality, XHTTP helps; for MTProto, fewer parallel connections.

Important: ping proves nothing. During some bans ping gets through while data does not, and a server set up from the AmneziaVPN app does not answer ping at all.

Sources

Next →3x-ui install and VLESS Reality setup