In the Linux kernel, the following vulnerability has been resolved:
Revert "wireguard: device: enable threaded NAPI"
This reverts commit 933466fc50a8e4eb167acbd0d8ec96a078462e9c which is
commit db9ae3b6b43c79b1ba87eea849fd65efa05b4b2e upstream.
We have had three independent production user reports in combination
with Cilium utilizing WireGuard as encryption underneath that k8s Pod
E/W traffic to certain peer nodes fully stalled. The situation appears
as follows:
- Occurs very rarely but at random times under heavy networking load.
- Once the issue triggers the decryption side stops working completely
for that WireGuard peer, other peers keep working fine. The stall
happens also for newly initiated connections towards that particular
WireGuard peer.
- Only the decryption side is affected, never the encryption side.
- Once it triggers, it never recovers and remains in this state,
the CPU/mem on that node looks normal, no leak, busy loop or crash.
- bpftrace on the affected system shows that wg_prev_queue_enqueue
fails, thus the MAX_QUEUED_PACKETS (1024 skbs!) for the peer's
rx_queue is reached.
- Also, bpftrace shows that wg_packet_rx_poll for that peer is never
called again after reaching this state for that peer. For other
peers wg_packet_rx_poll does get called normally.
- Commit db9ae3b ("wireguard: device: enable threaded NAPI")
switched WireGuard to threaded NAPI by default. The default has
not been changed for triggering the issue, neither did CPU
hotplugging occur (i.e. 5bd8de2 ("wireguard: queueing: always
return valid online CPU in wg_cpumask_choose_online()")).
- The issue has been observed with stable kernels of v5.15 as well as
v6.1. It was reported to us that v5.10 stable is working fine, and
no report on v6.6 stable either (somewhat related discussion in [0]
though).
- In the WireGuard driver the only material difference between v5.10
stable and v5.15 stable is the switch to threaded NAPI by default.
[0] https://lore.kernel.org/netdev/CA+wXwBTT74RErDGAnj98PqS=wvdh8eM1pi4q6tTdExtjnokKqA@mail.gmail.com/
Breakdown of the problem:
1) skbs arriving for decryption are enqueued to the peer->rx_queue in
wg_packet_consume_data via wg_queue_enqueue_per_device_and_peer.
2) The latter only moves the skb into the MPSC peer queue if it does
not surpass MAX_QUEUED_PACKETS (1024) which is kept track in an
atomic counter via wg_prev_queue_enqueue.
3) In case enqueueing was successful, the skb is also queued up
in the device queue, round-robin picks a next online CPU, and
schedules the decryption worker.
4) The wg_packet_decrypt_worker, once scheduled, picks these up
from the queue, decrypts the packets and once done calls into
wg_queue_enqueue_per_peer_rx.
5) The latter updates the state to PACKET_STATE_CRYPTED on success
and calls napi_schedule on the per peer->napi instance.
6) NAPI then polls via wg_packet_rx_poll. wg_prev_queue_peek checks
on the peer->rx_queue. It will wg_prev_queue_dequeue if the
queue->peeked skb was not cached yet, or just return the latter
otherwise. (wg_prev_queue_drop_peeked later clears the cache.)
7) From an ordering perspective, the peer->rx_queue has skbs in order
while the device queue with the per-CPU worker threads from a
global ordering PoV can finish the decryption and signal the skb
PACKET_STATE_CRYPTED out of order.
8) A situation can be observed that the first packet coming in will
be stuck waiting for the decryption worker to be scheduled for
a longer time when the system is under pressure.
9) While this is the case, the other CPUs in the meantime finish
decryption and call into napi_schedule.
10) Now in wg_packet_rx_poll it picks up the first in-order skb
from the peer->rx_queue and sees that its state is still
PACKET_STATE_UNCRYPTED. The NAPI poll routine then exits e
---truncated---
What the EG score is made of (scorer v4.0: the three questions the 0–10 score mixes, published apart · methodology)
The EG score (7.5): severity, escalated by confirmed exploitation. This CVE's 7.5 is its impact severity (EG Severity), not escalated.
- EG Severity
- 7.5 HIGHCVSS-B, impact only: no KEV listing, EPSS or EG-KEV tier raises it.GitHub Security Advisory CVSS
- EG Likelihood
- 0.2% within 30 daysSource: FIRST EPSS. FIRST's exploitation probability, not an EchelonGraph model.
- Confirmed exploitation
- None on record (CISA KEV, EG-KEV Tier 1)Reported apart from the likelihood: an observed fact, not a probability.
- EG Priority
- SSVC Track at medium mission impactTrack at low or medium mission impact; Attend at high (mission-essential systems).CISA BOD 26-04: 14 days if the affected system is publicly exposed; 60 days if it is not.Each input and its source, and whom BOD 26-04 binds: the SSVC and BOD 26-04 blocks below.
SSVC (CISA's decision table, applied by EchelonGraph)Track at low or medium mission impact; Attend at high (mission-essential systems).
A fix is available. Apply it within your standard update timelines at low or medium mission impact and sooner than that at high.
Exploitation none (CISA Vulnrichment) · Automatable yes (CISA Vulnrichment) · Technical impact partial (CISA Vulnrichment). All three are CISA's SSVC values (Vulnrichment). EchelonGraph applied CISA's decision table to them; CISA publishes SSVC inputs and the table, not a decision for each CVE. Mission impact is CISA's Mission & Well-being decision point, and only you can judge it. CISA's table makes it high when Mission Prevalence is Essential — the vulnerable component "directly provides capabilities that constitute at least one MEF for at least one entity" (MEF: mission essential function) — or when Public Well-Being Impact is Irreversible: "multiple fatalities are likely", the cyber-physical system "is likely lost or destroyed", "extreme or serious externalities" are imposed on other parties, or social systems such as elections or the financial grid "are destabilized and potentially collapse". CISA's decision table
BOD 26-04 (CISA's remediation timeline, Table 1)14 days if the affected system is publicly exposed; 60 days if it is not.
The score's inputs: In CISA KEV no (CISA's KEV catalog) · Automatable yes (CISA Vulnrichment) · Technical impact partial (CISA Vulnrichment). Whether your system is publicly exposed is yours to answer. BOD 26-04 binds US Federal Civilian Executive Branch agencies; for anyone else it is CISA's published timeline, a reference rather than an obligation. The directive · how EchelonGraph applies it