|

When Libvirt NAT Goes Rogue: A Tale of Fedora, Firewalls, and Virtual Mayhem

Like any sensible cybersecurity professional with entirely normal hobbies, I occasionally spin up a small digital nation-state of VMs on my Fedora desktop. Usually they behave. Usually.

Then one day, without warning, they staged a coordinated rebellion.

This post chronicles how my Fedora workstation decided to reinvent network segmentation, how libvirt lost its memory of the “default” world order, and how I spent an hour unraveling a problem that absolutely should not have been this complicated.

Spoiler: NAT rules, firewalld, and iptables all had something to say—and none of it was helpful.

🚨 Symptoms: The “Absolutely Not” Networking Situation

My KVM/QEMU VMs were spinning up normally. The virtual CPUs? Happy. The virtual disks? Present.
The virtual NICs? Well… they were experiencing an existential crisis.

Inside the VMs:

  • DHCP worked (mostly)
  • DNS worked, and resolve everything normally
  • VMs could ping the host’s IP (192.168.101.51)
  • VMs could ping each other
  • But they could not ping anything on my LAN
  • And definitely not the Internet
  • And every traceroute looked like it fell off a cliff after hop 1

This made precisely zero sense… until it made all the sense. See by considering what wasn working from a networking point of view literally only comms between the host and guests, as well as guest to guest. So the bridge must be working, right?

🔎 Initial Investigation: The “You’ve Got to Be Kidding Me” Phase

First, I checked the basics.

Is IP forwarding enabled?

cat /proc/sys/net/ipv4/ip_forward

Output:

1

Forwarding: ON.
So far, so good.

What does firewalld show?

sudo firewall-cmd --get-active-zones

Initial output:

FedoraWorkstation (default)
  interfaces: enp90s0
docker
  interfaces: docker0 br-423d0aa52ca9
libvirt
  interfaces: virbr0

Alright, libvirt has its own zone. Great.

What about NAT?

sudo iptables -t nat -L POSTROUTING -n -v

The output:

146  8760 MASQUERADE  all  --  *      !br-423d0aa52ca9  172.18.0.0/16        0.0.0.0/0           
0     0 MASQUERADE  all  --  *      !docker0           172.17.0.0/16        0.0.0.0/0

Notice anything missing?

Yep. No MASQUERADE rule for 192.168.122.0/24 — the default libvirt NAT network.

This meant the VMs were basically screaming into the void.

🧨 The Big Clue: virbr0 Was DOWN

Here’s the fun part: checking virbr0 showed this at first:

ip addr show virbr0

virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> ... state DOWN
    inet 192.168.122.1/24

DOWN? No carrier?
It’s a virtual bridge—what is it waiting for, an Ethernet cable!?

But this actually turned out to be important: virbr0 was up but not functional because libvirt’s backend services weren’t working correctly.

🧩 Plot Twist: The Default Network XML Was Missing

At this point I tried to destroy and recreate the default network:

sudo virsh net-destroy default
sudo virsh net-undefine default
sudo virsh net-define /etc/libvirt/qemu/networks/default.xml

Which gave me this gem:

error: Failed to open file '/etc/libvirt/qemu/networks/default.xml': No such file or directory

The entire default libvirt network definition was missing.
Just gone.
Vanished.

Fedora decided it didn’t need it anymore, apparently.

🛠️ Rebuilding the Network from Scratch

I recreated /etc/libvirt/qemu/networks/default.xml manually with the standard libvirt NAT definition and redefined/started the network.

Now virbr0 existed again.

Still down… but progress.

🔥 The Moment of Truth: Adding Forwarding Rules

The FORWARD chain looked like this:

sudo iptables -L FORWARD -n -v

Chain FORWARD (policy DROP 2610 packets, 167K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-USER
    0     0 DOCKER-FORWARD

Nobody said “ACCEPT.”
Nobody was letting my VMs leave the house.

So I manually added the three standard libvirt rules:

sudo iptables -A FORWARD -i virbr0 -o virbr0 -j ACCEPT
sudo iptables -A FORWARD -i virbr0 -j ACCEPT
sudo iptables -A FORWARD -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

And voilà:

ACCEPT     all  --  virbr0 virbr0
ACCEPT     all  --  virbr0 *
ACCEPT     all  --  *      virbr0     ctstate RELATED,ESTABLISHED

Now the VMs had both:

  • Permission to leave
  • Permission to come back

Just like well-trained packets should.

🎉 Suddenly Everything Worked Again

Once a VM restarted and reattached, virbr0 sprang to life:

ip addr show virbr0

virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP

That LOWER_UP felt like a warm hug.

DNS resolved.
VMs could ping the LAN.
VMs could ping Google.
Existence restored.

🧠 Conclusion: Fedora, You Chaotic Neutral Beast

What happened?

  • Fedora’s firewalld + Docker nuked libvirt’s NAT rules
  • The default libvirt network vanished from disk
  • virbr0 came up but with no backing rules
  • No forwarding = no routing = sad VMs
  • Rebuilding the default network + manually adding FORWARD and MASQUERADE rules fixed everything

This was a perfect storm of:

  • firewall changes
  • Docker being Docker
  • libvirt assuming everything is fine
  • Fedora being… Fedora

 

If there’s a moral here, it’s this:
When Fedora updates drop, maybe—just maybe—we should read the changelog before casually updating to the latest release and assuming all is well with the world…

But hey, where’s the fun in that?

Written by David, Made funnier with AI.