Troubleshooting Immich Performance Issues on Proxmox

Troubleshooting Immich Performance Issues on Proxmox

I recently installed Immich on my Proxmox VE server to replace commercial cloud photo storage with a self-hosted solution. Based on homelab discussions and benchmarks across the community, Immich generally performs exceptionally well on older server hardware, such as a Dell PowerEdge T20 or modest quad-core enterprise workstations.

Right out of the gate, however, performance and stability became a major blocker. The issue did not manifest during idle operation or single-file test uploads. Instead, it struck the moment I connected my mobile device over local Wi-Fi and initiated a full camera roll backup.

After syncing anywhere between 20 and 50 high-resolution photos, my entire Proxmox hypervisor became completely unresponsive:

• The Proxmox Web GUI timed out and refused incoming connections.
• SSH sessions to the Proxmox host dropped immediately.
• SSH and network traffic to all neighboring LXC containers and virtual machines ceased.
• Pings to the hypervisor management IP (192.168.x.x) returned Destination Host Unreachable.

To all external observations, the physical host had suffered a catastrophic kernel panic or complete system crash. Every incident forced me to walk over to the rack and perform a hard physical power cycle.

The Real Culprit: The Illusion of a Host Crash

Once the host rebooted, I pulled the persistent systemd journal from the failed boot session (journalctl -b -1). The logs revealed that the hypervisor kernel never actually crashed. Proxmox, the underlying Debian base, and all guest VMs were running normally in the background.

The complete network blackout was triggered by two distinct, compounding failure mechanisms at the hardware driver and kernel memory layers.

1. Hardware Unit Hang on Intel Gigabit NIC (e1000e)

The physical onboard network interface on the Dell PowerEdge motherboard is an Intel Gigabit controller (such as the Intel I217-LM or 82579LM) managed by the standard Linux e1000e kernel driver module. During sustained multi-stream TCP transfers from the Immich mobile client, the controller choked:

Oct 13 19:24:24 home kernel: e1000e 0000:00:19.0 nic0: Detected Hardware Unit Hang:
Oct 13 19:24:24 home kernel:   TDH <0x12>
Oct 13 19:24:24 home kernel:   TDT <0x14>
Oct 13 19:24:24 home kernel:   next_to_clean <0x12>
Oct 13 19:24:24 home kernel:   next_to_use <0x14>
Oct 13 19:24:26 home kernel: e1000e 0000:00:19.0 nic0: Detected Hardware Unit Hang:
Oct 13 19:24:32 home pvestatd[1234]: storage 'photos' is not online

Modern Linux network stacks offload packet segmentation to hardware via TCP Segmentation Offload (TSO) and Generic Segmentation Offload (GSO). When Immich accepts hundreds of raw multi-megabyte photo payloads, the kernel transmits large 64KB aggregated chunks directly to the NIC.

On legacy Intel silicon handled by e1000e, rapid burst traffic causes a race condition in the transmit descriptor ring buffer. The hardware transmit unit locks up completely, dropping the physical interface (nic0). Because Proxmox bridges all guest and management traffic through vmbr0 on top of nic0, losing the physical link instantly severs network connectivity for the entire server.

2. Kernel Memory Starvation (kworker GFP_NOIO Failure)

Compounding the NIC failure, the sudden flood of network I/O combined with rapid disk caching caused emergency memory allocation deadlocks in the kernel:

Oct 11 05:17:02 home kernel: Out of memory and no killable processes...
Oct 11 05:17:02 home kernel: kworker/u16:12 invoked oom-killer: gfp_mask=0x101c0a(GFP_NOIO|__GFP_HIGHMEM|__GFP_MOVABLE|__GFP_WRITE|__GFP_HARDWALL), order=1

Linux default configuration assigns a relatively small vm.min_free_kbytes reserve. When Immich ingested bursts of high-bandwidth image data into host page cache, kernel background workers (kworker) attempted to allocate contiguous atomic memory pages (order=1).

Because user workloads (VMs, containers, ZFS/ext4 ARC, and page cache) occupied nearly all available physical RAM, the kernel had no contiguous free pages left. Because the driver allocation specified GFP_NOIO, the kernel could not initiate disk I/O to flush caches, resulting in an OOM invocation with no eligible user processes to kill.

Stack & Prerequisites

Before applying the fixes, verify your hardware, kernel, and software environment:

• Hypervisor: Proxmox VE 8.x running Debian 12 Bookworm (Linux kernel 6.x or newer).
• Physical Network Interface: Onboard Intel Gigabit NIC driven by the e1000e kernel module (e.g., Intel I217-LM, I218, I219, or 82579LM).
• Application Stack: Immich running inside Docker Compose hosted in a guest Debian VM (deb-photos, VM 100) or an LXC container.
• System Utilities: ethtool, sysctl, iproute2, and root console or SSH access to the Proxmox host.

Step-by-Step Implementation

To restore permanent stability, I implemented a two-part host remediation strategy: disabling buggy hardware offloads on the physical NIC and expanding the kernel emergency memory watermark.

Step 1: Diagnose Prior Boot Logs

If your Proxmox server appears frozen during large file uploads, do not assume the hypervisor crashed. After cycling power, inspect the journal from the previous boot to confirm whether e1000e or memory deadlocks triggered the outage:

# Inspect the previous boot journal for e1000e unit hangs
journalctl -b -1 -k | grep -iE "e1000e.*hang"

# Check for atomic memory allocation failures
journalctl -b -1 -k | grep -iE "out of memory|kworker.*invoked oom-killer"

If you observe Detected Hardware Unit Hang referencing your physical interface (nic0 or eno1), proceed with the remediation below.

Step 2: Disable Hardware Offloads on the Physical NIC

Disabling TCP Segmentation Offload (TSO), Generic Segmentation Offload (GSO), and Generic Receive Offload (GRO) forces the Linux kernel CPU to handle packet segmentation rather than delegating it to the faulty NIC descriptor ring. The modern CPU overhead is negligible on a Gigabit network, and it completely eliminates the hardware hang.

1. Apply at Runtime for Immediate Testing

Run ethtool against the physical interface (substitute nic0 with your physical interface name, such as eno1):

# Disable hardware segmentation offloads on the physical network interface
ethtool -K nic0 tso off gso off gro off

2. Make the Offload Settings Persistent

In Proxmox VE, network interfaces are managed in /etc/network/interfaces. Add a post-up hook to the physical interface definition so that offloads are disabled whenever the interface initializes.

Open /etc/network/interfaces in your editor:

nano /etc/network/interfaces

Locate your physical NIC block (nic0 or eno1) and append the post-up directive:

# Physical Intel Gigabit interface connected to your switch
iface nic0 inet manual
        # Disable hardware segmentation offload to prevent e1000e unit hangs
        post-up /usr/sbin/ethtool -K nic0 tso off gso off gro off

# Proxmox standard bridge interface
auto vmbr0
iface vmbr0 inet static
        address 192.168.x.x/24
        gateway 192.168.x.1
        bridge-ports nic0
        bridge-stp off
        bridge-fd 0

3. Verify Active Offload States

Confirm that the segmentation offloads are successfully turned off:

# Inspect offload status on the physical interface
ethtool -k nic0 | grep -E "tcp-segmentation|generic-segmentation|generic-receive"

Expected output:

tcp-segmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off

Step 3: Increase Kernel Memory Watermarks (sysctl)

To prevent kernel workers from running out of contiguous pages during heavy burst I/O, increase vm.min_free_kbytes from the default (often 64MB) to 512MB (524288 KB), and reduce vm.swappiness to 10.

1. Create a Dedicated sysctl Configuration File

Create a drop-in configuration file in /etc/sysctl.d/:

# Path: /etc/sysctl.d/99-pve-memory.conf
cat <<'EOF' > /etc/sysctl.d/99-pve-memory.conf
# Reserve a 512MB contiguous page buffer to prevent kworker allocation deadlocks
vm.min_free_kbytes = 524288

# Reduce swappiness to favor retaining application memory over aggressive paging
vm.swappiness = 10
EOF

2. Apply Configuration Immediately

Reload sysctl parameters without rebooting:

# Apply new sysctl directives immediately
sysctl -p /etc/sysctl.d/99-pve-memory.conf

3. Verify Memory Watermark

Verify that the kernel accepted the new watermark:

# Check the active min_free_kbytes parameter
sysctl vm.min_free_kbytes

Output:

vm.min_free_kbytes = 524288

Step 4: Tune Immich Concurrency and Resource Allocation

In addition to host-level fixes, fine-tuning Immich job queues prevents thumbnail generation, facial recognition, and CLIP indexing from starving guest VM memory while millions of bytes are streaming over the socket.

1. In the Immich web interface, log in as an administrator.
2. Navigate to Administration > Settings > Job Settings.
3. Under Machine Learning and Thumbnail Generation, reduce concurrency to 1 or 2 threads if running on quad-core CPUs.
4. If deploying Immich via Docker Compose inside a dedicated VM or container, establish explicit memory and CPU constraints:

# Path: docker-compose.yml (Immich services snippet)
services:
  immich-server:
    image: ghcr.io/immich-app/immich-server:release
    # Limit process memory consumption to avoid ballooning guest page cache
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4096M

  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:release
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 2048M

Gotchas & Edge Cases

When resolving network freezes in virtualized environments, keep the following operational caveats in mind:

1. The Virtual Bridge vs. Physical Slave Pitfall

A common mistake is placing the ethtool directive on vmbr0 in /etc/network/interfaces. Because vmbr0 is a software construct, disabling offloads on the bridge does not propagate down to the physical silicon descriptor ring on nic0. The command must run directly against the raw interface.

2. MTU Mismatches

Ensure your physical interface and bridge are configured for standard 1500 MTU. If you attempt to enable 9000 MTU jumbo frames on an Intel e1000e controller experiencing ring buffer pressure, the hardware hangs become even more frequent unless every switch and endpoint on the VLAN is meticulously tuned.

3. Alternative Kernel Module Parameters (IntMode)

On select hardware revisions (such as older 82579LM chips), Intel controllers may continue to encounter hangs due to MSI-X interrupt delivery bugs. If disabling TSO/GSO does not fully eliminate the problem on your specific board, configure the module option in /etc/modprobe.d/e1000e.conf:

# Force legacy or MSI interrupts if MSI-X causes interrupt throttling lockups
options e1000e IntMode=1,1

After modifying modprobe options, regenerate your initramfs (update-initramfs -u) and reboot.

Verification & Operational Benchmarks

To validate the remediation, I monitored kernel ring buffer events in real time while initiating an aggressive sync of over 2,500 photos and 4K video clips from my mobile client.

Real-Time Monitoring Command

While the mobile backup ran, I monitored the live Proxmox host kernel console over SSH:

# Stream live kernel messages filtered for hardware and memory alerts
dmesg -w | grep -iE "e1000e|oom|hung|page allocation"

Telemetry & Stability Comparison

Conclusion & DenverNerd Homelab Checklist

When a Proxmox hypervisor becomes unreachable during heavy network I/O, do not jump to the conclusion that the CPU panicked, RAM died, or the power supply tripped. Virtualized media applications like Immich create unique burst transfer profiles that easily expose subtle firmware and driver limitations on older onboard NICs.

Before replacing hardware or migrating workloads, apply this quick homelab checklist:

1. Check persistent logs from the crashed boot: journalctl -b -1 -k.
2. Disable TSO, GSO, and GRO on physical interfaces driven by e1000e.
3. Reserve sufficient emergency kernel memory headroom via vm.min_free_kbytes = 524288.
4. Throttle heavy background worker queues in Immich during large initial ingestions.

With these three adjustments in place, my Dell PowerEdge T20 now processes massive photo backups without dropping a single packet or locking up the hypervisor.