Troubleshooting Immich Performance Issues on Proxmox: My Experience

I recently embarked on a journey to enhance my photo management workflow by installing Immich on my Proxmox server. Excited about the benefits of having a dedicated application for managing my photo uploads, I encountered unforeseen performance issues that left me baffled. Many articles suggested that Immich could run smoothly even on older hardware, but as I soon discovered, the reality was quite different.

The Issue

A few weeks into using Immich, I faced a significant roadblock. While attempting to back up and upload photos from my mobile device, the application would crash frequently. This led to an inability to SSH into my main Proxmox server, as well as restricted access to any of my LXC containers or virtual machines. It felt as if the entire host had crashed, which was particularly frustrating given my expectations of a seamless experience.

Identifying the Root Causes

Determined to resolve these issues, I delved deep into logs and system telemetry to pinpoint the causes. The logs revealed that the Intel Gigabit controller (e1000e) had encountered a hardware unit hang during periods of sustained TCP transfer bursts. This issue led to the network interface dropping out entirely while the operating system and hypervisor kernel continued to run beneath—illustrating a classic case of hardware bottlenecking during heavy I/O.

Alongside the NIC hang, I discovered kernel memory starvation caused by high memory utilization during these bursts. The kernel workers (kworker) faced out-of-memory (OOM) errors due to insufficient contiguous low-memory page pools, particularly when trying to allocate memory during heavy networking conditions.

Remediation Steps

To effectively tackle the performance challenges, I implemented two main remedial strategies:

1. Disable Hardware Offloads on the NIC

To prevent the Intel e1000e controller from hanging during intense data transfers, I disabled TCP segmentation offload (TSO) and generic segmentation offload (GSO) on the physical NIC with the following commands:

Runtime Application:

ethtool -K nic0 tso off gso off gro off

Persistent Configuration: I ensured these settings would persist across reboots by modifying the network interface configuration:

iface nic0 inet manual
    post-up /usr/sbin/ethtool -K nic0 tso off gso off gro off

2. Increase Kernel Memory Watermark

To guarantee that there were sufficient emergency reserve pages for kernel workers during I/O bursts, I updated the kernel's memory management parameters:

Persistent Configuration: I created a sysctl configuration file:

# Reserve 512MB memory buffer to avoid kworker allocation deadlocks
vm.min_free_kbytes = 524288
vm.swappiness = 10

Immediate Application:

sysctl -p /etc/sysctl.d/99-pve-memory.conf

Monitoring and Future Considerations

To maintain oversight on the operational health of my Proxmox host and the Immich application, I began employing these commands for ongoing monitoring:

  • Monitor live kernel events:
dmesg -w | grep -iE "e1000e|oom|hung"

Conclusion

By implementing these adjustments to my Proxmox server, I successfully stabilized the performance issues I was facing with Immich. This experience underscored the importance of understanding the intricate hardware interactions in a virtualized environment. If you're considering using Immich on your Proxmox setup or if you're facing similar issues, I hope this post offers valuable insights into troubleshooting and remediation strategies.

Feel free to share your experiences or additional tips in the comments below!