FreqCache: Frequency-Aware Adaptive Branch Routing for Training-Free Diffusion Acceleration

Diffusion models have achieved remarkable success in image generation, but their iterative denoising process requires repeated evaluations of large neural networks, resulting in high inference latency. Recent training-free acceleration methods, such as DeepCache, exploit temporal redundancy in U-Net features by caching and reusing high-level representations across adjacent denoising steps. However, existing caching strategies usually adopt a static skip-branch selection throughout the sampling trajectory, ignoring the stage-dependent frequency evolution of diffusion sampling. In this paper, we propose an interval-guided adaptive branch routing strategy to improve training-free feature reuse. Motivated by the observation that low-frequency global structures change rapidly in early denoising stages while high-frequency details dominate later refinement, our method dynamically adjusts the skip branch according to the timestep. It preserves deeper computation in early stages for semantic reconstruction and progressively shifts to shallower branches in later stages to reduce redundant computation while maintaining fine-grained details. The proposed method requires no retraining and can be directly applied to pretrained U-Net-based diffusion models. Experiments show that FreqCache achieves up to 1.93× speedup on CIFAR-10, 1.50× speedup on LSUN-Bedroom and LSUN-Churches, and 10.60× speedup on ImageNet 256 × 256 compared with the baseline, while maintaining an Fréchet Inception Distance (FID) score comparable to or slightly better than DeepCache at the same cache interval.