In 2018 26th European Signal Processing Conference (EUSIPCO), pages 156-160, Sep., 2018. Paper doi abstract bibtex
In modern video codecs, such as HEVC and VP9, intra-frame blocks are decorrelated using DCT or DST. The optimal separable transform for intra prediction residual blocks has been determined to be a hybrid transform composed of the DCT and the odd type-3 DST (ODST-3), independent of block size. However, the ODST-3 has no fast algorithm like the DCT. Hence its use in HEVC and VP9 has been limited to only 4×4 blocks. For larger blocks such as 8×8 or 16×16, HEVC replaces the optimal ODST-3 with the conventional DCT while VP9 replaces it with the even type-3 DST (EDST-3), both of which have fast algorithms. The EDST-3 has better coding gain than the DCT but it has still a coding gain loss with respect to the optimal ODST-3. This paper attempts to optimize some parameters of the EDST-3 to reduce this coding gain loss while still retaining the fast algorithm. In particular, the 8-point EDST-3 is represented as a cascade of Givens rotations and some rotation angles are optimized to reduce the coding gain loss with respect to the optimal 8-point ODST-3. By replacing only the 8-point EDST-3 in VP9 with this optimized transform, while leaving other transforms with different sizes unchanged, average Bjontegaard-Delta bitrate savings of -0.13% are achieved with respect to the standard VP9 codec.