Effect of self-invertible rhotrix on cipher pentagraphic polyfunction

Abstract

Traditional cryptography has long relied on matrices for encryption transformations, our work takes a different path by exploring the use of rhotrix, an extension of matrices that offers extra structural flexibility. Recent research has shown vulnerabilities in some self-invertible matrices, particularly those of dimensions 4 × 4 and 6 × 6. To enhance the security of systems such as the
Cipher Tetragraphic Trifunction and Cipher Hexagraphic Polyfunction, these matrices should be strictly avoided. Building on this knowledge, our study seeks to identify patterns in self-invertible, rhotrix-based encryption keys and evaluate their impact on the security of Cipher Pentagraphic Polyfunction transformations. To this end, we derive solutions for self-invertible rhotrix keys that
satisfy transformation conditions analogous to L2 5×5 ≡ A5×5 (mod N ), where A5×5 are diagonal and symmetric rhotrices. By implementing these rhotrix keys, we analyze their structural influence on the security of Cipher Pentagraphic Polyfunction transformations, aiming to contribute a new perspective to cryptographic design that is both theoretically sound and practically robust.