2009 International Symposium on Nonlinear Theory and its Applications

NOLTA'09, Sapporo, Japan, October 18-21, 2009

Compact matrix form of the d-dimensional tensor decomposition

Ivan Oseledets†

†Institute of numerical mathematics, 119333, Moscow, Gubkina 8.

Email: [email protected]

Abstract—We present a new tensor decomposition, 2. Computing compression ranks

which is nonrecursive, does not suffer from the curse of
dimensionality, but has a viable stability properties and can Compression ranks are computable and their computa-
be computed by a robust algorithm via a sequence of SVD tion is reduced to the estimation of the ranks of the special
decompositions. The new form gives a clear and conve- unfolding matrices of a tensor, defined as
nient way to implement all basic operations efficiently. A
Ak = A(i1 , i2 , . . . , ik ; ik+1 , . . . , id ),
fast recompression procedure is presented, as well as basic
linear algebra operations. i.e. the first k indices enumerate the rows of Ak and the last
(d − k) enumerate its columns. Then the following theorem
1. Introduction holds:
Theorem 1. There exists a decomposition of form (2) with
A new decomposition for multiway arrays is presented,
which generalizes the singular value decomposition of a rk = rank Ak .
matrix, inherits its “good” properties (for example, exis-
tence of best approximation with fixed rank) stability with
respect to perturbations and allows to use well established The compression ranks are bounded from above by the
algorithms for the SVD to compute this new decomposi- canonical rank of a tensor.
tion. The number of parameters is linear in the dimension Moreover, instead of the canonical rank R we can take
d and it is not recursive and has a very simple form. Instead so-called effective tensor rank. A [2, 7].
of the canonical decomposition (known also as CANDE- Using theorems 1 and 2, the following estimate is ob-
COMP/PARAFAC model) [4, 3] written as tained (2)
R
X Theorem 3. If tensor A has canonical rank R, then there
A(i1 , ..., id ) = u1 (i1 , s), . . . ud (id , s) (1) exists a representation (2) with the number of parameters
s=1
(d − 2)nR2 + 2nR.
(with total number of parameters dRn), we consider the
representation of a tensor in form
Using additionaly the Tucker decomposition [6, 5], this
A(i1 , . . . , id ) =
X estimate can be improved to (d − 2)R3 + dnR.
= G1 (i1 , α1 )G2 (α1 , i2 , α2 )G3 (α2 , i3 , α3 ) . . . (2)
α1 ,...αd−1
3. Basic linear algebra operations
. . . Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id ),
As an example of how to use TT-decomposition (2), con-
where G1 has size n1 × r1 , Gd has size nd × rd , and sider the evaluation of the multidimensional contraction:
for 2 ≤ k ≤ (d − 1) Gk is a three-dimensional tensor
of size nk × rk−1 × rk . This decomposition is called TT- X
decompositions (from tensor-train decomposition) Num- W= A(i1 , i2 , . . . , id )u1 (i1 )u2 (i2 ) . . . ud (id ),
i1 ,i2 ,...,id
bers rk will be called compression ranks, and for simplicity
assume that they are all of the same order: rk ∼ r. which appears for the numerical computation of the multi-
Also (2) can be written in the equivalent matrix form: dimensional integrals. Using the matrix representation (3)
the problem is reduced to the sequence of one-dimensional
A(i1 , . . . , id ) = G(i1 1 )G(i2 2 ) . . . G(id d ) , (3) convolutions
nk
where G(i1 1 ) is 1 × r1 row, Gk(ik ) is a rk−1 × rk matrix, and G(id d )
X
Γk (αk−1 , αk ) = Gk (αk−1 , ik , αk )uk (ik ),
is rd−1 × 1 column. ik =1

- 388 -
and to the evaluation of the product shown that this simple to implement algorithm gives an ex-
act QR-decomposition of the matrix Uk with a Q factor in a
v1 Γ2 . . . Γd−1 v>d , TT format. The same holds for Vk , so the final recompres-
sion algorithm works from left-to-right, successively com-
where v1 , vd are rows of the corresponding vectors.. That puting QR decompositions in the TT format, and truncated
is the problem is reduced to d matrix-by-vector multiplica- SVD decompositions.
tions. The total cost is O(dnr2 + dr2 ) also using prelimi-
nary Tucker decomposition this number can be reduced to
O(dnr + dr3 ). 5. Comparison of two formats
All basic algorithms can be developed: additions of ten- In the end, let us compare the two formats. r can be
sor, matrix-by-vector product, norm.
Canonical TT
4. Recompression in TT format Number of parameters O(dnR) O(dnr + (d − 2)r3 )
Matrix-by-vector O(dn2 R2 ) O(dn2 r2 + dr6 )
The most important procedure is the recompression pro- Addition O(dnR) O(dnr)
cedure which consists of the following. Suppose some de- Recompression O(dnR2 + d3 R3 ) O(dnr2 + dr4 )
composition of form (2) is known and we want to get an- Convolution O(dnR) O(dnr + dr3 )
other decomposition with a fewer number of parameters.
Such algorithm is absent for the canonical format and that Table 1: Format comparison.
is probably the most serious drawback of this format. For
TT decomposition the situation is perfect and such an al- much smaller that R and the new format will be more ef-
gorithm exists and it is based on the standard algorithms, fective than the old one. Moreover, the estimate for the
SVD and QR decomposition. The idea is based on the fol- recompression procedure in the canonical format is given
lowing. If some TT representation is given then for any as in the work [8], where no theoretical estimates are pre-
selected mode k it gives a skeleton (dyadic) approximation sented (for some cases the method may not converge, or
of the corresponding unfolding Ak : converge to a local minimum due to the unstable nature of
the canonical decomposition).
Ak = Uk Vk> ,
where Uk is nk × rk and Vk is nd−k × rk . For the matrix References
case the recompression consists of two steps. First, QR [1] Beylkin G., Mohlenkamp M. J. “Numerical analysis in
decompositions of Uk and Vk are computed, higher dimensions” Proc. Natl. Acad. Sci. USA, 2002,
vol. 99, No. 16, pp. 1046–10251.
Uk = Qu Ru , ; Vk = Qv Rv ,
[2] Bini D., Capovani M. “Tensor rank and border rank of
and then for a “small” rk × rk matrix Ru R>v , its truncated band Toeplitz matrices” SIAM J. Comput., 1987, vol.
singular value decomposition is computed, and that gives 2, pp. 252–258.
the truncated SVD of the initial matrix. The problem is that
the row dimensions of Uk and Vk are large and depend on d [3] Carroll J.D., Chang J.J. “Analysis of individual differ-
expontially. However, they have a special TT structure and ences in multidimensional scal- ing via n-way gener-
its QR decomposition can be computed fast in a structured alization of Eckart-Young decomposition” Psychomet-
way. Uk is represented as rica, vol. 35, 1977, pp. 283-319.
[4] Harshman R.A. “Foundations of the Parafac proce-
X
Uk (i1 , i2 , . . . , ik , αk ) = G1 (i1 , α1 ) (4)
α1 ,...,αk−1
dure: models and conditions for an explanatory multi-
modal factor analysis” UCLA Working Papers in Pho-
G2 (α1 , i2 , α2 ) . . . Gk (αk−1 , ik , αk ).
netics, vol. 16, 1970, pp. 1–84.
To compute QR decomposition, first QR decomposition of [5] Oseledets I., Savostyanov D., Tyrtyshnikov E. “Tucker
the n1 × r1 matrix U1 is computed, yielding a r1 × r1 matrix dimensionality reduction in linear time” SIAM J. Ma-
R1 , which is transfered to the second core: trix Anal. Appl., 2008, vol. 30, No. 3, pp. 939–956.
X
G02 (α01 , i1 , α2 ) = G(α1 , i1 , α2 )R(α1 , α01 ). [6] Tucker L.R. “Some mathematical notes on three-mode
α1 factor analysis”, Psychometrika, 1966, vol. 31, pp.
279–311.
Then the second core is treated as a α1 n1 × r2 matrix,
G02 (α01 i1 , α2 ), its QR decomposition is computed, the Q fac- [7] Oseledets I.V., Tyrtyshnikov E.E, “On the recursive
tor is reshaped into a new core Q2 (α01 , i1 , α02 ) and the R fac- representation of multidimensional tensors”, Doklady
tor is transfered to the right core G3 and so on. It can be RAS, vol. 427, No. 1.

- 389 -
[8] Espig M. “Effiziente Bestapproximation mittels Sum-
men von Elementartensoren in hohen Dimensionen”
Phd thesis, 2007.

- 390 -