Success probability of the Babai estimators for box-constrained integer linear models

In many applications including communications, one may encounter a linear model where the parameter vector $\hbx$ is an integer vector in a box. To estimate $\hbx$, a typical method is to solve a box-constrained integer least squares (BILS) problem. However, due to its high complexity, the box-constrained Babai integer point $\x^\sBB$ is commonly used as a suboptimal solution. In this paper, we first derive formulas for the success probability $P^\sBB$ of $\x^\sBB$ and the success probability $P^\sOB$ of the ordinary Babai integer point $\x^\sOB$ when $\hbx$ is uniformly distributed over the constraint box. Some properties of $P^\sBB$ and $P^\sOB$ and the relationship between them are studied. Then, we investigate the effects of some column permutation strategies on $¶^\sBB$. In addition to V-BLAST and SQRD, we also consider the permutation strategy involved in the LLL lattice reduction, to be referred to as LLL-P. On the one hand, we show that when the noise is relatively small, LLL-P always increases $P^\sBB$ and argue why both V-BLAST and SQRD often increase $P^\sBB$; and on the other hand, we show that when the noise is relatively large, LLL-P always decreases $P^\sBB$ and argue why both V-BLAST and SQRD often decrease $P^\sBB$. We also derive a column permutation invariant bound on $P^\sBB$, which is an upper bound and a lower bound under these two opposite conditions, respectively. Numerical results demonstrate our findings. Finally, we consider a conjecture concerning $\x^\sOB$ proposed by Ma et al. We first construct an example to show that the conjecture does not hold in general, and then show that it does hold under some conditions.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.