I saw an analysis about 7 years ago (Boulder startup week) that adequate spatial resolution, fps refresh and tracker response would require about 100 teraops of local processing power in the display device. At that time it was just one percent of that power. Apples family of top end ARM devices are closing that.

Another bottleneck was that standard graphics pipeline in GPU had too much latency to keep up with user movements.