With virtual reality video, users can explore an environment without being physically present. The immersive viewing experience offered by virtual reality is limited, however, because users cannot move forward or backward freely: they can essentially only ‘look around’ by rotating their head. (Users wear a virtual reality helmet that projects the environment onto their eyes.) This limitation is overcome in volumetric video, also known as hologram video, offering users six degrees of freedom (free forward/backward, up/down, and left/right free motion). Volumetric video can be used for education purposes and in healthcare and is at present being researched intensively. It is expected to become a typical application of 5G (and beyond) wireless communication — indeed, networked volumetric video, allowing users to be untethered, is in critical demand. Still, efficiently transmitting volumetric video data over wireless networks remains a key challenge. Zhi Liu from the University of Electro-Communications, Tokyo, and colleagues have now explored point cloud video streaming, a promising approach for representing and wirelessly transmitting 3D environments.

Point cloud video uses data recorded by several cameras located at different locations and angles around a scene. The scene is divided into volumetric elements called voxels, with each voxel having several attributes such as 3D coordinates and colour. These data form a so-called point cloud. Point cloud video streaming poses several challenges, which essentially come down to the large size of the raw video data, the high complexity of the computations involved, and the high computing power required. Liu and colleagues provide suggestions for overcoming these challenges. For example, rather than transmitting complete video content, it suffices to send only data of the field of view of a user. Also, predicting user behaviour can help to efficiently use available computer resources; such predictions can be based on machine-learning as soon as enough user view logs — needed to train machine-learning models — will become publicly available. In addition, by transmitting uncompressed video tiles (tiles that do not need to be decoded again at the user side but require greater bandwidth) in addition to the compressed video tiles, the computational burden on the user device is greatly reduced.

The researchers demonstrated the feasibility of point cloud video streaming in an actual experiment with ‘off-the-shelf’ devices (a laptop as the server and a PC as the client). The approach of Liu and colleagues uses a technique called tiling, in which a point cloud video is partitioned into cuboids referred to as tiles. Tiling increases the video streaming efficiency. Uncompressed video tiles were used to release high computational requirements. The scientists also employed an international-standard encoding scheme called MPEG Dynamic Adaptive Streaming over HTTP (MPEG DASH) and optimized its usage. Comparing original video images to images received by the client showed that the proposed streaming solution provides good quality in an efficient way.

Finally, Liu and colleagues verified the performance of their algorithm in simulations with different numbers of CPU cores and different network bandwidth conditions. More CPU cores and higher bandwidth of course increase the quality of the streaming, but the proposed scheme optimally uses bandwidth and computing resources so that ‘video freezes’ are fewest and the received video quality is the best. The researchers concluded that they have achieved satisfactory performance and envision “that the point cloud video streaming system will play a vital role in future society” and that it “opens up many exciting and critical future research directions.”