Improved Driver’s Safety with Mixed Reality and Haptic Solutions
By Nokia
INTRODUCTION
This use case focuses on improving driver safety by combining mixed reality (MR) and haptic solutions in the real scenario of the port of Valencia. The objective is to achieve remote control in the transport of goods with Automated Guided Vehicles (AGVs) thanks to different solutions.The technologies that are combined to achieve the proposed objective are the tactile internet, edge computing and immersive enablers such as Mixed-Reality (MR) engines and haptic gloves.
The two pillars on which this use case is based are telepresence of the operator to know the environment in which the vehicle moves and interaction with the operator to control it. To support the telepresence, several 360º video cameras with low latency have been installed both in the AGVs and in the road vehicles. In the case of the interaction for vehicle’s control, it is achieved with the implementation of haptic gloves that allow the sensors to be managed remotely and to obtain feedback from them in real time.
Potential benefits that can be achieved in this scenario include, automatic handling of assets, human-machine iteration by working remotely in unexpected circumstances and scalability (e.g., working remotely in multiple sites governed by AGVs).
ARCHITECTURE
The proposed architecture consists of several elements that are interconnected by different fiber and mobile networks in the 4G and 5G bands depending on their needs. The main scheme of this architecture is as follows:
This scenario can be broken down into two different environments: that of the operator who has control (indoor) and that of the vehicle (outdoor).
On one hand, the operator’s side is composed of all the devices that allow him to have a two-way communication with the vehicle for its correct control. These components are:
- An augmented reality glasses to which the 360º video captured by the cameras arrives with information about the different sensors superimposed. All this information is obtained in real time in such a way that it is useful for driving.
- Sensor straps and haptic gloves to capture the operator’s movement and send simple, pre-recorded commands to the vehicle (e.g., start/stop)
- A steering wheel and pedals to, if necessary (e.g., an obstacle is detected), take control of the vehicle and drive it remotely.
On the other hand, the exterior stage consists of the necessary elements for communication and the vehicles themselves:
- The different antennas and modems placed on the stage establish a connection between the different components. For it, a network type is chosen between fiber, 4G, 5G and mmW, depending on the demand in terms of latency and quality that is established for each type of communication.
- The AGVs that move around the stage are equipped with the needed elements to communicate and drive correctly:
- Mixed reality 180º video cameras: send video about the surroundings of the AGV to the operator so that, in case he has to take control, he can drive it without incidents.
- Proximity sensors: allow the AGV to detect obstacles in its path.
- 5G & mmW modem: uses the networks to send the information captured by the AGV and receive orders from the operator.
FIELD TESTS IN VALENCIA PORT
The video shown below summarizes the different tests that were carried out in the port of Valencia for this use case.
