Initial thoughts on mobile AR (SDKs)

Novel augmented reality SDKs allow developers to create a new generation of innovative augmented reality games and applications for mobile devices that use Android. This new technology is able to estimate the handhelds’ pose from almost any desired 2D image or photograph.

Thanks to increasing performance of modern mobile devices, innovative operating systems and powerful third-­‐party libraries, developers are able to create a new generation of augmented reality applications. Such apps are able to locate their position and orientation (pose) in space, using cutting edge computer vision algorithms. This way, virtual information can easily be integrated into a real environment in real time, which is known as mobile Augmented Reality (AR).

One of the first mobile AR applications was Layar. It used integrated sensors, like GPS, compass and accelerometer, to determine the device approximate position and orientation. In the following, virtual information about the environment was integrated in the captured video image.

Another approach to augment reality uses computer vision algorithms. Instead of using sensors, it determines the cameras’ pose using a visual reference, which is usually a 2D image (marker). One popular application that proposes such an approach was ARToolkit. ARToolKit was early ported to Android (NyArtoolKit), but has never been really successful due to bad performance and the inflexible marker design.

Today, a novel generation of computer vision algorithms are able to estimate cameras’ pose from natural features inside the environment in real time. This means, that almost any 2D image or photograph can be used as a reference, if it contains a minimum amount of visual information (features). Therefore, almost any 2D image can be used as a reference marker to allow the tracking algorithm to extract the camera pose in real time on a mobile phone.

In October 2010, Qualcomm released a free SDK (QCAR) for Android powered devices, which uses such advanced visual tracking algorithms. The SDK can be used in combination with the Android SDK or NDK. There exists also a plugin for Unity3D. Using these tools developers can create mobile augmented reality applications, like games.