Unity3D “Special Effects with Depth” talk at SIGGRAPH

Unity3D continues to publish their SIGGRAPH 2011 talks.

Thanks guys!

Original Unity blog post.

Ole and I promised to share the slides from our talk at the SIGGRAPH 2011 Studio. The presentation covers various graphics techniques that use depth, normals and color buffers. The methods range from improving standard techniques to creating entirely new ones, like approximate volumetric effects or advanced image post-processing. The buffers used are in many cases the natural products of the rendering pipeline and offer a way to infer surprisingly much information about the scene, thus allowing for very rich and fast graphics effects.

It may be a worthy read both for artists and programmers, as we discuss the possibilities of some unusual special effects and their technical side.

Fast mobile shaders or rather Mobile! Fast!

I post the slides of what seams to be a great talk about 3D rendering on mobile devices (with Unity3D) by Aras Pranckevičius (@aras_p) and Renaldas Zioma (@__ReJ__) Unity Technologies.

The presentation is kind of general in the beginning focusing on the global differences and challenges between the "new" mobile and "classic" desktop rendering environments. Later they go in detail with actual techniques you can or should use to improve your rendering with Unity3D on a mobile device.

The presentation is quite extensive and all slides contain comments, which give a lot of details.

To give one example, MipMaps are your friends, even better with anisotropic filtering for cryspiness and use texture compression if your device support it. Moreover you can filter the Android market for texture compression support, which allows you to optimize your app for devices with different compression systems. If you want to!

How can the CPU be used to improve quality and speed of your 3D without draining the battery and how Unity3D does it.

Have fun!
 

Unity Track Day at GDC Europe in Cologne

As part of European Game Developer Conference (GDC) in Cologne Germany, Unity3D will present a full day tutorial known as Unity Track Day. The different talks will give details about cross platform games, performance optimization and better social integration. Registration is already closed, but tickets are still available on site. More information here.

Here is the detailed program.

Talk on mobile Augmented Reality at Paris Android User Group (Paug)

Hi there,

here is the video of our mobile augmented reality talk given at Paris Android User Group in june.

In the first part Atef Haouari presented a technique to create a mobile AR application for Android using position (GPS) and orientation sensors. There is a recap of the entire talk here.

In the second part (from 1:07h), I present some of our work in the area of vision based mobile augmented reality for Android using the Qualcomm AR SDK (QCAR) and Unity3D.

The presentation was held in french. Sorry for all the non-french speakers.

Watch live streaming video from parisandroidug at livestream.com

Qualcomm AR SDK for iOS (IPhone & IPad)

Hi there,

just got my hands on the new Qualcomm AR SDK for IPhone and IPad, which you can download here. I tested the plugin Unity3D 3.4 and it works as smooth and easy as the earlier released Android SDK . There is also an SDK for XCode available, but I didn't test it yet. It's just so much easier to do some initial prototyping with Unity.

On a first glance it is very much the same as the Android SDK. There is the ARCamera object and the image targets. So if you familiar with the Android release you could probably port your project very fast to iOS. Which is great news.

An initial test (image right) using the standard Teapot example with two markers runs perfectly on IPad2. I did not measure the framerate for the moment, but I think it is far from touching the IPad2 resource capacities.

To get started Download the QCAR Unity plugin for Unity3D (you need the most recent Unity3d 3.4!). Unzip, install the included app and it will extract multiple Unity package files among you will find QCAR-ImageTargets.unitypackage. Load (just double click) this package into Unity3d and choose the delivered test scene. It contains two targets, each visualizing a classic Teapot. Deploy for iOS. Unity3D will create now the corresponding XCode project, which you can compile in XCode and deploy on your IPhone or IPad.

Have fun

Plugins for Unity3D

Hi there,

during the following weeks I will write a small series of posts about plugins for Unity3D. In particular how to create plugins for Android and Windows applications.

While Unity3D already comes with a big amount of features, not all of them are available on all platforms. Especially the mobile platforms lack of many functionalities. Using plugins, you can use additional features in your project. You can integrate your own sound plugin, give access to a particular render engine or use other external APIs. Plugins can also be very useful if you already possess a library with functionalities that you are interested in using.

Unity3D allows you to write your individual plugin, which you can access through interfaces (C# or Javascript) inside your project. The integration of your own plugin is not necessarily difficult but can easily become cumbersome, if you are not familiar with concepts, like Dllimport or JNI.

The basic structure of a Unity3D plugin consists of multiple layers. At Unity level a Javascript/C# interface layer, a layer that serves as a wrapper (like JNI), and finally your_library layer (C/C++/Java). The underlying concept of such an approach is to define interfaces (interface layer) that will give you inside Unity access to methods implemented outside of Unity, for example in a particular library (your_library Layer). Such access is often realized using a wrapper, like JNI (wrapper layer). JNI (Java Native Interface) is a popular tool to access native (C/C++) code from Java, which you might know if have already worked with the Android NDK (Native Development Kit).

In the following, I present an overview of the some common methods to access external functionalities on the Android and the Windows platform.

Android
A very easy way to access a particular Android function from Unity3D, without the need to implement a particular plugin, is the AndroidJNI interface provided by Unity3D. The wrapping is done internally by Unity3D, so you do not have to worry about that. The drawback is that you can only access functions provided by the Android SDK Unity3D is linked to.

Accessing Android from Unity3D via AndroidJNI interface

There are basically two ways to implement Unity plugins for Android applications. Firstly, using Java and the Android SDK or secondly in C/C++ using the Android NDK, as illustrated in the following illustration.

Accessing Android from Unity3D using JNI

Windows
Plugins for Windows applications are implemented using a so called Dllimport, that gives access to an external shared library (dll) under Windows.

Access to an external dll library

In the several follow up posts I will go in more detail on each of these approaches.

Links
Unity3D Plugins Documentation

Breizhcamp Recap

Hi there,

Last friday I went to Rennes to give a talk at the Breizhcamp on mobile augmented reality for Android using the Qualcomm AR SDK and Unity3D. Bassically an extended version of my previous PAUG talk in Paris. Slides are below. There will be as well a video I will post as soon as I get my hands on it.

Anyway, Breizhcamp was really cool. A lot of nice presentations around GWT, .Net, Android, iOS etc. There was also a talk about Kinect, which was interesting and quite entertaining at the same time.

Breizhcamp Rennes 2011

Slides of my mobile AR talk at PAUG now online

Interview at Droidcon 2011 in Berlin

My interview (in german!) with Golem.de at Droidcon in Berlin march 2011 on mobile Augmented Reality. Link to the article Golem.de http://www.golem.de/1103/82345.html

Initial thoughts on mobile AR (SDKs)

Novel augmented reality SDKs allow developers to create a new generation of innovative augmented reality games and applications for mobile devices that use Android. This new technology is able to estimate the handhelds’ pose from almost any desired 2D image or photograph.

Thanks to increasing performance of modern mobile devices, innovative operating systems and powerful third-­‐party libraries, developers are able to create a new generation of augmented reality applications. Such apps are able to locate their position and orientation (pose) in space, using cutting edge computer vision algorithms. This way, virtual information can easily be integrated into a real environment in real time, which is known as mobile Augmented Reality (AR).

One of the first mobile AR applications was Layar. It used integrated sensors, like GPS, compass and accelerometer, to determine the device approximate position and orientation. In the following, virtual information about the environment was integrated in the captured video image.

Another approach to augment reality uses computer vision algorithms. Instead of using sensors, it determines the cameras’ pose using a visual reference, which is usually a 2D image (marker). One popular application that proposes such an approach was ARToolkit. ARToolKit was early ported to Android (NyArtoolKit), but has never been really successful due to bad performance and the inflexible marker design.

Today, a novel generation of computer vision algorithms are able to estimate cameras’ pose from natural features inside the environment in real time. This means, that almost any 2D image or photograph can be used as a reference, if it contains a minimum amount of visual information (features). Therefore, almost any 2D image can be used as a reference marker to allow the tracking algorithm to extract the camera pose in real time on a mobile phone.

In October 2010, Qualcomm released a free SDK (QCAR) for Android powered devices, which uses such advanced visual tracking algorithms. The SDK can be used in combination with the Android SDK or NDK. There exists also a plugin for Unity3D. Using these tools developers can create mobile augmented reality applications, like games.