Category Archives: Note

Personnal note.

Report from the VR World Congress 2017

Bristol was hosting this three days congress. What a good excuse to explore the West Coast of England. Loved it! How to report from such an big event pact of keynotes, talks, debates and demos about Virtual and Augmented Reality, interactive storytelling and immersive art, architecture visualisation and video game development, to name just a few of the field involved ? I will start with  the general trends, then some key points from the talks, to finish with what really got me hooked.

This event was a solid push from AMD. As far as I can remember, AMD had always an edge to better process 3D on his most known rival Intel. Well, it looks like they are still in the fight, but now to process VR wirelessly with their Nitero tech. And this is important because, being in a virtual environment is pretty cool, however, without any cable in our way, it will be much better.  Hololens has taken that mobility party from the start. They made quite a demo there, bringing forward the Mixed Reality concept. That being said, I am still not convinced with this small field of view and the basic interaction where you pinch things around.

Mk2 in Paris
SoReal in china

In the meantime, the big hype is around VR location-based experiences. Mk2 in Paris looks very neat, curating only high quality content and SoReal, a VR theme Park in China sounds epic. On a hardware note, I am curious to see what kind of headset the Chinese will bring on the market with their DeePoon!

Another main trend is the good old haptic feedback. They are working hard to bring the third sense into the game. We are not sure what shape it will take:  gloves, waves, arms, sensors,…but it was explore in every corner of the congress.

Most important is the effort given to produce high quality content. At this stage, only great content will bring VR to the mass.

Follows bullets points of my tweets and comments of the different talks I followed:

On Wednesday:

  • Vive and the Evolving Ecosystem of VR” with Graham Breen from HTC.
    HTC Vive – Haptic

    What’s next for Vive and his ecosystem evolution? Not much on the hardware side, a part of those haptic gloves shown there. They are focus on helping and promoting content creators with their Viveport platform and the ViveX accelerator.

  • Streaming VR/AR Content over the Web”  with Jamieson Brettle from Google. That’s where it get really exciting! AR/VR through the browser. He was telling about pretty good compression performances for cloud point 3D with Draco. For sound volumetric or spatial audio, they are using Opus with ambisonic compression.
  • “Ghost in the Shell VR – A deep Dive” with Sol Rogers from Rewind.
    He delivered a great talk about how he and his team made Ghost in the Shell. He gave no numbers and ask very nicely not to take a photo!

    That’s all I got from Ghost in the Shell
  • “The Importance of touch: Mid-air haptic feedback for VR & AR” with Tom Carter from Ultrahaptics.
    How cool is that, conducting emotions through mid air haptic feedback? Because, at the end, it is the sense of touch that makes things real.
  • Perception = Reality: Exploiting Perceptual Illusions for Design of Compelling AR & VR Interfaces” with Hrvoje Benko from Microsoft Research. Don’t try to be perfect. Use tricks and perception illusion.

Using perception illusion to extend the experience; Passive haptic; Perspective Projection Mapping;  The rubber hand effect revisited as Haptic Retargetting; Body warping and world warping are very promising technic that make you  believe you interacting with objects and that gives you this haptic feedback.

  • “Virtual Futures Track: The Ethics of VR Risks and Recommendations” with Michael Madar from University of Mainz
    • Why should we care?  Our environment influence our behaviour.
    • VR brings the illusions of place, embodiment and agency.
    • Beneficial vs manipulative in VR, tricky balance to make.
    • Are we going to create a rating system for VR experience?
    • “Users should be made aware that there is evidence that experience in VR can have a lasting influence on behaviour after leaving the virtual environment.”
    • Users should also be made aware that we do not yet know the effects of long-term immersion.”

On Thursday, I went to only one talk from the lovely blue hair Mary Kassin from Google who explained her day-to-day process. She prototypes a lot. Trial, user feedback and iteration make the most of her time. She also mentioned the Tango ready phone to play with AR.

Alright, this is all very exciting, full of ideas, however, what made the biggest impression on me was the couple of VR visualisation agencies showing off their architectural Immersive and Interactive walk-through made with Unreal engine.  With this very realistic real-time rendering, we are getting closer to eliminate the long hours of rendering time for still images for a full immersive experience in the early stage of the process.  That is happening right now!

360 Panoramic Views with WebVR – Production Workflow

Optimising work is part of me. I hate repeating the same task again and again. We are using computers, that’s what they are good at, repeating a task for us. When it comes to work in a team, the workflow get even more important so that two different persons don’t overlap each other work or come back with always the same questions of “how you doing this?” or “where is this file?”. This is all good sense in theory, very far for being the norm in practice, at least from my experience, working mostly with architects and designers. Furthermore, as a freelance, months will pass without using a specific workflow,  working on a different project with different software. Being able to look back at how we did something  is a must if we don’t want to repeat the same mistake again.
On the Saydnaya project I was working on with Forensic Architecture for Amnesty International, we produce a series of interactive 360 panoramic views developed using the fresh WebVR. We didn’t get enough time to develop it in VR, although it is real-time 3D online in a browser, no plugin, no viewer needed. Pretty awesome stuff! Here is the workflow we used to get those 8 Interactive 360 Panoramic views online:
Saydnaya Circulation in 3dsMax
  1. Some of the 3d models were made with Rhino then exported to 3dsMax, most of the models were made only with 3dsMax avoiding any mistranslation between Rhino and 3dsMax, two very different modellers.
  2. All the texturing and lighting was done in 3dsMax, then baked to texture.
  3. Optimization was a big part of the process.. Models were  exported in OBJ (only meshes) and optimised with MeshLab. Maps were optimized with Photoshop.
  4. Everything was tested in a sandbox in WebGL using MAMP as a Apache Local Server, Atom to edit the PhD and Python files.
Saydnaya Circulation WebGL

If you are interested  and would like more details, let me know in the comment below or via email.

What is WebVR? It is 3 things, see this Reddit post here for more details:
  • WebVR is configuring the VR HMDs
  • WebGL is about drawing in WebVR using mainly three.js
  • HTML5 is the sandbox to play in.

Saydnaya Project link

Creating Memories in Virtual Reality.

I love  Metaworld ‘s introduction about creating memories despite the fact they are using the social presence as their main attraction, which I completely disregard in my research, … for now. Anyway, you are wondering why would we want to create memories in a virtual environment at the first place? Isn’t it better to go out in the physical world and meet real people?

metaworld-chess
Playing chess inside Metaworld

Well, it doesn’t compare. The whole idea of using VR to create memories is based on a couple of principles that are much more powerful to set up in VR : associations and embodiment.

Firstly, memories love associations. It is by associating one new information in a network of previously formed memories, that we are going to make sense of it and remember it. We will be able to retrieve the memory more efficiently later on by accessing the same network.

Secondly, memories are embodied. We don’t work like computers. We don’t have a hard drive or a cloud to engrave information one by one in a linear manner. We are storing information on a multi modal level within a mind-body system . So the more senses we are using when accessing a new information, which call upon the previous principle of association, the better we will be able to form new memories.

By proposing a persistent online world where we’re entering by immersing ourself with a HMD (Head Mounted Display), a head phone and a couple of controllers in our hands, we are already covering most of our senses. However, in VR we don’t have the same constraints as in the physical world. Anything is possible that can serve the purpose the participant is after. We can fly, we can travel into space, swim in the ocean or go inside a volcano.

Now, one of the debate about embodiment is the question of the avatar. When we enter into VR as ourself, we can see our hands, we don’t want to see a cartoonist body of ours. Although, if we want to interact with other people, we need to see them, somehow. Metaworld solves this issue by using a quite cartoonist avatars with non attached detailed hands. That way, you can see your own hands and other people hands with their avatar.  Let’s try it.

VR for the mass market with Google Daydream?

The next move in the VR world is the Google Daydream initiative. It sounds more like a big marketing push to take advantage of Google Cardboard success though. Nothing is really new in that product. The neat fabric covered headset comes with a remote that nicely fits inside. Take the remote out, place your phone instead and there you  go in VR. As we can all do already with any cheaper cardboard like version. So what is the big deal?
Daydream Home
Daydream Home
Well, it is about the experience, the user experience (UX). A comfortable headset, a powerful phone and a VR app interface will allow you to navigate between your different apps without the need to take the headset off. They want to keep you inside for longer. Sounds good to me.
However, the first question that comes in mind is what phones are Daydream-compatible? Obviously, the recently released Google Pixel phone is. How clever, right? But what qualifies a phone to be Daydream-ready or compatible? This is not so clear yet and it gets too technical anyway. (check this article if you want to know a bit more). My LG G5 with his LCD screen will probably not make it because of the too high persistence. Only the low-persistence of a Full HD OLED display allows to reduce motion blur enough to make the VR experience long lasting (explanation on low-persistence here).
Anyway, it is good to propose a better VR UX at that affordable scale. Clearly the “future proche” of VR is within the mobile market. Now, do we have the content to keep people inside for longer? And do we have the platform to develop more content? Perhaps we do with the WebVR. I will dig that up in a future post.

Transforming the classroom into a learning environment.

The purpose of school

The purpose of school hasn’t changed since classical antiquity: grouping students in a centralised location so they could learn from the same teacher. They are able to develop and acquire new skills on different levels: intellectual, economical, political and social  (The purpose of school). A school can have one or many classrooms. A classroom is the space, the environment in which class are held.

Classroom Factory

However, this purpose has always been a double edge sword: opening the mind of each student on one hand and teaching the same knowledge and the same skills to the group on the other hand. The roman empire in need for efficiency started to educate clever but like minded citizen to govern over their vast territory. More or less, two thousands years later, the British Empire was in need for just more of the same. With the demand for mass education started at the end of the 18th century and his industrial revolution, the classroom model was pushed to the extreme. This has barely changed since then. Actually, there is not a field depending on information and communication that hasn’t dramatically changed with the booming of new technologies apart from education.

Why is this model not valid any more?

The main issue I would like to discuss about is related to the way students are still taught in the classroom: one teacher instructing un large group of students using rote learning as the main method to retain information. If you are not going to use the information you have rote learned on a regular basis, it will not last in memory for long. Hence, why do we have to remember it? To pass exams. Why do we pass exams? Because it is the best way they came up to standardise everyone’s level in a factory based education. That way, wherever they were send in the empire, they would be able to fill in the same files, the same way. Hundred fifty years later, our civilisation has evolved. On one hand, the empire doesn’t exist anymore, there is no more centralised administration; on the other hand, we don’t need to rote learn because information is accessible in real time at the tip of the finger from anywhere.

What twenty first century world needs is people that can think differently, that can associate things that has never been associated before, people that can come with completely new ways of thinking. To understand how to shift our perspective, first, let’s go through the different models of the classroom?

Four classroom models

There are four main  models of classroom: the “classical-model” with a master and an apprentice, the “analogue factory-model” with one teacher lecturing a group of students, the “flipped-model” which is more student-centred using digital technology but still based on “classroom model”, and the “immersive model” which would immerse the student in an environment customised to his own level of learning and experience which is actually much more like the “classical-model” but scalable to today’s population.

Plato's Academy

The classical-model was a very human way of sharing knowledge. Going back as far as Plato’s school of philosophy that took place in Akademia, the model was based on dialectic. The main difference was made between seniors and juniors. They were learning by trying to resolve different problems and through dialogues with their pairs.

The Factory Model

With the industrial revolution and the exponential population growth, we had to rationalise the classroom. Following Horace Mann approach from the mid-19th century, the classroom was shaped around a group of 28 students in a 800 square foot room: the analogue-factory model. This is the model that still represents the vast majority of classroom on earth. (The following post gives a deeper view into the problem: How to Break Free of Our 19th-Century Factory-Model Education System.)

 

The last couple of decades have seen all sort of experiences using online technologies to propose “new ways” to learn. The “flipped classroom” is when students learn most of what they need outside the classroom, usually online, and use the classroom to do their “homework”. The teacher can then assist them if they request any help.  Student can learn at their own pace although online video-lecture are actually still based on the analogue-factory model where one teacher is lecturing many students. This usually happens through a screen using a mixture or graphic interface and videos that can bring a lot of other distractions as well. We can find some thorough practical comparisons between 2D and 3D  learning environment in K. Kapp’s book: “Learning in 3D: Adding a New Dimension to Enterprise Learning and Collaboration”. This brings us to the last classroom model: the “immersion model”.Flipped Classroom

Immersion has always been the best way to learn anything. Language is a great example. The best way to learn a language is to leave with a native speaking family in their country and city. The association between words, people, places and overall cultural context will drive the speed with which the language will be learn. Of course they are many reasons stopping us to apply the immersion method to acquire more knowledge or to learn new skills. We can’t afford sending each kid all over the world in families to learn new languages. That being said, can we use technologies to simulate this kind of immersive environment?

Immersive environment.

Let’s take another example. How can we apply the “immersion model” to learn mathematics. Depending on the specific area to be covered, the participant would immerse himself inside a mathematical universe, guided by mathematicians. They would explain different concepts with all their heart, walk the participant through, then propose to practice with some practical situations. For instance, a child that is learning division would be presented with cubes that could be cut into smaller cubes and put back together as many time as needed. The child can manipulate the cubes, throw them around, put them back together, watch them from every angles, compare with bricks or planets, only his imagination would be a limit. Watch this demo from Goldsmiths Computing Department  to get a hint on the potential of VR.

Mixed Reality Car

This is where Virtual Reality (VR) comes into play. VR has the potential to immerse a participant in such a way that his experience would be as powerful as experimenting in real life or even more powerful as it can be specifically design. Moreover, participants are able to experience it from anywhere, any time without the physical world constraints. Future posts will expand my research in this exciting field crossing over technology, art, science and imagination.

Questions for future posts:

  • How does humans acquire a sense of space in the physical world? How does it compare to virtual environment?
  • What is Virtual Reality, how does it have the potential to expand our real life experience?
  • What is the role of architecture in designing Virtual Learning Environment (VLE)