Technical AR implementation: Reli Magnezi
At the beginning of 2022, I participated in my third course in my undergraduate degree in Interactive Experience Design. Until then, I was mainly familiar with relatively simple tools such as EKO (Z"L) by Yoni Bloch, and with design tools that formed the background of my non-interactive creative work. In another, parallel course during the same semester, I began struggling with the Unity game engine and the C# programming language. This later became much easier, about a year later, when artificial intelligence tools became widely accessible.
However, what truly amazed me at the time was something else entirely: Snapchat, and specifically its augmented reality (AR) capabilities. In Unity, we mostly learned technically demanding and often discouraging processes, and my weekends were spent writing and debugging code. In the other course, by contrast, I could dive directly into writing stories.
To spark our enthusiasm for the technology, the course instructors invited someone from Snapchat to explain its capabilities and demonstrate how to create lenses. At its core, the idea is simple: define a trigger object that, when recognized by the camera, causes another object to appear. He demonstrated this using a drawing placed on a table, onto which a smiling emoji was overlaid on the phone screen. Naturally, someone observing from the side could only see the drawing, not the emoji. This moment immediately struck me. An entire world of hidden message transmission suddenly revealed itself.
In fact, the system consists of three parts: the trigger, the key, and the content. The "trigger" can be any object in physical space, for example graffiti, a bus timetable, or money. The "key" is the code that links the seemingly innocent object to the content, and it does not necessarily indicate which object should be scanned. The content is the content itself, potentially something secret: instructions for preparing a popular drink, the location of a treasure, or a computer password. Minimal traces. Only the combination of the trigger and the key leads to the hidden content.
At the end of the session, we were sent home with an assignment: to create our first lens. After finishing it, I decided to test an additional idea. What would happen if I showed the camera only half of the trigger? Would it still display the additional layer? To my surprise, even 1/16 of the trigger-object was enough to activate the overlay. In other words, it is possible to distribute the trigger among sixteen different people who all share access to the same content layer. However, in order to maintain discretion, one must find a way to prevent someone who steals only part of the trigger from uncovering the entire content. One possible solution is that scanning only part of the trigger reveals different information from what is revealed when scanning the complete object.
With the ambition to create a mystery story based on these technological properties, I began working on the final project for the course. The idea itself was generic but effective:
A hacker named “Jaguar 411” has broken into the player’s accounts. The player must communicate with him via WhatsApp over the next 20 minutes in order to prevent the information from being leaked. During the WhatsApp conversation, the player receives clues about what to scan, along with the lenses themselves.
In practice, this was the inverse of an escape room: an “entry space.” We had to think about how to guide the user toward a specific location, our computer lab classroom. The starting point was the entrance to the Mexico building. After the opening video, we sent the user the first lens (you are welcome to try it with your phone on the image to the right). When the player scanned the central wall, a GIF of a hand appeared, instructing them to approach the wall. As they moved closer, the trigger became only the blue figure, and the overlay content changed accordingly:
Another discovery that helped shape the gameplay was that although Snapchat’s software does not recognize three-dimensional objects as triggers, when the trigger is a staircase with a sufficiently distinct appearance, the player can stand beside it and the lens will still recognize it. This allowed us to guide the player to the second floor.
When the player reached the lab door and encountered a keypad, they received another lens:
Anyone with experience in escape rooms or puzzle games would immediately understand that, in order to enter, four objects must be found to form the code. The colors and shapes hint at the sequence of inputs on the keypad (a green circle, followed by a blue rectangle, and so on). But how are the numbers themselves found? By searching the surroundings for objects to scan according to the colors, and by following hints from Jaguar:
Anyone who managed to enter the lab effectively completed the mission and won.
Like many projects that remain within the boundaries of academic courses or research, the ideas I discovered here did not lead anywhere beyond this framework. I did not continue working on hidden messages or further testing the capabilities of AR tools. Still, I believe this is a technology with significant potential due to its low physical footprint in real space. Vast amounts of writing already exist on the potential uses of augmented reality, and I wish to add just one more point to that discussion this time.
