Long-lived analogy of ‘desk’ in computing

The foundational analogy for computing we use today is a ‘desk’. Imagine a desk, with different functional objects, like a calendar, a calculator, post-its, documents and folders. And our computer and smartphones are abstraction of that. The programs and apps represent the tools on the virtual desk in a computer and a smartphone, with different functionality and interfaces.

Blank note — nothingness that can be anything

Comparing to that, if you think about a paper notebook, the one and only interface, the ‘blank page’ offers you the infinite potential functions. Sometimes it’s a calculator, it’s a calendar and a to-do list, you jot down memo, or outline the draft of complex ideas.

I’ve been lately thinking, maybe this ‘blank note’ is the next analogy for computing and interactions. If AI can interpret the intention of those handwritings and notes and help connecting those dots, instead of us reaching our hand to use different objects and following the pre-defined interface, we can enjoy this most natural form of thinking and creating, on the infinite freeform space in a way that makes sense just for ourselves, and summon AI to interpret the intention and trigger functions or take actions, like to calculate, search, visualize, update calendar or to-do list, send to someone, and many more.

Very rough prototype of what the Note OS could look like. You don’t need different interfaces for different features like an app. It’s driven by what’s on your mind.

Similar interactions appears in… a fantasy movie, not sci-fi

We often think our future will look like the sci-fi movie, like Iron man using Jarvis and all those gesture interactions and holograms. The development in AR/VR is slowly but surely taking us there.

But maybe instead of creating this highly immersive new virtual layers into our perception, I want something on the opposite side — a sheet of paper, with the infinite potential as it’s blank, with a bit of intelligence behind it to accompany and follow my train of thought. Instead of ‘Iron Man’, I dream of ‘Harry Potter’ interactions to be the future.

This is a very early idea that will get more tangible and mature, or, turn out to be a just a passing thought ;)

My retrospective about why and how I updated my portfolio website

If you’re a designer with your website running, you often feel the urge to rebuild everything from scratch — your design feels outdated, or you come across a really great portfolio of others… many more reasons. I always had this feeling, and have been procrastinating for a long time. And recently, I finally refreshed my website.

But not because the design is outdated. I updated it, because my perspective on design, craft, and the website has changed, a lot. The new website is just a reflection of it. This article is a self-retrospective to capture these changes and musings while building the new website.

sml on Twitter: "Finally renovated my digital home! I'm inviting you all so feel free to explore, leave something there, and share your thoughtssss 👉 https://t.co/Y8Kwonas8P pic.twitter.com/Qu9XdNd5Pm / Twitter"

Finally renovated my digital home! I'm inviting you all so feel free to explore, leave something there, and share your thoughtssss 👉 https://t.co/Y8Kwonas8P pic.twitter.com/Qu9XdNd5Pm

1. It’s not about getting a job.

I used to think of a portfolio website as a gate to a new job. My imaginary audience was recruiters and potential managers. To confess, that was why my previous website was in dark mode, as I thought it makes me look like a cool design professional. As a fresh graduate, I focused too much on hiding my lack of experience, putting details about projects, and sharing my vision with fluffy words. I was embarrassed to see those pretentious efforts from my past self.

For the previous 3 years, that website has never been used as a way to get a job. Instead, it was where people from Instagram, Read.cv, and Medium visited to know more about who I am, after coming across my design work or articles. My real audience was the people who were interested in me and my projects, willing to follow for more updates and get in touch for a chat, questions, and feedback about my works.

I realized the role of the website as a way to make the right impression on people that I want to connect and interact with, not just to showcase my work for a potential job offer. My old website was not serving the purpose, with a lack of reflection on my interests, works, and my personality.

2. How different are they — inviting guests into my home vs. having visitors on my website?

When I was designing my website, I was also setting up my home to invite my friends and family for a few days — becoming their hotel for their travel to Singapore. I wanted their time in Singapore and my house to be special, personalized, and memorable. It was lots of work than I thought. It started with simple tasks like buying extra bedding and cups but in the end, I designed a custom transportation card, set up a Notion page for detailed instructions and recommendations, and even put NFC stickers on complex gadgets like a washing machine and oven for a manual.

When it comes to those guests, I was putting a lot of care into how they feel and experience in detail — maybe beyond necessity. But when it comes to designing a website, how much do I care about the guest? Aren’t visitors to my website also guests in a sense to my digital home? Crafting a clean layout and an interesting scroll interaction can be important, but have I considered how they ‘feel’ while they browse my space? What kind of impression do I want to leave to them?

This was a eureka moment as an interaction designer. I realized I’m interpreting ‘interaction’ as ‘human-to-computer’, ‘human-to-technology’, and ‘human-to-information’ interaction, not about the actual human-to-human interaction happening behind the medium in between. With my website, the visitors are interacting, scrolling, and clicking my works, but what’s actually taking place is an interaction between me, showing and telling about myself, and the guests, looking around and listening to my story.

That’s why I added a few experimental features, like a unique hex code for each visitor, and a guestboard. I’m calling visitors with a unique hex code, and letting them use it as their color to leave their trace on the website. It has been one of a feature that visitors love and comment about!

3. Design is just a fraction of the experience — not only for the website but for everything.

I started designing and building the website in January 2023 and finished it in the same year June. But all the design and the interaction were already built around April. Then what took me 2 months before the release? When I thought it was almost done, it was a new beginning — I started to run a performance check and SEO checker. I expected a few fixes, but there were ‘TONS’ that I didn’t know about.

Are images and videos loading fast enough? Do I have an unused package, or is there a way to cut them down? Is this CSS feature or video format supported on other browsers and different OS? (Safari and iPad were a nightmare) I learned a lot about how the search engine works from an SEO checker.

And when it’s done, can I make it spread to more people? What’s the effective channel? Can I share it in advance to get some feedback? Will people share my project links somewhere else?

All these are necessary questions, but things that I’ve never considered for my previous website. And this is not only about my tiny website. It’s about how my point of view toward a product and an experience has changed — from focusing on polished design and interaction to the overall experience, from start to finish, beyond design. Now I know that when there’s a successful product or experience, it’s from a great ‘execution’ from many different angles— from design, engineering, marketing, and many more.

Sometimes it’s painful to see your own past projects and website. I know that the time will come again for my current website as well. But it’s actually an indication of growth in yourself.

Curious about what my next few years will look like, with new projects filling up my digital space, and eventually how my perspectives will evolve again for the next renovation.

Visit my renovated digital home :)

Seungmee Lee

Refreshing my website in 3 years —  and things that changed in between was originally published in UX Planet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Stable Diffusion by stability.ai is one of a number of exciting AI methods used to generate images from text descriptions. You might have seen its different demos and early use cases flying around social media — from generating avatars and hero images, to even making full music videos.

These AI-assisted creator tools work similarly, providing the user with a number of variations based on a prompt, from which the user chooses the one they like the best.

Sometimes the variations can be completely random and unexpected, which in a creative process can actually be very helpful with many AI tools designed to use this unpredictability as a feature. For example, with a new AI-assisted feature introduced on Notion, you can almost ‘collaborate’ with AI to get inspired and improve your writing.

Trying out these AI-assisted tools and seeing their results have been interesting and thought-provoking. But as I started to take one step further, and make a simple AI-assisted tool by myself, there were even more self-questions and insights. These were the two main questions that I’ve pondered upon.

Aside from the results of AI-assisted tools, how do these tools impact the way we think? And as a designer — how can we make the UX of AI tools even better and more useful?

Rather than a ‘how to’ tutorial for Stable Diffusion, in this article, I’d like to share learnings and realizations about the questions above, through a practical example where I built my own app called ‘Viewmaker’.

Building ‘Viewmaker’

What I’ve made is a web app that uses a camera on mobile to enhance the photo with the Stable Diffusion API.

The level of enhancement you can make is way much more powerful than simple editing, and filtering apps, as you can use text prompts to describe how you want to ‘recreate’ the photo you just took. That’s why I’m calling this project ‘Viewmaker’ in comparison to the original role of a camera as a ‘Viewfinder’.

I’ll be talking about where you can access it at the end, but before that, I’ll explain some of the decisions and challenges I noticed when building with Stable Diffusion. Let’s start with a story on how Stable Diffusion can be more interesting to the end user.

What makes stable diffusion an interesting tool for creators?

1. A tool affects the result, but also the approach

Over the weekend or on travel, I carry my film camera, which is heavy to carry and more costly to buy films and develop photos. I like the unique texture and colors it creates, but more than that, I choose it over other digital cameras, because it changes my attitude toward the world surrounding me.

The heavy weight on my shoulder and the limited number of shots make me aware of every moment that I encounter and be ready for it.

So what does the film camera have to do about the hot topic, stable diffusion? The tool you use for creating something not only affects the result, but also the approach and perspectives toward a subject. In contrast to holding my film camera with conscious attention to subtle moments and good lighting necessary for the film, when using smartphone cameras or DSLRs, my attitude and subjects change to things that are better captured in higher resolution, wider angle, and even in video.

Now bringing an AI assistant to the smartphone camera changes that perspective once more, and therefore how the camera is used.

2. How did ‘Viewmaker’ affect my perspective?

Viewmaker can change the approach to using a camera, as the goal is to augment the picture with creative prompts to create new and different scenes. Some use cases might be an interior designer getting inspiration, or a storyteller creating new and imaginative scenes.

However, in the first few attempts of using it, it was hard to break out from what was in front of my eyes. If I was looking at a plant, I could only think of words like ‘flowers’, and ‘butterflies’. But after I realized this, I started to question myself, ‘what would be the opposite, unexpected, unnatural, and the weirdest thing in the scene I’m looking at?’. From then on, I started thinking completely outside the box while I was on the same road that I go for a run.

We’re excited to see how stable diffusion will increase the efficiency of creating images. After building and trying the app by myself, I’m more curious to see its impact on how it changes creators’ questions toward the world.

3. I ‘like’ when the result is different from what I intended.

The model I used, in the beginning, was an outdated version, so even when I had straightforward prompts, the result was far from what I imagined. As I tried out different versions and tweaked the properties, it started to return quite accurate images. Here’re a few early results that didn’t turn out well:

But looking back at images that I made with Viewmaker, I enjoyed this ‘unexpectedness’ from the outdated version. Adding on to earlier points about how Stable Diffusion introduces us to a new perspective, these unexpected images were so intriguing, as they made me question myself further:

‘How come AI interpreted my words into this image?

‘How this image is correlated to my prompts?’

We usually seek an exponential increase in productivity with AI and ML. But this ‘unexpectedness’ can also be something creators can make use of the technology as well. This brings me to how we could make this explorative type of AI tool more user-friendly.

How we can design stable diffusion more user-friendly?

1. What do you mean by describing an image with text?

Stable diffusion gives lots of freedom and flexibility to users to create an image just by describing it in text, but you still need to learn some tricks to get the exact image you want.

When I shared my app with my friend who doesn’t know anything about stable diffusion, she came back to me with the screenshot below, saying it doesn’t work. She was literally ‘describing’ what she imagines in long sentences.

To introduce stable diffusion to general users, we have to keep in mind that prompt engineering is still quite far from natural language. Instead of ambiguously saying ‘describe what you imagine’, we need to provide more specific guidelines on how to choose the right words.

2. Providing full control might not mean ‘better’

There’re quite many properties you can tweak to control the result from stable diffusion than prompts and image themselves. But instead of providing the full flexibility to control the result which can be confusing, selecting the most relevant properties that suit users’ needs and purposes will simplify the experience.

And even some prompts can be provided as default as well. For example, since Viewmaker is using a photo as an input, when the result involved a flat drawing or an illustration, it didn’t go along with the base image. So I added a few default prompts, like ‘HD, detailed, photorealistic’. Also, it helped to add a lens spec, as other articles introduce as a way to improve the quality of the result. (Kudos to Elenee Ch on a great article!)

3. More to explore — what’s the ideal prompt interface?

The interfaces and interactions to use stable diffusion should be designed with consideration of the context. In the first version of ‘Viewmaker’, I used text input as most of the stable diffusion interfaces do. But it didn’t take long to realize that it’s not ideal for mobile and for inpainting, as the keyboard will take almost half of the screen and block the image. That’s why I replaced it with voice recognition, as I felt it’s better to keep looking at the image to come up with prompts.

But there’s also a downside to using voice to add prompts. Even for me, I was not fully confident in saying words out loud when other people were around me. This is something that I’d like to improve later on.

Outro, and the next step

In all, the interesting part of many AI-assisted tools is that the output is often unpredictable, which can inspire the user to explore ideas they wouldn’t have thought of, or from a totally different angle. At the same time, different users interact with AI in different ways by providing prompts that only they may have thought of. So it becomes an interesting problem for designers and developers to prototype and improve the experience.

It’s also interesting that as there’s a really low, or no learning curve to use AI-assisted tools, an approach, and the result will really vary depending on ‘who’ uses it for ‘what’.

While playing around with Viewmaker, I could think of many types of creators who’d find this tool useful. From a fashion designer visualizing ideas from impromptu inspiration on the street, an interior designer or architect bringing imagination to life and sharing with clients on the spot, or a storyteller imagining the most unexpected narrative that can happen in the space… It’d be interesting to hear more people’s thoughts about how this tool changes your perspectives toward your subject.

Although ‘Viewmaker’ is still in ‘work-in-progress’ status with pending improvements in mind, I’m open to sharing this with more people. If you’re an artist, designer, photographer, or any type of creator interested in trying what I’ve built, please reach out to me via Instagram or Twitter

Check out other articles from the community about generative AI

• Generative UI Design: Einstein, Galileo, and the AI Design Process
• ChatGPT Is Constantly Tricked by People, but Who's Fooling Who?

I connected Stable Diffusion with my smartphone camera. And here’s what I learned. was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Years of exploring where to track habits and failures

It’s the time of the year when everyone starts to think of new habits to build over the new year. Same for me, I have a few habits I want to stick to myself that I’ve been struggling with over a few years.

I’ve given a shot with a few different habit tracker apps, but so far, I needed more than what’s out there. First, I wanted more flexibility to track different types of information, from checking if I had a reading time of the day, to logging the amount of money I’ve spent today.

And I wanted more convenience — like automation. For example, I’ve been trying to reduce time with my phone, so I wanted something that can 1) keep track of my screentime, and 2) check the day I’ve succeeded in using my phone below 4 hours so that I can see my progress visually. I didn’t want to forget checking my screen time for weeks and typing all the records by myself.

To achieve those points, I’ve tried to build a system with Google spreadsheet and Notion. They provide the flexibility and room to build a system by using formulas, but they weren’t still enough. The biggest issue was the interface as a tracker. Since both are ‘table’ and ‘database’, it was so hard to browse where I needed to put my input on my phone.

And the answer is — Notion with iPhone shortcuts

And I finally found a solution this year. It was to use Notion with iPhone shortcuts. I’ve built a few custom iPhone Shortcuts by myself previously, so I decided to figure out how I can use Notion with it. Notion doesn’t provide those shortcut options yet, so you need an additional app to enable this. It’s called ‘Nautomate’, and it’s paid one, so if you see the value of it from this article, be ready to spend a few bucks for it!

How to set up one for yourself

I’m sharing my shortcut here, but you definitely need some customizations, depending on how you set up the database. But as an example, I’ll walk you through how to set up a screen time tracker as I’ve described above. Hope this is a great starting point for you.

• Shortcuts
• Shortcuts

In the nutshell, this is what happens when I trigger the shortcut.

1. Extract the screen time data and turn it into a value.

• Open the screentime page and capture it
• Crop the part with the hour and minute
• Extract numbers from it and calculate it as hour + minutes/60
• Set the dictionary (Dictionary on iPhone shortcut means a set of data with fields and values. For example, I can set a dictionary with a field named ‘📱’ with a value of the number I’ve calculated above.)

2. Find the right Notion entry, and update it with the value above.

• Get the URL of the database and get the unique ID of the database.
• Find the entry named ‘2023–01–05’ (today) from the ID of the database.
• Update the entry above with the dictionary value above.

And here’re a few things you need to do from your end.

1. Connect your notion workspace with Nautomate.
2. Set up your database first.
3. Get the URL of your database.
4. Replace the link here and the type of data you want to update with the iPhone Shortcut. You may need to tweak the area below.

Closing

I’ve been really enjoying what I’ve set up for the new year, and hope this helps those who have a similar issue with existing habit trackers. I’m not part of Nautomate, and will not have every answer for any issue you might face, but open to help. Reach out here!

How to use Notion with iPhone Shortcuts was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

As always, the best way of learning is by doing.

How people without technical knowledge about Machine Learning can better understand it?

From creative AI tools to intelligent features at our fingertips on our daily devices, AI/ML-applied services are now prevalent everywhere. And especially if you’re responsible for envisioning the next features or products, it’s so natural and timely to think about how AI can improve your product, no matter if you have a good understanding of what AI or Machine Learning is.

Although the recent innovations that AI is bringing in are amazing, to be honest, I was a bit overwhelmed because I felt like I don’t really know what AI is and what Machine Learning is. I know the basic definition and mechanism of AI/ML, but still, I wasn’t perfectly sure if I can suggest ideas for future experiences and say it’ll be done by AI, a bit irresponsibly. That’s when I realized I’d want to know more. What is the task that we can rely on AI? How it becomes ‘artificially intelligent’?

That’s when I started to read, ask, and search for how I can better understand what AI/ML is. There really were lots of resources out there, even to the extent that I felt overwhelmed by them and not sure where to even start.

I’m still working on this, but among the learning materials that I came across, there was one thing that really gave me a feeling of ‘Oh, this is what it is!’. This eureka moment came when I tried to train a model by myself. But how, especially when I’m just starting to learn about it?

The best way to learn AI/ML is to make one by yourself. But how?

Less known than creative AI tools, there are actually quite lots of no-code ML training tools out there. There are tools for enterprises and businesses for their products, but there are also more accessible options — even free to use, and without using code.

I tried Lobe.ai and Teachable Machine so far, but today I’ll introduce Teachable Machine because it’s more accessible since you can do it on your browser, and also easy to see it applied in your web/apps.

What is ‘Teachable Machine’?

Teachable Machine is created by Google Creative Lab, and introduced as an experimental tool to let everyone train their machine learning model. You can train your own model by using photos, webcam input, and microphone input as a source.

The models you make with Teachable Machine are real TensorFlow.js models that work anywhere javascript runs, so they play nice with tools like Glitch, P5.js, Node.js & more. It can even work on Arduino, which opens up an opportunity for more hardware-related projects. It’s a pretty straightforward tool that doesn’t require any learning curve, but I’ll guide you briefly on how to use it.

1. Train your model on Teachable Machine

The first thing is to decide what you want to build and figure out what you need as samples. It can be like a thumbs-up detector based on your webcam input, a whistle detector based on your microphone input, or a dog breed detector trained with images from the Internet.

1. Choose what to build and define classes

Let’s say you are now building a cup detector, and you’ll train a model with your camera input. Now you need to decide how many classes you want to split. For now, I’ll keep it simple— cup, and no cup.

2. Collect samples

As you have the classes to train, the next step is to collect samples. For the ‘cup’ class, I’ll keep pressing the record button to capture myself holding a cup, and for the other class, I’ll record myself without the cup.
You need to make sure you’re capturing the right samples for the class so that it doesn’t get confused during the training. Also, try to make the number of samples for each class to be similar.

3. Train

Ta-da! Within a few minutes, the model will be ready for you. See if that works as you want on the preview section. You’ll be surprised how quickly and easily you just came up with a machine-learning model!

4. Refine your model

But before getting too excited about it, try to challenge the model. Give trickier situations for the model to tell. You’ll find some situations that the model doesn’t understand and gets confused. This is where you need to start refining your model. To solve this, you can go back and provide more samples that can help the model to understand more diverse situations.

For example, since I only trained the model to recognize ‘no cup’ status with myself inside the view, when my face is not present, you can see the confidence level trembling.

These are some potentially tricky situations for the model to tell, considering what I provided as the initial set of samples.

• I trained the model to recognize the cup with the right hand. What if I hold it with my left hand?
• As I’m showing above, it’s confusing when there’s no one present in view while holding the cup.
• Obviously, the model is only trained to spot the cup I showed, so I need to provide more samples with different cups.
• Would it still be working if I switch my background to another space?

2. See the model working on a browser

Now you have your model, but do you want to see how it can be applied in other projects, like on your browser? I also wanted to see it by myself, so I’ve quickly built something where you can import the model you trained and display an image for each condition.

This was to demonstrate that your model can be used, so the interaction is still limited to the model based on webcam input, and to display some images.

1. Export your model

First of all, you need to export your model. If you click ‘Export Model’ from the preview tab, you will see the modal window like the left image above. As you click ‘download’, you will see the option to download the model. It’ll give you a compressed .zip file that contains metadata.json, model.json, and weights.bin.

2. Fork the codesandbox template and load your files

Then you can go to the template that I’ve shared. Fork the codesandbox, and so that you can import your exported model under the folder ‘tm-my-image-model’. Make sure the files have been replaced, and try to refresh the page, or even the codesandbox server if needed.

When your model is ready on the template, open the page in a new window and start importing images for each condition.

3. Initiate the webcam, and interact!

All set! As you click ‘start’, it will initiate the webcam and use your model to tell the current status, and display the image that you’ve uploaded. Though it’s limited to displaying images, you’ll get a rough sense of how you can use your model for your project and how to design the visual feedback for it. Feel free to fork and tweak for more delicate interactions with it.

Lesson learned from trying it by myself

1. Demystify AI/ML

The biggest lesson from Teachable Machine is that it almost changed my mental image of AI and get away from the overly romanticized impression of it. Even though it’s so amazing to see how it can learn from provided samples to detect something, it made me realize that it’s heavily dependent on what we teach it, and how we teach it. It almost felt like I’m teaching a baby to do one simple thing, and I should tell him or her what to do for all the edge cases that I can think of.

I hope this article is not interpreted as I’m underestimating AI with limited knowledge. The magical interactions and features that we can build with AI and Machine Learning can be easily seen as magical technology, if you just see the result of it. But what you learn from trying to build one with Teachable Machine is that, it’s far from magic. It’s still the result of people’s efforts involved in coming up with something, which sometimes can be affected by bias and lack of consideration.

2. Open up your imagination about ‘user input’

Another advantage of this exercise is that after trying this, you’ll naturally come up with more ideas on what you can build with it, and rethink the notion of ‘user input’. As you can see below, a certain sound that a user or the environment makes can be input, or you can communicate with images or drawings, or even objects and posture within the view.

If you’re a designer working on some user flows, most of the time, it’s always the user’s intention that triggers some reactions to the product or service you design for, like ‘when the user clicks’.

But with Machine Learning, we will be able to understand more about the ‘context’ that the user is in, so that the service can provide more personalized, adaptive experiences for the user, beyond the user’s intention. Still, there’ll be lots of necessary considerations, like privacy, and how to design the experience to be adaptive, not intrusive.

3. Can it be used in projects?

Rather than as a learning experience, how we can make use of Teachable Machine in actual projects? I’m not going to say it can be used for the actual production of features, just because I haven’t tried it, but at least for designers, I see a huge potential to use it as almost a ‘prototype’ or ‘testing asset’ for user experiences, especially about more intelligent, contextual features.

There’s a user testing method called ‘Wizard of Oz’ testing. It is a method where participants interact with a system that they believe to be autonomous, but in reality, is controlled by an unseen human operator in the next room. It’s a fantastic way to explore the experience of a complex, responsive system before committing resources and development time to actually build that system.

In my work, we need lots of this ‘Wizard of Oz’ testing since I usually work on early ideas and test assumptions we make for new concepts. But even when there’s another person controlling the setup, it is actually quite hard to follow the testing flow or participants’ behavior — it’s not easy to keep tracking the participant’s subtle behaviors, especially when you’re trying to hide that it’s controlled by a person.

But Teachable Machine can be the perfect option to consider, since it’s so easy to train a model for the controlled setup. As the point of Wizard of Oz testing is not to commit to the development, you don’t need to refine the Teachable Machine prototype to be a fancy final design. Even the codesandbox I’ve shared above to show some images for the certain status can do the job.

I hope this was an informative and inspiring introduction to Teachable Machine for you to try it! I’d love to hear others’ thoughts, impressions, and any questions regarding the codesandbox template I’ve provided.

And if you want to take a look at what others built with Teachable Machine or more in-depth tutorials, check out this great document!

Teaching a machine with ‘Teachable Machine’ — also teaches yourself about what ML is was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building a hand gesture interaction design tool, Handzplay, and learnings from it

What is Handzplay?

I recently built and released a website where you can try and explore hand gesture interactions — Handzplay. It’s a self-initiated project for me to play around with hand gestures, but then I realized I wanted to go one step further, allowing others to play around it and create interactions.

You might have already seen images like below— how ML is used to detect users’ hands, body movement, and facial expressions from photos or camera inputs. Though there are more advanced sensors and products for that you can purchase, I was fascinated by the fact that anyone with a laptop or webcam can see detection working, from demos provided by Tensorflow.

As an interaction designer, I was naturally intrigued to build some interactions out of it. The biggest difference of hand detection demo and interaction demo would be giving control and feedback to users by using their hands. I wanted to design the demo of technology into more experienceable tool.

So what can you do on Handzplay?

Handzplay aims to let everyone explore the potential of hand gesture interactions. With any device with a camera, you can use your fingers and hands to interact, and create new ‘rules’, defining what triggers which reactions.

Instead of defining a set of interactions by myself as I did with my previous project, I wanted to build a ‘tool’ that everyone can try and explore it.

You can simply start from some pre-configured settings from the template, or go with a blank page and create a new ‘rule’. This rule consists of two parts —a trigger, defining what kind of pose, action, or finger-finger distance will trigger something, and a reaction, choosing the type of reaction that will be prompted by the trigger.

Lesson learned

If you’ve already tried Handzplay by yourself, you might have some impressions, takeaways, and hunches about how to design for gesture interactions (Curious to hear your thoughts!). Same for me, as I’ve spent quite a time in front of my laptop trying all these hand postures and motions, here are some learnings and takeaways.

1. Gesture interaction might look cool, but it’s not for everything

We’ve imagined using hand gestures to control devices, but we might have overrated and romanticized it since we’ve always seen it in cool future sci-fi movies. When we apply this in our daily lives, there’re lots of cases where it’s not appropriate to use hand gestures.

Imagine you’re doing the following activities with gestures. You might already feel pain and soreness in your fingers and arms, just by imagining it.

• Designing a user interface on Figma with hand gestures (Uh-oh)
• Type lines of codes with a virtual keyboard (Seriously?)
• Play a game for an hour (Let’s say it’s a typical game you play on PC, requiring you to move around in 3d space, attacking enemies, etc)
• Skipping 30 songs in a row with air-swipe (You’ll probably have a cramp when you reach the song you wanted)

These might sound extreme, and we already know that we cannot use hand gestures for everything just because it looks cool. But thinking about what won’t work will eventually help us understand what hand gestures are good at.

2. The type of gestures can be almost infinite, but how can we categorize them?

When we say hand gestures as an interaction, everyone has a different image. Some may first think about over-the-air motion, like swiping in the air, and others may think of certain postures, like thumbs-up and victory.

But while I was playing around with Handpose, I especially enjoyed gestures like putting two fingers together, like index-thumb click, and middle-thumb click. This was interesting gestures as an interaction, as this sense of tactility that’s usually missing for hands-free interactions functions as ‘feedback’ in interaction.

For example, the movement of ‘air-swiping’ to the left itself doesn’t give any feedback to user, unless you see visual/audible changes on a screen. But if you touch your finger to finger, you can feel it without secondary feedback, because it’s your finger.

That’s why I’m providing three types of gestures as triggers for creating a rule on Handzplay. I still think those are not enough to capture all the possible hand gestures, but here’s the way I’m categorizing them at the moment.

‘Pose’ is certain postures that you’re already familiar with, like thumbs up, victory, and pointer, and those are highly affected by your cultural background.

‘Action’ is the movement of your hands, like swiping in the air. It can be linked to change the status of element on screen, or play an audio, but it’s different from other types of gestures, since it’s not related to position of your hands at the moment.

And ‘Fingers’ is a type of interaction detected by ‘relation’ between fingers. For example, you can create a trigger like ‘if your left index finger and left thumb finger are within 10px to each other’, meaning those two fingers are touching each other.

3. What would it look like to onboard users to use hand gestures?

When there’s a new type of interaction introduced to a user, from peripherals like a mouse, keyboard, or game controller, to voice interaction, the time it takes for a user to get used to it varies. If we were to implement hand gestures as an interaction, what’d be the learning curve like for newcomers? For ease of imagination, let’s say we’re creating a smart TV app that you can interact with hand gestures, saving users from frustrations finding a remote.

It depends on how many gestures are necessary to interact with the app, but the first thing would be to let users know that their hands are a cursor and a TV remote. Visualizing their hands would help to communicate that. Similar to hover status with a mouse cursor, changes on the button when it’s ‘hovered’ with your hand would also help. Maybe that’d be the perfect time to tell users that you can use a certain pose to click it. And what’d be an adequate ‘click’ pose? Every user will have different mental images for this ‘click’ gesture, so it’s important to assign something natural for most people, and acceptable for the rest who didn’t come up with it at first.

As you start to assign more gestures for everything, it’ll get really confusing for users. It’d be essential to think about what are primary, mandatory gestures that users will need, and what are secondary, ‘nice-to-have’ gestures. And it’d be important to consider when / how we tell the user to learn these new poses.

4. Some sweet spots would be…

I’ve already alluded to this in other sections. Hand gesture is likely to be supplementary when it comes to small personal devices, like mobile, tablet, and PC, since it’s easier to use them with a touch display and mouse. But when users are leaning back in their chairs and couch, or slightly away from their devices to do some multi-tasking, like cooking, reading, drawing, and writing something, hand gestures can be a good option for simple controls.

While the use-case for these personal devices is quite a niche, it’d make more sense for a large display, like a TV or projector, or for multi-user intended devices, which often require additional accessories to interact.

In next article — how I’ve built it

In the next article, I’ll start to unpack more about the process of building Handzplay. I hope you enjoyed the tool and the article here, and hit ‘follow’ if you want to keep up with my articles! You can also keep posted via the following links.

> Instagram

> Linkedin

> website

Your hand as a mouse — exploring hand gesture interaction (1) was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

As a designer in an innovation lab in the IT industry, most of my projects are about envisioning the future and how technology can play a role there. As a team, we think about how work and life will change on the next horizon, picture ideal experiences, and bring them into life with some proof of concepts. It requires insights to see arising needs that might not be apparent at the moment, and optimistic perspectives to ensure we don’t limit our imagination within current technologies.

Ironically, as I keep picturing the ideal future with technology, at some point, I start to become really skeptical about what I came up with and have fundamental questions about this ‘tech-ful’ future. But I think it’s a natural, and even necessary process to have a ‘balanced’ point of view towards the future, not just an ‘optimistic’ point of view.

If you’re a futurist, technologist, strategist, designer, and engineer who are contributing to shaping our lives with technology in some ways, I do believe that it’s worth taking a pause from being innovative and optimistic and being skeptical about our visions. In this article, I’d like to talk through some of the self-questions I’ve been having.

Wow, this future sounds cool! But do you want to live in it? 🤔

The picture of the future has not changed much from the past. The way 80s sci-fi films like ‘A.I’ describe the future is not that different from what big tech companies like Microsoft picture their vision in their vision videos (except having a human-like AI robot with emotions 😂).

We imagine our future to be powered by technology in every aspect, automating everything, getting smarter suggestions from AI, and experiencing things beyond our imagination. For example, our day will start as the curtain automatically opens, with a coffee that’s automatically brewed based on my preference. We’ll consume suggested healthy food based on our conditions and connect with coworkers based in different countries in super immersive virtual reality, almost as good as meeting in person.

Sounds cool, but sometimes, when I take a step back to see the whole picture of the future we just described, I start to question myself, ‘Is this the future that I want to live in?’ Do we really want to live a life where everything is suggested by AI and we’re just following it? Aren’t we saying these as if we’re not going to be a part of it?

For me, brewing coffee itself is a ritual to warm myself up for the day. I enjoy talking back and forth with my friends to figure out what we want to eat together and know more about each other. We might be able to do anything in virtual reality without the hassle to go somewhere, but I don’t want to spend all day long in virtual reality, doing all the work, social hang-outs, watching movies, meditation, workout… literally everything there. I love going somewhere, taking a bus, zoning out while looking out at the window, and coming across new shops that I didn’t know. I trust my instinct to pick a book by cover from a bookstore, rather than all the suggestions based on my previous reading history.

And yes, the picture of the future we have is not the definite destination. It’s one of the speculative futures we can think of, and it might end up just as a picture, not our reality. But this question itself helps us to bring the focus back to ‘people’ at the center of the experience, not ‘technology’ that enables all those images.

Looking ‘backward’ is as important as looking ‘forward’ to picture the future.

While we think about how technology can make our life easier and more meaningful, sometimes we need to consider what will be left out in the new experiences that we design. There can never be the same experience replaced by technology. We should be aware of these missing values so that we can question ourselves if it’s okay to leave these behind or come up with a way to bring them back. Looking ‘backward’ gives us a clue for richer, more authentic experiences for the future.

For example, video conferencing apps like Zoom and Teams are allowing people to work and connect with others from anywhere. But the experience can never be the same. If we look back and see what we’ve lost compared to the time we always went to an office to work, we started to ‘time block’ all the communication, turning all the conversations into ‘meetings’ and losing a chance to have a small chat and build a bond with a team. We started to realize the role of ‘commute’ in a way it puts us into work mode.

These changes started to create new needs to fill up what we’ve lost. We’re starting to see lots of other services coming up with a better way to bring back those missing values. These services are looking backward to look forward and create a better future collaboration experience.

Technology can be an ‘enabler’, but also a ‘disabler’

We fix problems and get help from technology to back up what we’re not good at. We hope technology provides insight beyond what we cannot see. We imagine the future where we have full control over everything with technology.

Maybe I’m getting too philosophical here, but aren’t we trying to fix what makes us ‘human’? We’re trying to remove concerns, inconveniences, and uncertainties which, in the big picture, lead to our growth, various emotions, coincidences, inspirations, and serendipities. In the future where everything is in control by technology, isn’t it actually the life where technology controls us to live the life as technology guides us to?

Again, I’m not saying we should boycott all the technologies to stay human. But for futurists and innovators in the technology domain, I believe it’s important to spend some time questioning ourselves, what is something that technology is ‘enabling’ in your future scenario, and what is it ‘disabling’ in turn?

Closing — ‘serendipity’ as an antonym of ‘technology’

When there’s a huge shift in daily technology we use, there’re always people who miss the previous experience. There must have been people who reminisce about the time they spent to think what to write in a letter after we started to call or email people. I sometimes miss the time without smartphones when I was able to focus on the moment without distractions from notifications and my habit of constantly checking the phone.

We all know that this nostalgia doesn’t mean that the new technologies degraded our lives. But it is also important to think what’s the missing value in this nostalgia in current experiences so that we can consciously picture the future with it. I feel like ‘serendipity’ is the antonym of ‘technology’, that is constantly threatened as technology constantly gives us more awareness and control over everything. I believe that for everyone working on the picture of the future and technology, acknowledging the value of serendipity in our lives is as important as understanding technology. Or would we be able to create a nice joy and serendipity with technology?

Let’s take a pause to be skeptical, not to devaluate the potential of technology to improve our lives, but to create a better picture of the future with it.

Picturing a ‘tech-ful’ future with skepticism was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Start playing around with ‘interactions beyond tap’

A scope of interactions is going beyond simple interactions on screens — voice interactions, camera visions, 3d dimensional interactions, and so on — at some point, you might start to realize you need something more than visual design tools for that. That is the biggest reason I’m using Framer to have a playground for the actual ‘interactions’, not the visuals.

In this series of articles, I’m going to share some of the components that I’ve built, so that more designers can start to play around with diverse interactions by themselves. The first set of components is video/audio interaction components that you can use for your video/audio chat apps, voice commands, and so on.

• Audio Visualization
• Audio Interactions
• Voice Command
• Dictation
• Webcam (+)

Duplicate the project and play around with the components from there, or copy components into your project! You can also access the code of the components.

Link to the Framer project 👉

Framer Login

Inside the components

You don’t really need to know the code part if you’re fine with the properties that components provide. But if you’re interested in tweaking the code to better suit your need —I’m highlighting some details about the overall structure of each component. Hope this helps Framer / React noob and designers to understand how these components work.

🎤 Audio Visualization and Interactions

Audio interaction is becoming prevalent. From a voice command to a video /audio chat app, you’ll need some visuals that react to what users are speaking. These components can be a starting point to create similar interactions like the examples above.

Just to give you a rough explanation of how ‘Audio Visualization’ components work, we’ll be using frequency data from audio and turning them into visual. The code below is returning dozens of values from different frequencies in real-time, and the values can be used as the height of audio bars, or the scale of the shapes.

If you’d like to understand more about the code used for the components, you can check the web audio API document here. https://developer.mozilla.org/en-US/docs/Web/API/AnalyserNode/getByteFrequencyData

In a similar way, what the ‘Audio Interaction’ component does is rather than translating the changing value of frequencies, it triggers the animation that you’ve assigned when a certain frequency value gets larger than a certain level.

🗣 Speech to text — Voice command / Dictation

If the animations above are not enough, you can add transcriptions. From picking up the voice command to transcription of something longer, this gives clarity to users whether the device correctly picked up what they’re saying.

And with the voice interaction component, you can assign specific command words that will trigger the interaction you assign.

I’ve assigned ‘hello’ as a trigger word, so as transcription picks up the word, it’ll then execute the interaction that I’ve assigned — moving onto the next page. It’ll get complex if you’re trying to build a voice assistant with this, but will be enough to bring your idea to life for simple voice interactions.

Instead of building something from scratch, I’m using a library called ‘react-speech-kit’ (https://www.npmjs.com/package/react-speech-kit). This allows you to do ‘text-to-speech’, or ‘speech-to-text’ with just a few lines of code.

As a noob coder who started as a designer, if there’s a complex interaction that I’m not sure how to create, I always check if there’s a library I can use, and most of the time, yes, there’s the shortcut you can use.

👩‍💻 Webcam (+)

To be honest, there’s not much added to this component compared to the basic webcam component provided by Framer. But for the project I’ve mentioned above, I needed multiple camera inputs, so I wanted to choose a specific webcam input from my device, which I couldn’t achieve from the basic component.

In the project file, you’ll see a list of available virtual cameras and webcams from your device, and if you need to put something other than your default webcam, you can basically put the name of the camera you want to use. So now you can choose and use multiple camera inputs for your prototype.

FYI, I’ve used Camo (https://reincubate.com/camo/) to connect my iPhone as an additional webcam, along with the default webcam on my laptop.

Mix & match the components!

Each component is responsible for small interactions, but as you combine them, you can create more interesting stuff. For example, what you are seeing below is a combination of the webcam component and some audio interaction components. As I speak, I’m using some opacity/scale transition.

Final thoughts…

The ability to create new interactions and turn them into components with controls and properties is quite compelling, especially to give other designers chances to play around with actual interactions. I hope these components are a good starting point to play around with interactions beyond tap for designers!

As I’ve mentioned above, I’m planning to use this project as an archive of components I’ll build beyond simple interactions. So if you’re interested in what’s going to come more, keep an eye on more components to come, and let me know if you have any questions/suggestions 😘

Create interactions beyond tap — use camera/audio inputs was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Lovely, creative, and useless websites for your inspiration

As an interaction designer, it’s a pure joy to find a great website with great storytelling, motions, and graphics. But personally, there are ones that stand out more to me — the useless websites created without the purpose of serving businesses, promoting oneself, goods, and services, more for experimental purposes. The successfully useless websites ended up making me stay on the website longer to try them out or registered themselves with me for a long time.

Those are some of the treasures I’ve found, and I’m planning to keep this list updated when I find a new one. While I started writing this posting, I realized many of them I used to know are now closed, which is understandable since most of them are the result of pure experiments, not for profit. Hope I can encounter many more fun projects, but also keep seeing them live.

> last updated : Jun 27, 2022

PointerPointer

This is my favorite one, and honestly, I want to meet a creator of this website and ask what made him/her come up with this idea, and how it was built.

As you leave your cursor on any part of the website, it’ll pull up a photo that someone is pointing at your pointer. I’m sure you’ll end up staying on this page for quite a long time!

pointerpointer.com

Cybeer

This has been one of my favorites, but seems like it’s down :( But to keep a record of it, when you go into their website you’ll see a beer tap, with an empty beer glass. And it will give you a QR code, and as you scan it with your mobile, now you can pour beer into the empty glass that you see via the mobile! Yeah sounds fun, but you actually get nothing.

Paper Planes

I’ll speak highly of this one for its unique interactions and experiences. You’ll see a small planet with flying paper planes. They’re messages from random people out in the world, and you can catch them to read, or you can throw one by yourself. It’s a 21st-century version bottle message.

Paper Planes

Drive&Listen

I should confess my standard of ‘useless’ is vague, and this might look somehow quite useful. If you’re down for something random as your WFH playlist, what about heading to your dream city on a ride? I actually stayed on the website quite a long time.

Drive & Listen

Another one with a similar concept — WindowSwap allows you to open a new window somewhere in the world.

WindowSwap

Checkbox Race

This reminds me of the chrome dino game with its simpleness, but I’d give more points to checkbox race by the fact that it brought familiar checkboxes into a part of the game.

Checkbox Race

Receiptify

It’s inevitably lovely when digital meets analog. Still in a digital format though, but this website prints your log of music on a receipt.

Receiptify

At last, the Useless Web

Yes, the title of this posting came from this website. This site will lead you to a useless website in their archive. Unlike mine, their curations are better in terms of the ‘uselessness’ of the websites, so check it out!

The Useless Web

Closing

For me, those useless websites were actually great sources of inspiration and references. Their lovely interactions that keep users on their sites somehow give a hint of what people need or look for.

We’re seeing more and more dynamic digital experiences with webVR, AI/ML, and visually creative experiences made with p5js and THREE.js, so looking forward to seeing more of those useless, but creative and lovely websites!