Weaving the web – Programming textile-based interactions in JavaScript by Charlie Gerard

Weaving the web – Programming textile-based interactions in JavaScript by Charlie Gerard



so first of all thanks everybody for being here there's actually more people than I thought because when talking about text log based interactions in JavaScript it's always a bit something like it's not something that you're gonna go back to work with and say let's see I quit my job I'm just gonna do textile interactions so I'm actually surprised with the amount of people that's here today so thank you so much before diving into the topic of text analyst interactions I thought I would give a little bit of context so let's say it all started five years ago when I started using javascript to do things outside of the browser and my first experiment was to control a sphere or robotic ball with the movement of my hands over a leap motion the new motion is this device that you plug on your computer and it detects the movement of your hands and I was able to kind of like move around and control the movements of this ball and as I said it was the first time that I was using JavaScript outside of the browser and this is where I started playing around with all different kinds of human-computer interaction then I moved on to things like using a conductive ink on a canvas to trigger projection mapping and it was just another way of using touch you know not not at all using any keyboard or any phone to trigger stuff but then instead of just the body I tried to use Arduino and hardware to trigger things with without something or with something else in my body so I'm just hiding the light here but it's basically using the flashlight from your phone to trigger light sensors that trigger all the lights that trigger licence is like a chain reaction my goal for this was to recreate an installation that I saw all of this was actually done in JavaScript and then I also moved on in interactions with the browsers using machine learning and had motions so I was really interested to see if I could use motion recognition and machine learning to allow people to communicate without the movement of the head so again no keyboard and no phone just being able to really quickly train a machine learning algorithm and have different kinds of interaction that type of experiments also led me to play around with brain computer interfaces where I was able to interact with the browser using brain sensor and thoughts Oh javascript and more recently and a bit more fun I kind of transferred my living room into an immersive game using the web Bluetooth API and JavaScript where I attached my phone to a normal skateboard and using the accelerometer and the gyroscope being able to kind of like I just moved around on a real skateboard and was able to actually play around with a game and the purpose of that particular example was to be able to use things that are available for everybody and just transform that into something a lot more fun than just a website so my name is troy rod I'm a developer Atlassian I live in Sydney Australia what I do during the day is not at all this I just you know work on a product that you might be using I just started last week the product I work on is JIRA I don't know if you know Hey all right I mean some of it some people hate it some people love it so you know don't even talk to me about it but I just started it's pretty cool at the moment but outside of my job I'm also a Google developer expert for web technologies and the Mozilla tech speakers so these two programs in the community actually are groups of people who like to share knowledge online through blog posts or conferences and you have access to camera like some exclusive news from Google and Mozilla and you're able to you're supported in sharing knowledge in the community so if you're interested in these programs you can actually apply online it's open to everybody but most of all as I said before I love experimenting with human-computer interaction I really believe that the keyboard and the phone is kind of a very restrictive way of playing with technology and I'm really excited about the things that we could be doing but for some reason we still kind of like using the keyboard but today I'm going to talk about textile interfaces if you've never looked into this maybe that doesn't really make sense so I'm gonna talk about an example so usually a lot of the time or well let's start with the definition first but it's basically using textiles so the stuff that you wear or a textile all around us to interact with interfaces or and devices so an example of that is usually wearables are seen as the output of the interaction so you might have seen this example but it was in 2016 for advertising campaign for Vodafone I think where the goal was to create a jacket for cyclists that would help in terms of safety so the jacket would connect to Google Maps on your phone and it would have LEDs at the back that would reflect the direction that you're about to take so that the drivers behind you would be aware of your direction and hopefully not run over you but this is an example of there's a few example out there where the wearable is actually embedded with LEDs and it becomes the output but what I'm interested in is using wearables as an input so I'm gonna hopefully there's a sound will work it is project jacquard I don't know if you've heard of it so I'm just going to play the video it's only a minute and a half come from unknown getting next direction turn left on June 15th Avenue texts from Emily I'm running late see you at the shop you are five minutes away and we'll reach your destination by 7:05 a.m. so basically you can see in this video that the jacket is the way to interact with the phone rather than the other way around and you can control things like hang up a phone call or changing music will get you next direction and I'm talking about this one especially because it's been the main inspiration for what I've been working on and for this particular talk so why am i talking about this so I know that as I said it's not something that you're probably gonna you know go home and and do but I think that it's still important for a few reasons interactions are evolving even if it's not there right now we can see that for example voice your eyes like you might have a Google home or an Amazon Alexa so we are starting to actually communicate with devices using our voice maybe at the moment people are not really using them because maybe you think that they're not that performance I don't know if you've been following Google i/o but Google assistant is becoming actually quite impressive if you have not looked watch the videos from Google i/o you should really check it out the demo about Google assistant it's pretty impressive but it's not only voice your eyes for example at the airport more and more they are avoiding your contact with people and you have passport checks with face recognition so you're not really interacting with people anymore it allows you to go a lot faster and even in China sometimes you can play with your face as well so facial recognition is using is being used more and more as a way of interacting with in energies so maybe take star interactions are not there or not that evolves right now but I do believe that there is a lot of potential and it's coming and yes so there are exciting possibilities I put together a few examples in this talk that I'll talk about right after and I think there's even more possibilities than I thought of when I started working on that talk but after doing quite a lot of research I realized that there's applications for it in so many different domains and then finally it's important to me because I like to demystify some stuff that you can see so project jacquard I came across it maybe three years ago and my first reaction was like how does that even work I don't understand it seems so complicated and stuff and then after spending some time thinking about it and reverse engineering the way they did it I realized that it is possible and it is possible by a lot of people you don't have to millions of dollars you don't have to have a team of researchers or a PhD or whatever there is actually material that is available to a lot of people not too expensive and it's all about reverse engineering the way it would work and actually this information online all around so I'll show you a bit later what I did and how to build it so how does it actually work so we saw in the example that you can interact with textile but yes how can you actually do that and it's just kind of as simple as this and what this is is conductive thread so if you think about normal thread that is you know in your clothes basically it is thread that is mixed with stainless steel strands and stainless steel is conductive so mixing the thread with stainless steel makes the thread conductive and what is really cool about this is as it is conductive you can use it to create circuits so an example of that of that is this illustration where instead of wiring components together with wires that are not always that flexible you can actually wire everything together with thread which makes everything really flexible and something that you can actually wear so in this example you have three different LEDs connected to a power module and all of this can actually yes use thread in between the components to create the same kind of circuit that would be created with just wires but it's not the only thing that's important about this concept it's also the fact that the concept of electrical connectivity so this diagram or like picture describes more the way a capacitive spring strains work I couldn't really find one that was about textile but it's the same concept so when you think about a circuit where everything is assembled with conductive thread there is current going into that circuit and as the body is conductive when you touch the thread there is like a really really small electric charge going through your body to complete the circuit so you don't feel it because it's really small it's not the same way as if you are putting your fingers in a PowerPoint it don't do that of course but it's it kind of works the same way it's like you your dailies really does go through your body to complete the circuit and what and what me what it means is that if you read the current that goes through certain things when you touch the the conductive thread you actually have a drop of both in the current that goes through so it means that you can track that you can know the difference between am i touching the thread or am i not touching the thread and as soon as you can see this difference between touch or not touch then you can use that data to make something interactive like anything that you want so different applications so this is where you can see that it has potential in a lot of different fields so first of all augmented furniture I'm really excited about this one because I feel like my house could be a lot more exciting so this particular example was called the emotive couch it's developed by a team of researchers so it's not something that you can buy but I think the concept was to be able to understand the emotion of the user based on how they touch the couch so if you move like you're annoyed then it would dim certain lights to calm you down or if it feels that you're laying down on the couch then would maybe turn off the light because we're about to sleep or whatever so if I wanted to embed my couch with stuff that wouldn't really be what I want to do what I what I'm more excited about is to eventually use my couch as remote control so instead of having a remote control I would just be able to like touch the side of my couch and turn my TV on or off or switch channel or put the volume up down turn on my light whatever and all of this is already possible and it means that yes you wouldn't have you would have I don't know you would waste less plastic because you wouldn't have a remote and it would be in the couch and I think that's you know pretty cool yeah so that's just one example then there's also accessories for your house as well but this is already available actually this speaker is a kind of like tangible yeah texture interface I think I'm going to show you the video quickly because it explains it whoo this is really small explains it a lot better whoops yeah that should be alright I'm not gonna play the whole thing but you'll see what I mean as you can see you can just touch the textile on the speaker and you have different types of interactions and the fact that this is available in the fact that you can buy it online is there I don't know if I'm gonna stop there if you want to watch the entire talk there's there's more to it they basically there are sensors in her gloves different types of sensors flex sensors or rotation accelerometer and stuff like that it's all put together in in the fabric and you're able to create a new type of musical instrument so we can think about music a different way than we have before it's like your body becomes the musical instrument so if you're into music I think there's quite a lot of potential out there and the clothes that in this video that was quite a few years ago and it was just a prototype but I think that now is actually going to become available to you by this photo is like really pixelated and horrible but I couldn't find something that would illustrate better what I'm talking about but in terms of health monitoring you could use clothes for newborns to detect the way they're moving so for example when they sleep you would know if they're sleeping on their back or you know on the tummy or whatever you could have clothes embedded with hot sensors to make sure you could track the health over time and and I'm talking about newborns but I'm sure that you could help also other people and yes it is where you could see also dependent the potential in terms of health remote communication so this one is not my favorite but I you know it's there so I need to talk about it it's called the hug sure where you have different sensors all around the body I think it was made at the beginning for like long-term long distance relationships where you could send a hug to your partner or things like that I see more potential in the relationship between a doctor and a patient that's remote so you may be in London you might not think about it but in Australia you have a lot of people living very far from any city or any Hospital and if you actually had the chance to be able to communicate with your doctor and tell them exactly where it hurts or how much it hurts you could actually have maybe or you know eventually a better diagnostic of what is wrong so that could help in that way this one accessibility you might have seen it as well that's a really cool example so I'm also just gonna show the video instead of hello my name is Thomas and this is Navid we are inventors in the Lemelson m.i.t student prize competition so yeah I'm probably not gonna show because I probably need to move on but you can watch it later if you want I think you understood the concept it's sensors embedded in gloves as well I think it's correct it could probably work the same as the gloves for a musical instrument where you use the Flex of the fingers and the movements and you probably get all that data you fit it to a machine learning algorithm and you're able to recognize who to translate movements into into words and being able to communicate with people this one is not conductive thread but it's a really cool type of textile where it changes based on the amount of pollution in the air so the color changes its textile and you add a certain chemical on to it and in terms of being able to understand your own environment and make conscious choices I think it's pretty cool that you become more informed about your environment and instead of just walking around the city and not knowing what you actually breathe you could actually see it on your clothes maybe you don't want to but at least you'll be able to make better decisions about the type of air that you breathe and things like that and finally final example I'm going to show that's a bit more extreme it's around a human body augmentation where I don't know if you seen this talk there's a project called neo sensory where it's a jacket embedded with a vibration sensors to well understand your environment where the speaker in that talk was actually pulling tweets from Twitter and trying to get the emotion of the tweets and he was sensing on his jacket if the audience was happy or sad or angry and it's like a way to add senses to people so that seems a bit extreme but when you think about it some of you might have smartwatches and it vibrates when you get a call when you get a text or when you get an email and you might not realize that but it vibrates differently depending on what the kind of notification that you get so in a way we have already started training ourselves to understand vibrations or adding a certain like kind of sense to our body anyway so I'm gonna talk about what I've been building because I know that we saw these stuff that are pretty cool mine are actually not that cool but still I got somewhere so the first prototype that I built was based on one of the early prototypes of project Jack where I think it was commercial case at a conference where people could interact with the fabric and see your visualization so I was quite intrigued by this particular example because I really know how they would get the right X&Y axis and stuff like that but after working on it I kind of made my own that looks like this so it doesn't look as good but it kind of works the same way so on the right my hand is on a piece of fabric with conductive thread and on the computer is a browser and that you know has just some 3d using the three J's framework so there's there was quite a few limits with that one that was really my first one so the thing is kind of like there's like shiny front end and in the background it looks like this so it was really horrible because it's obviously this cables everywhere it's not portable like it was really my first my first thing and I just wanted to be able to validate the concept you know is it possible with almost no resource so yes so there was a few limits with that so obviously a portability was really bad because well I had cables everywhere and there was no way I would have been able to travel with this there was a problem in terms of connectivity because I used an Arduino Uno that had to be plugged in to the computer to be able to upload the sketch and things like that so I didn't have any Bluetooth Oh my file and latency because I was using the Arduino I actually was sending the data from Arduino to another server and then using WebSockets to the browser I was variable like it was going through so many layers that there was a bit of latency between the touch and the response on the screen so then I moved on to my second prototype that I did bring with me and my main change is that I changed the board I changed from an Arduino to an ESP 32 which is all you know arduino compatible so you can still run arduino sketches but it was better it kind of fixed my problems because i it's it has bluetooth so you can communicate via bluetooth and also some of the pins from the board have kind of like a touch sensitivity already so you can track the the capacity like it's yeah capacitive touch without having to write your own program so they have using the SP 32 they have a function called touch read that gives you the amount of capacitance without you having to write anything more so yeah that was a main change that really helped me so in terms of you know kind of like high-level architecture you have the fabric on the left with the conductive thread connected to different things of the SP 32 I have already small Arduino sketch that I upload on to it and then you kind of broadcast as a Bluetooth device and that can be picked up by any other device it can be a computer a phone a light bulb a speaker or whatever and all of this can be done in JavaScript because of the way Bluetooth API so you can connect directly to the browser no need for a node.js server anymore and I'll show that a bit later so in the end it looks like this so there's like I kept the embroidery hoop around because it was easier to transport but usually you would remove it but it looks so the wires connected to the board are actually not necessary you could use the you could kind of link the thread straightaway to the pins the issue I had is that my little my needle was too big to go through the pins so I had to actually switch that to to use cables instead but you know I would have loved to actually know how to use cables because then the amount of material I need would have been a lot smaller so if you think about a before and after that's kind of like a lot better it's just my second prototype so it could be even better later I could find like a smaller board or make it better but this is that it fixed all my issues it is all of sudden portable there is connectivity by bluetooth and there is a lot of less latency because I just couldn't extroyer way to the browser so let's talk a little bit about demos I put together a few things I don't know if you have noticed but you actually you have been in a live demo for the last 22 minutes because I have barely touched my computer I've actually been switching my slides with a piece of fabric that I brought with me so basically I treat a switch side and Magic attached a piece of fabric and it just switches my size something that's kind of like my first demo what everybody wanted to do is embed it in my clothes but for I mean I wanted to switch my side with my pants you know that would have been more fun but for a lot of reasons I wasn't able to do that I didn't really have the time and I thought I would do it on the way here but you can't bring scissors scissors or anything on the plane so obviously I didn't so I brought what I had and I wanted to see something else actually forgot yes at the moment it is connected to my computer just because same on the plane you can't get you can't bring with you certain type of batteries and the batteries that I had at home to connect to that were the type of ones that could explode in planes so hope you didn't bring them so it is connected to my computer just to have some power but usually it would be totally portable I could be over there and switching my size and you wouldn't really matter but the second demo that I put together is being able to control Spotify so I put a video there because I'm not sure it's going to work it's gonna be flaky especially I tried to do it just before coming here and for some reason it wasn't really working very well so I have a backup video but I'm gonna try to exit and alright anymore okay oh I should have actually my bad I forgot that I need to upload this before don't look at my if statements they're horrible let's look at this okay so it's gonna do it's gonna take a minute it's gonna take a minute to upload so that's just a little bit annoying I'll go through this piece of code a bit later unfortunately I have to talk a little bit about Arduino because so far you can't do absolutely everything with with JavaScript if I had picked another board Arduino you can use 25 without we know that allows you to do most of it in JavaScript but for some reason I tried an Arduino Wi-Fi board and it just wasn't working with what I wanted to do so I had to move on to the ESP if it wants to upload it's almost there come on he will do it but I don't know why I think it's taking so long oh this is not feeling so that's better than nothing no no not yet Donnell pudding cool all right so I'm gonna put that here hello all right I'm gonna I was a singing to Queen just before coming here so you got you're gonna have some Queen but if I turn on my server and so yeah so what you have to imagine I'm gonna step in there cuz you got you got the concept but so of course right now it doesn't look like okay right I'm doing as a demo but in my house I wanted to try and do it on my couch so of course this is what I brought with me so it's not embedded in my couch but it works and that's what I would have liked to do in the end is being able to just sit there and be like dinging and be super lazy and just do stuff so you can embed it with Spotify but you can you can do whatever whatever thing is that is connected to to the Bluetooth or the Internet can be controlled so let's talk about code samples so the let's say that you just want to track if one pin in is conductive or like you know the amount of capacitance that goes to one pin so you in Arduino if you've never worked with it before you have a setup function and a loop function the setup is just what runs once when you flash the board and loop is what will run over and over again so what I'm just doing is opening a serial communication on a certain port and then on in the loop function I'm just checking that you know if the touch on a certain pin is less than the threshold it means I'm touching and I'm just sending that to basically to my let's say terminal in Arduino just to check that it's working fine the thing is that this is just locally I'm not sending the data to anywhere so what we move on later is being to send the data of a Bluetooth and if you don't understand all of this it's fine because it's actually coming from one of the example that you can use out of the box so you don't actually have that much to change but basically you just have to declare a few variables you have to have new ideas for services and characteristics if you actually don't know how the Bluetooth protocol works you it works with services and characteristics that you can subscribe to to get data then in the setup function we just give a name to our board so I just called it my ESP 32 but it can be anything that you want then you create a server you create also a service that you add to this server because this is what you're going to be able to actually get in the front end later the characteristic you just have to give it a few more parameters you have to kind of say do you want to read it write it notify it indicate it and things like that then you start the whole thing and you start the advertising for the front end to be able to figure out that there's a device that it can connect to so this is pretty standard you will barely change any of this it's in the loop function that you will change stuff because you're saying again if the you know the data that you get from that pin is less than the threshold you just store it in an integer set this value in the characteristic and notify it so that the front end can have it so this is you know Arduino but the cool thing is the fact that you can get in the front end so in the front end you declare variables as well with the name and the services and characteristics ID which are exactly the same as we as the Arduino code and then using the web Bluetooth API you request that that particular device you connect to the device and you get you know promise back with the server and you ask for the primary service then you get the characteristic the characteristic that we defined at the top and in the end when you manage to get that you're going to start notifications so that every time the value changes from the Arduino you're able to get that in in the function you know that I called handle notifications but it could be anything and in here you actually get the data coming from the Arduino and this is in this part that you can do whatever you want so as soon as you touch then you can trigger something on the screen or on Spotify or really whatever you want so this is pretty cool that you can go from just fabric to the browser but there's no of course a few limits there's always limits so it is partially waterproof so if you like from the Google jacquard project jacquard project oh yeah they say that you could wash it maximum of ten times so they had the I did I guess a special coating on top of the fabric I didn't do it with mine because I didn't really know what type of product I need but usually you can actually add a certain spray on the thread to make it more waterproof so it should be alright but if you do something at home on a couch I don't think you really care about the rain it's more if you do a jacket yes you will care apparently conductive thread works for two years maximum so I think if you want it to be quite performant it will be it will work for two years and I don't really know what happens after maybe it's just not really performant after that but that's why if you're thinking about embedding that into things instead of putting it straight away in my couch I might just you know you know yeah just sew it on a patch and other on my couch so I can change it every two years the fragility part I think is more just my fault because I did everything behind and I think if I had a machine it would have been more robust this is where you know you kind of start to see the difference between doing stuff at home and being Google and Google had like a team of like awesome stuff and when I just had me so it's fragile because because I'm not very good at this needle it was the first time I was doing that stuff so that's probably why but of course I was really scared coming here that it would fall apart but it didn't so it's actually not that fragile and the capacitance changes with external factors and that's quite interesting because it means that depending on the level of humidity your threshold is going to be different so maybe here I would have to you know adapt the threshold compared to Australia because rain but you know like it does depend so even if you do it at home you're probably would be fine but if you do a jacket you would you might have to change the threshold every day I don't know I haven't tried but it's quite interesting you have to keep that in mind so what about what is next though because there's always more things you can do so I would really like to try to put it in a jacket because I feel like so of course Google built their own custom boards and I would be interested interested in doing that being in of being restricted by something like that I have to find online I could actually I know that I could make my own which would be really interesting so that's like one of the things that I would like to try that's all the type of materials that would be really cool to play with so you have a material that changes with the amount of UV light so that would be I mean in Australia at least that would be quite interesting to see if you could have a t-shirt that changes with the amount of light skin cancer is a massive problem in Australia because of the how harsh the Sun can be especially in summer so if you were reminded to put sunscreen on when it's you know too bright or something it could actually save lives potentially this one is really cool because it's actually so you have color-changing textile but it works with electrical current so there's a certain so it's textile augmented with thermochromic pigments which means that when you apply a certain electric charge to it it will change color so I don't have to really see on there I have a video in my resources where you can see a lot better but the textile can become some kind of screen based on the amount of current that you put through it so I have no idea what I would do with this but I would love to try and see how it actually works one thing that is probably a more accessible is stretchable textile where you can actually get the value of the stretch and yeah it's like so that is available you can you can buy it online but it could be interesting for maybe athletes being able to control their or to check their performance based on how much they move or even for your posture if you were a teacher and it sees that you you know start to kind of like bend and not have like a straight back you could remind you like hey make sure that you sit straight I've seen some devices that tell you oh wear that on your back and it will tell you if you stretch or not but I think what I like about textile is that you wouldn't have to add something rigid you know like a Fitbit or something that is quite just not comfortable it would be part of your clothes and you wouldn't even think about it but it would still be tracking what you do and one thing that I saw just before I came here and I added it is that you can actually do some really really cool stuff with stretchable textile stretchable just textures in general so I'm just gonna show you quickly just a part of the video where you can see what they've been building that's so that's yeah that's researchers again but still we propose I'd love for interactive hand posed capture its stretch sensing fully soft thin easy to put on and comfortable to wear it can capture complex hand poses with high accuracy and as an interactive framerate without requiring an external optical setup our glove successfully captures hand poses in situations where camera based trackers fail for example when a collusion occurs due to object manipulations or body occlusion or self occlusion or the hand moving out of field view also changing light conditions or motion blur due to fast translational movement are challenging for camera based systems but not our sensor based glove we get so again I'm not gonna play the whole thing but I think that there really captured the real benefits of looking at interactive textiles it's like it's really thin and flexible you can really bend it in any direction and it won't really break and you can really do pretty interesting stuff with it one other thing that I would try to do is what it's called electrical tomography where instead of having conductive thread everywhere you will actually add capacitive sensors all around a surface and by checking the changes in capacitance in all sensors like you know one after the other you would actually have a pretty accurate X&Y position over a surface so there's more of like a nerdy thing that I would love to be able to do it's not I don't really have any application for it right now but it means that yes instead of having to put you know thread all over a surface you would just have to add sensors and then use math to figure out the position of the of the finger so you could kind of like recreate the trackpad pretty accurately but on a piece of fabric so I'm almost done I may be early but so I have a few resources that I put together if you're interested there's quite a few people actually doing projects like that so there's a few tutorials as well and I kind of wrote a blog post about my first experiments with it I can share the slides later if you want but some people are doing really pretty interests interesting stuff way better than just my you know Spotify track thing but yes so basically and that was it about textile interfaces so you know it's fine if you don't go home and start playing with that and my main goal was to make sure that it kind of makes more sense now that you know how to build that type of stuff and hopefully you kind of see the applications that could be done a lot of the stuff I've been Makena have actually been like in progress for a few years now so people have been working on that for a few years and even the Levi's jacket was actually available I don't know if it's still available but you can buy it online so people are looking at embedding stuff into textile anyway if you have any question I'm here or I'm on twitter at def def charlie I often you know playing with that kind of stuff if you have any ideas that you would want me revealed and can do that too but anyway thank you very much [Applause]

Leave a Reply

Your email address will not be published. Required fields are marked *