Fronteers — vakvereniging voor front-end developers

Hands-On Web Audio by Soledad Penadés

Transcript

[APPLAUSE] Cool.

So I'm going to-- I've been speaking NOTE Paragraph about Web Audio for a while, but I think the issue that we have so far is that I always tell people like, Web Audio is so cool.

But I never show them how cool it is.

So I decided instead of saying that it's cool, I'm going to show you how cool it can be, and what you can do with some WebGL and Web Audio.

So I'm going to show you my into.

[VIDEO PLAYBACK] [END PLAYBACK] [APPLAUSE] So it could be actually nicer, but it does reduce the number of polygons of everything.

Like this has not been pre-rendered.

This was just happening at my computer in real time.

Actually, the whole slides overall are, like, one megabyte and 200 kilobytes, and generates audio for this, like the raw audio itself would just be, like, two megabytes of data.

But just happened on my computer.

And this is like super high-- you can't really see well, but this is retina kind of quality.

So downloading this kind of video over the internet is really expensive, but if you use WebGL and Web Audio, you can generate all these things on your computer, and it's way more effective bandwidth-wise.

So hopefully at the end of this talk, you should be able to understand how this is built, and maybe get super excited and build-- sorry.

I'm [INAUDIBLE].

So can you hear me now? It's too loud now.

At least we're not like bom, bom, bom.

So hopefully you should be able to do this thing, and maybe like build experiences like this.

So I work in Mozilla, and I am focused on the platform team, which means that I go through new APIs that are available to people.

And maybe they are not using them yet, so I try to look at why are people not using these kind of things yet.

Is it because they're complex, is it because they're not readily available, is it because the docs are horrible.

Why is this? And then I try to make sure that the APIs are usable and people are using them.

So one of those APIs that I'm working on is the Web Audio API.

And it's pretty stable.

It's been on browsers from like-- it's pretty much available everywhere.

And yet, there's not many people using that.

So we're trying to fix that.

The cool thing about Web Audio is that it's pretty small API-wise.

There is what I call, like, small vocabulary, and because there are many few things you can learn, learn new concepts.

Also, there are few grammar rules.

If you say that Web Audio is kind of like a tiny language, NOTE Paragraph [INAUDIBLE] is that it's pretty easy to learn a language that has pretty few things to learn.

And also what is also important is that if you want to master a language, you need to master the items.And in this

case, they are just, again, very few items you need to master.

So once you learn those little things, you're like on your way to becoming a Web Audio master, which is pretty cool.

So if-- has any one of you done, like, any Canvas programming? You might be familiar with the Canvas context.

So in this case, we have the audio context.

But instead of doing as we do in Canvas where you get the Canvas and then you say, get context 2D or 3D, to just create the audio context with new audio context.

And yesterday I was complaining on Twitter, because Safari still has Webkit prefixes of their context.

So it's like oh, everyone has this fix on this thing, except Safari.

But you don't want to keep using WebKit audio context examples.

So there is a polyfill for this.

So you can start writing code that will work from some years forever.

So once you get that thing, you can start using the context as a toolbox, as well, or what Java programmers would say the instance factory.

And so you can start getting things out of the toolbox.

So you say ac-- which is the audio context-- create oscillator or create gain or create any of the other things that you will see later.

And there's a number of things you can do.

And I separated them in three kind of things you can do.

You can generate sound, you can manipulate the sound, and you can analyze the sound.

When I say sound as this abstract thing, this is actually numbers, because computers are digital.

So when we as humans listen to sound, we are actually listening to waves hitting our eardrums.

But computers cannot really actually hit our eardrums, so they need to figure out a way to go from nothing to something that we can hear.

So what they do is digitize sound.

And they represent that thing as-- how do they call it.

They represent that as a number of very, very, very highly, quickly sample sounds.

And it ends up being, like, a bunch of numbers.

And what these nodes do is just manipulate numbers in different ways.

Some of them create sequences of numbers, and some of them took those sequences and changed them in some ways to alter the sound.

And I'm not going to make a whole course of digital sounds processing, because that would take us way more about 45 minutes.

But that's generally what we want to do.

We want to take numbers and move them around and generate sound with them.

So the way we do that thing is making the numbers flow through the audio graph.

And the audio context is the one that contains the audio graph.

So to create a graph, you need to connect things together.

So you need the Connect method, which is in every node that we create.

So that's what enables us to create modular routing.

And what that means is that, instead of just having a thing that does all the functions you possibly need, what we do is we have a number of nodes, and each one of those does a very simple thing.

Like we have a node, like the oscillators, that would just generate sound.

But they just generate the sound.

If you want to change the amplitude of the sound, you need another node to change that thing.

And so you will start combining things together to build something which is bigger than the individual smaller nodes.

So that's-- and if you look at any picture of early synthesizer people, you will see them surrounded by lots of things and pluggings and cables.

That's what modular routing is.

But we're doing it now a nicer way, with just like connect and functions and nodes and objects.

So it's way cleaner.

So one of the simple nodes that we can build is just taking an oscillator and connecting it to the destination.

And that's just a two-nodes graph.

And the destination is a representation of the speakers.

So there is only one destination per audio context, and it would sound like this.

So that's pretty simple.

It's just an oscillator that goes to the destination.

So that's kind of boring, but it's the start.

You need to start somewhere.

So then let's suppose you want to do something a bit more complex.

So if you have a Mac computer, you might be familiar with what I'm going to play you, which is a major thing.

So what I'm doing here is I'm creating a number of oscillators.

Each one is just in a different frequency.

And you change the frequency because you want to change the node.

And so once we do that thing, we connect every oscillator to the destination, and we play-- [MUSIC PLAYING] NOTE Paragraph [INAUDIBLE] sound.

And so that's cool, but maybe [INAUDIBLE].

Today, you can get sounds that are a bit more raw than what I just played to you.

So what musicians do, or what sound designers do, they take the raw sound and they start plugging things in the middle.

And one of the things you can connect is filters, which would take kind of like this sound and say, well, out of this spectrum, I'm just going to let these frequencies go through.

And that's how you alter the way the sound feels to you.

So in this case, we are creating also filters.

We have equal filter, which is another of the nodes we can create.

And we connect them in the middle.

So we have the oscillator, the filter, and then everything goes to the destination.

So it's the same sound done before, but a little bit richer.

[MUSIC PLAYING] So you can see that we have all the filters in the middle, and the oscillator.

So it's getting nicer.

And we just used three types of filter, I mean of nodes.

So it's cool that we get this kind of rich sounds.

But that's not all the best that we can do, because a person has modular routing where you connect to simple units.

You can also do what we call sound modulation that we could also call using audio as input data.

So that means that those numbers tell you that every node is a meeting.

We can connect those numbers, that stream of numbers, to the parameter go for another node.

So as I said, instead of connecting one to the other, instead of getting one node to the other, the output from the previous node, what we do is connect the node to the node property.

So then what happens is that if I connect this output, it's going to halve the values of the stream to the previous value.

So if you have a baseline number, you can get this thing to oscillate or fluctuate or whatever.

So that's cool.

But the problem is that this is streams of numbers.

The things that generate sound only output from minus 1 to 1.

So if you want to change the frequency in a wider range, and I connect the output of this oscillator, you're just going to get nothing.

You can't notice what's going on.

You are going to-- it sounds slightly funny.

But I don't know if it's my perception or what is.

So we're getting to this to use the gain nodes, as I say, we use simple nodes and simple functions.

And we kind of like crash things together.

So we can take the gain node and connect the oscillator to that.

And then whatever we connect to the gain node is going to multiplied.

So we have this oscillator, which is getting something into minus 1, 1, and we connect it to the gain node, we're going to get something which is bigger.

So we created the gain with that.

And then if we create the oscillator, we create the gain, we connect the oscillator to the gain, and then we say, OK, the gain value is 100.

So the oscillator used to be emitting something in -1 , 1, but now, the output of the gain, which the oscillator is connected to, is going to get the output of minus 100 and 100.

So if you're connect the thing to something, it's going to get wider value changes.

So, for example, I could use this thing to change the volume of the oscillator by just moving the mouse.

[BEEP] So it's [BEEP] a function of my move.

So that's an example of how to use gain.

And then another thing you can do is use the thing, as I say, modulate the frequency of our oscillator.

So if I have an oscillator that is moving really slowly, and I connect to an oscillator device oscillating at a normal frequency, I am going to get the funky result.

[BEEP] So let me explain to you how this works.

The yellow one is the low frequency oscillator.

And the green one is the thing we are listening to.

So as I move the mouse up and down, I'm going to change the rate like how quickly the yellow one is going to be modulating and also and the depth of the modulation like how much am I going to be affecting the green one.

So the frequency of green one is going to change depending on what I do.

And with just two oscillators and a gain node, this is going to get very interesting.

[MODULATING SOUND] That was three.

So that's how you can build really complex things.

I was just using two nodes and the gain to multiply the value of the low frequency one.

So that's pretty cool.

And I think this is the most important thing you actually need to understand about Web Audio, because otherwise, you can be connecting things together, but you don't get the interesting things.

So remember, if you connect the null output to the prime input, you add the value.

And if you need to multiply things to make them bigger, you need to use a gain node.

And the cool thing about this thing is that once they stopped doing things this way, you don't try to change things with JavaScript, which is going to be way more performance.

And performance is something that's really integral to Web Audio.

And you have heard maybe that Web Audio promises that you can't have this low latency and sample accurate playbacks.

So things are going to be triggered at exactly the point that you say.

But you might be like I'm a JavaScript programmer.

That doesn't sound really realistic.

So the way you do it we give Web Audio its own thread.

But instead of letting Web Audio process our code in its thread, well, it actually processes only scheduled events.

So nothing that we program is going to run in the thread.

The thread is super pure in that sense.

It's just doing this processing of events.

And the way you get those events is via the node instances, because some of those nodes have properties like frequency.

And the frequency, those type of properties are actually audio parameters.

And those are the ones have set value time, linearramptovalue time.

So it's not simple values like .type or something.

It's a special kind of properties that will give you these special methods to actually end up introducing those events in the list of things to happen.

So you get those events by using those methods or by using methods like Start or Stop.

And this is the time that it should happen.

So if you want things to happen in Web Audio, you don't use a timeout or a callback or anything.

You just lay down all the things that need to happen and just let these Web Audio threads do the things when it has to do them.

And that's the way you can get really smooth things, because suppose you want interpolate between values from 0 to 880, if you try to do that thing with JavaScript, I don't know if you have tried this thing.

But it's going to sound awful.

You're going to get things like (SINGING) ba, ba, ba, ba, instead of vooooo, because actually the engine doesn't have time.

You're going to get all the junk.

And that's awful.

Just don't do that.

So if you make sure to use those methods, you're good.

And the other things that-- this is something we tend to do.

We tended to be, oh, I just make sure the oscillator is never there.

So I just make sure that it's always working and just have one instance.

But actually, that's a horrible thing to do, because then you're trying to do if this is that, then do this thing, then create these instances.

That's not very optimizable for processors nowadays.

You just want to lay down things and let the computer decide for you because it's cleverer than you probably.

So adding things to the graph, or removing things to the graph, or creating such things, this is all super-efficient.

The graph is super-optimized for this kind of work.

So people get worried about, oh, I just have to create 100 oscillators.

This is very expensive .

No, it's not.

Just do it.

And don't worry about that thing because the people that are writing and implementing, they suspect that you might need to create lots of sounds.

So they are trying to make sure that these things really, really, really, really perform.

So doing this thing is super cheap.

You don't need to worry.

The other thing you need to be really careful about is-- well, keeping track of state is expensive.

But you don't need to worry about that.

But you don't need to keep references to nodes because you might incur either own memory leaks or you might not remember or to stop and disconnect the things.

And the problem is that Web Audio doesn't know if you want to keep using that node if it works.

So it's not going to garbage collected.

So you have things which are running.

And you didn't really clear the reference.

So you cleared the reference but didn't stop them.

It's going to be running.

The oscillators are oscillating with a given phase.

I dont' know if you remember from your physics classes.

But sometimes, musically speaking, it's interesting to use a certain phase.

So Web Audio in the interest of musicality is preserving the thing running.

So you can always connect and be back to the way it was.

But maybe you don't want to do that.

Maybe you actually wanted to stop this thing.

So always stop and clear all the references so the garbage collector does it right next to them.

So apart from this lesson on how not to mess with your web audio thing, what else can we do? So we can play some samples.

We can do more than play in oscillator thingies.

Web Audio doesn't actually introduce any new thing for dealing with data.

It just uses whatever you can already use with JavaScript.

So what you need to do first is load the data, and then decode it, and the last two things is Web Audio centric.

But loading it could be as simple as HTTP requests.

You could use Fetch.

You could also in-line it.

You could even go as far as generating the data if you feel like that.

But suppose we just the sample with an HTML request.

We need to make sure that it's an array buffer.

Otherwise, you will get funky results.

And then once you load it, you the response in here.

And this is where we get into Web Audio stuff.

So we just take the response and send it to the context and say decode audiodata.

And we can have the callbacks version or the promise version.

But Chrome hasn't implemented the promise version.

So in the interest of making sure that this works, you could use this version.

So when the thing is decoded, it would take whatever you loaded, which is an OGG or an MP3, or a WAV, and it will take that and read the header and all the binary data, which encoded in the file, and produce a buffer that the context can use for playing.

So if everything goes well, we get decoded buffering, and we can do things.

If something goes wrong, we say, ooh, and then deal with error.

So suppose it went well.

We just create a buffer source and then set the property buffer to the decode buffer that we just magically produced.

If we connect this into the destination-- this is wrong.

It should say, ac.destination.

And we start it .

We get something that sounds like this, [MUSIC PLAYING] which is the famous Amen loop.

I hope someone knows about this.

So it might sound a bit convoluted to just playing a sample We could just do this the same with the other type.

But the interesting thing in Web Audio is when you start playing with the properties like the parameters like they playback rate in this case.

So with that, you can do things like pitch bending.

And you just need to change the value with this.

You could also do it linearly.

But in this case, we just set it as a function of the mouse value. [MUSIC PLAYING] So

it can get really funky.

This is actually how I feel right now-- really low.

But [INAUDIBLE].

So it's interesting.

So that's for pitch bending.

That's the kind of things you can do with samples.

You could also place things in 2D, which is something you cannot really do with the editor.

And why you would want to do this thing is, for example, you're building some kind of music up or website or whatever.

And you want to get things to specialize.

This is my exercise to you.

When you go home or when you commute and can listen to music, just pay attention to how things are specialized.

And notice how the drums are generally in the center.

The bass might be a little bit off.

The guitars might be a bit off on the other side.

Maybe the voices are a bit panned and make this interesting thing.

So this is why you counter it with the StereoPanner.

And the way you do this thing is to create an instance of StereoPanner.

And whatever you want to specialize, you connect to the StereoPanner.

And for changing the position to change the pan property that we set by the time or linear, you can do this automation thing and everything.

So if it's minus 1, it's going to be to your left.

And if it's 1, it's going to be to your right, and with that, I can build a hypnotizer.

[HUM] So five seconds, it will do whatever I say.

I have been trying hard to find something to say.

And that's I will never be dominating the world, because I can't think of something to say.

So I will just skip to the next thing.

So that's cool.

Here goes my hope of being an average person.

You can do things in 2D.

But if you are working on games or interactive experiences or maybe VR or something like that, you need to be a bit more complex, and you need things to be in 3D.

People thought that you actually have the five speaker system for doing this thing.

But you can actually simulate 3D space with just two speakers.

It works better with headphones, but hopefully, if you were here.

Before we start with this example, we need to understand how it works.

There is a listener for context.

And that represents us or the camera.

You have a 3D world and as many position sorts of sound as you want.

And everything you want to position, you need to connect to a panner.

And we will need to set the position for that thing.

So we set the position for the listener, which is us and the position for audio things that we need to position as well.

You need to remember to update the things when you expect them to be in the right place.

And also use meaningful units.

I think they use meters.

So if you're 3D thing is using thousands, you're going to get very low sounds, because it's too far away, essentially.

So you might need to think of that, because they're trying to be realistically physical.

So units, in this case, are important.

So once you make sure that you're in a place where that thing would make sense to be sounding, you would get sound.

So you could get things like birds in 3D.

I don't know if that works in this theater, but that would be cool.

Like when they go behind you.

I could really hear that in the headphones.

And it's pretty cool.

OK, you can try this thing online later.

So that's cool for specializing.

And if you want things to be a bit more complex, as I said before with the more complex graph, we use filters that will say, OK, just let this frequency pass.

And, again, creating this thing is pretty simple.

[MUSIC PLAYING] Oops, go away.

OK, can we just be in peace.

So you have several types of filters you can use.

In this case, right now, I'm using a high pass filter.

And you can't hear anything because it's super-high.

But before I was using a low pass.

And it's very simple to get good effects with this.

But actually, if you try to program these kind of filters with C or C++, you're going to have a hard time.

So this is great if you are a programmer, and you don't want to worry about this kind of things, because this is the really for you.

This is super cool.

And it's got some change in the frequency of the frequency of the filter as I move the mouse.

So that's why I get these transitions like DJs.

So we can have a party here right now [MUSIC PLAYING] and get super-interesting.

But, yeah, it's kind of like a party.

Although, I don't see anyone moving, because you're trying to be polite and all that.

But there is no party without visuals.

So for that, we have the analyser node that will give us real-time frequency and time-domain information.

So for the ones of you who are wondering, this is the real-time frequency information is the FFT.

It will run on FFT for you very quickly.

The time-domain represents the chain of the data that is actually being played.

So this is slightly different from the other nodes in that you have to create the node.

You can connect whatever you want to analyse to that analyser node.

And you can connect it to the destination and just go through.

So it doesn't modify the sound, but you need a place to dump the results into.

And in this case, I'm using an unsigned integer.

So I'm getting values from 0 to 256.

And you can say, I just want this number of results.

So this is a property from that analyser.

You could get less.

But this is just an example.

When you have an unsigned integer, you want to use bite frequency functions.

If you want to get flow results, you would create a flow 32 array.

And the difference between one and the other is essentially the bite results actually go in the flow results and do some interpolation and some adjustments.

So the range is the 0 to 256.

I don't really know why that number.

It doesn't seem useful to me.

I kind of prefer the float.

But there is a choice.

You have a choice in that.

So the reason that you drop the results-- the reason that you provide the results array to the analyser is because if you were creating this array each time, it sounds like garbage collection time all the time.

And this is a function that you usually call in every requestanimationframe.

So that could be 60 times per second.

And these arrays might be 2,048 values.

They are kind of big.

And you don't want to be dumping those kind of things 60 times per second.

So that's where you allocate this data and just use that thing from now on.

And with that, we can analyse sounds.

So here I'm cutting out all the frequencies, and now, I'm letting more frequencies pass.

[MUSIC PLAYING] So you can see what the filter does, which I think is pretty neat if you don't understand how physics works.

And now I am going to change to a high-frequency filter.

So now I have all the high frequencies, but now I am cutting them.

So just it's super-super high frequencies I got through.

So it's interesting.

So you kind of like start getting creative.

And this is something new.

If you've seen my talk before, this is the new part.

So people are always asking like, yeah, that's cool.

But how can I also use the web cam? And I say, yeah, you can do this thing.

There is another array thing they can use.

Blah, blah.

So I thought that I could maybe show you how to do that thing.

So I'm using getUserMedia.

I'm just asking for this sound.

And I'm connecting that through another media stream source node.

And I'm using the thing to connect it to the delay node.

So I'm just showing you nodes that I haven't told you about before.

But that will make things intriguing, and you will like to see the source of this thing afterwards.

So the input is good to the delay node.

And both the input and the delay are piped back to the delay node.

So we get interesting feedback.

And I'm concerned about the feedback and the amount of delay using the mouse again.

So hopefully, we'll get funky results.

And it's totally unpredictable.

So I guess this is the kind of things that Bjork and Imogen Heap would like to use on stage.

Maybe they will hire me at some point.

[ECHO] So this is longer delay.

[ECHO] And then this is shorter.

[ECHO] I was trying to sing.

This is [INAUDIBLE].

[APPLAUSE SOUND] I think it's impossible.

In the hotel it was pretty easy to sing like a chorus to-- stop.

OK, now it should be stopping.

No, it won't.

Well, it's impossible to control.

Go away.

Yeah, I was trying to sing in the hotel.

I could play something.

And it would loop.

And I could sing another note and make nice textures and things.

But here it is impossible because it's louder than I was thinking.

I was in the hotel like, haaa.

People might be thinking, who is singing in the hotel room.

That's weird.

But, yeah, anyway, wrapping up-- this is very minimal because in the last thing I showed you, it took me like maybe 25 minutes to build things together.

So you can do many things with the existing functionality.

And once you understand the correct rules and the ideas show you having blast out things and add values and gain to multiply things, you can start doing really interesting stuff that doesn't actually need you to spend a whole week trying to learn how to compile things in your computer, which was the case for me when I was interested in audio some years ago.

So if you're a creative type, or if you know someone who is a creative type, you could make their lives amazingly great by showing them this kind of thing, because they can get started on playing these things very quickly.

And also, if you're not a creative type, but you work in games, or this kind of things that need real audio, this is great, because you don't have to worry about finding some algorithm somewhere in the internet that someone says looks like a filter, but doesn't actually and creates segmentation faults.

So this like lots of existing nodes which have been well proved and really well-tested.

And if they are not well-tested, people will help you make sure that they work.

Like, I actually made one of the implementers run to fix a bug.

So that was pretty exciting to have an effect on people.

So we're trying to make sure that these API's are really usable by people and robust and efficient.

We're now working on making sure that things like generating the sound is as efficient as it can be per type of processor.

So this is like getting into an assembly line, making sure this works for ARM, or for Intel, or whatever.

So this is the kind of level of quality that we want in these API, because yeah, this is new and everything.

But this is really promising.

And everyone should be using that, well, at least exploring that.

If nothing else, because it's great for these applications I tell you, and if you're not using them for games, or VR, or interactive applications, you're making a disservice to your users, and also to yourself actually, because you're going to be having lots of issues that you could avoid by using this.

And as I say, Web Audio API is available pretty much everywhere like even in iPads.

It's super-cool because you could be writing games that work on iPad with good quality of sound instead of fighting with audio tags and weirdness.

So if you want to be able to distribute really easily, this is the way to go, and if nothing else, because it's fun.

Just like I was playing with this thing, and I was like, I might be spending too much time with this.

Maybe? I don't know.

Maybe it's good.

And everyone I saw on the slides, they spend some time playing with them, and I am having fun with modulation and everything.

So this is great to build things together.

And I don't know why not more people are using that.

So my homework is now that you know how this works, hopefully, you should go and explore and build whatever you think might be interesting.

Some people are building graph visualizers because they don't like too imagine things.

Other people are building synthesizers.

But you can build whatever you're interested in because this is super-open.

Also, you should come to the Web Audio meetups.

We have one in London and one in Berlin.

I don't know if there are more which I don't know about.

There is Web Audio slug, which slugs are totally unsearchable.

So I can't really point you to the URL, because the website they're using to guide people to their thing is [INAUDIBLE], which is down, so hi, slug.

But there is a slug somewhere.

Otherwise, you can just ping me, and I will forward you to the London meetups.

And if there is not a meetup in your place, maybe you should make one, because it doesn't really require much more than you want someone else to discuss Web Audio.

And I think is pretty great, at least the ones that I've been to.

It's cool because we are having this mixture of people, the artists, or people that are audio people, but they came from other disciplines.

So they can teach things that we don't know yet like peer data or super-collider.

This is a good crossroads of disciplines, like multi-disciplinary thinking, and it's really inspiring because if you're doing JavaScript all day long, your mind is bent into discussing about frameworks, or whatever.

But if you go to one of those things, it's like, oh, there's so many more things that you do on the web other than just discussing about frameworks and not going anywhere.

Like, you can actually make art.

And you can make cool things, like, we're in a theater.

This is where art is important.

And actually, games is one of the biggest industries in the modern economy.

And so all these also are very important.

So you should discussing this more of than frameworks.

This is where the slides are.

There is also a number of resources that you can check.

I've been trying to compile some of them.

But I don't know if I should guide you to some horrible places.

But, yeah, I will post them on Twitter.

This is in case-- it is H-O-W-A like Hands-on Web Audio.

And that should be it, thanks.

[APPLAUSE] [INAUDIBLE].

Thank you, Soledad, please come over to the comfy chair, and sit where the audience can see you over there.

[INAUDIBLE].

Tremendous and hypnotic stuff, thank you.

Lots of questions from the audience.

One of the first ones, which I think you can easily answer is which browsers can I use this in? All.

Including IE? I think so.

Or they were going to.

Yeah, I don't know if they can in IE.

But you can in Microsoft Edge.

They should.

They should.

I think they are.

Yeah.

And do the sounds sound the same in all browsers? They mostly do because here's a funny story.

They started with webkit.

So when Chromium split, they took the same code, so the generation sounds the same.

And then when Mozilla implemented that, they just looked at the formulas.

So they're pretty much the same.

And can I play Web Audio with my MIDI keyboard? You can with the Web MIDI API, which is a different API.

So you have the Web MIDI, which is just for dealing with devices and just with devices, like physical devices.

And you have the web audio, which is for generating and manipulating the sample.

You can get the data from the MIDI device for the Web MIDI API, which is not in Firefox yet, but should be.

It's in Chromium.

I don't know if it's in Safari.

I don't know.

So the Web MIDI API listens to the keyboard.

And then decodes the data.

And you can pass that to Web Audio? Yeah, it enumerates all the devices, and you can say I want to use this device.

And then you just get raw metadata.

So there is 0 to 1 to 7 values.

And you need to parse the thing.

But I'm pretty sure people will start writing libraries for that, because it's a bit harsh.

Got it.

And one of the questions was we have the audio element.

So why do we need Web Audio? Why do we need JavaScript? Yeah.

That's interesting because people are like, this is super-complex.

I just want to play sounds.

So I'm like, well, keep using the audio tag.

But if you want to do more interesting things, you should start using Web Audio.

If you want to, for example, with the audio tag, you can say, I want this sound to start at exactly 2.345

seconds, you might be something like, OK I'm going to have this timeout.

I hope that maybe when I press play there with my callback, this thing starts.

But it might be that, actually, when you do the thing, the dom element for the audio tag is not ready yet or something horrible like that.

And you don't get the thing to play.

So I guess it's a bit like comparing the image tag with WebGL.

It's totally different.

And you should use the image tag.

You should use the audio tag.

But if you want to do complex things, use Web Audio and WebGL.

I guess because you're generating the sound with JavaScripts, you don't have to worry with audio about grabbing an MP3 or grabbing an OGG.

And those are compressed, so not great fidelity.

With Web Audio, you are just generating sounds.

But you can also play the samples, anything the browser can play.

And you can also actually include audio tag output inside the graph.

So for longer sounds, it actually is recommended to use the audio tag because it doesn't try to the decode the whole thing in memory, because that's a bit too much for if you have a huge thing.

So you can take the audio for audio tags and for video tags as well and make funny things with them.

So I can plum in real sounds from the audio tag or MP3s or OGGs and muck about with them using the Web Audio API and play real sounds? Yeah.

Cool.

There's loads of opportunities for games there.

One last question is, how low level can I go? So, for example, can I write my own nodes? You can.

There is a thing called a script processor.

But they are trying to change it to use audio workers or whatever it's called this week, because it's not actually workers.

They want to let you write your own scripts that actually generate audio.

The way it's currently done is it was kind of like a hack.

And it runs in the JavaScript thread.

So that's the very bad idea, because it's fighting for resources with the rest of the elements.

So the audio worker should run in a different thread.

And it should only be generating streams of numbers.

But that should be safer than running it the JavaScript thread.

But in short, yes, I can.

Yes, you can.

Excellent.

And I need to apologize because I accused you for being from Barcelona, and you're from Valencia.

Well, some of my family come from there.

OK, then I don't apologize.

That's OK.

Ladies and gentlemen, Soledad Penades.

Thank you very much, Soledad.

[APPLAUSE]

Post a comment