Fronteers — vakvereniging voor front-end developers

Responsive images: ain't we there yet? by Marcos Caceres

Transcript

All right.

So I'm actually here to sell you a phone.

No, I'm not.

So, yes, I work as a platform architect.

What that means is that I generally just read email for a living and get paid, which is great.

I do formally work on the web APIs, stuff that you heard about before in the previous talk about Firefox OS.

I'm the fool that's supposed to go out and try to standardize some of this stuff.

So anyway, let's get on to responsive images.

So I've been involved with the Responsive Images Community Group not from its inception, but for the last year.

Essentially, as Paul said, I edit the specifications.

So when the group got kind of started, some of you know the history.

It was very messy.

It was very chaotic.

And so I came in and tried to put some order to things, because I kind of tried to make my little website, and then I ran into the responsive images problem, and I'm like, oh, that's what all these developers are complaining about.

So that's how I kind of got into it.

The group itself was funded by-- or founded by Matt Marcus, and it's probably about 2 years old.

It is now the largest community group of the W3C.

So we have essentially 300 members.

Not all them uploaded images.

This is the little wall that I got.

It's a very grassroots effort, as probably some of you know.

It really just was, like, developers just trying to make a difference.

So they all got together, and this has kind of snowballed quite a bit.

We now do have the browser vendors involved as well-- so Mozilla, Microsoft, Apple indirectly involved, and so on.

We had a meeting in Paris like two weeks ago where we actually managed to get all the browser vendors together.

So yeah, it was like a massive achievement.

It's not to be understated.

It's like, when you hear, like, Apple going somewhere, when developers kind of go, please come and hang out and talk to us about this problem.

it never freakin' happens.

They don't go to conferences.

And they showed up, so it was like a huge achievement.

And developers did that, which is fantastic.

Anyway, what you'll want to know is, are we there yet? Well, no, but we're a little bit-- we're closer, yeah? Hang in there.

So we've heard over the last two days, you know, why we should care about performance, and why we should care, particularly about HTTP requests and all that stuff.

So it turns out that images are-- as the image shows-- they constitute like 60% of the average web page.

This turns out to be like a huge performance bottleneck and a huge perform cost, and a cost to users when they're basically downloading stuff on mobile phones.

So I'm not going to ramble too long about why you should care, because, like, is anybody not convinced that you should care? So it's like, of course Bruce is not convinced and Doug is not convinced, because W3C is not convinced.

Opera-- that's you.

Opera Software not convinced.

So anyway, if you want to know more, like, in depth, go and search for Matt Marcus, for his talks.

He does, like, amazing talks about why, as a developer, you should really care.

I'm just going to be very brief about this.

So this is from the HTTP Archive.

It basically shows images-- you probably can't see it-- constitute on average 970K out of a total of 1.5

megabytes for the average web page.

This is going up all the time.

So if you go and look at the HTTP Archive, you'll see it's incrementally going up.

So the situation is not getting better.

So then there's the-- what I'm going to call the responsive web design conspiracy.

And this was uncovered by a guy called Guy Podjarny, who works for Akamai.

So he did a bunch of research into sites that claim to be responsive design websites, and what he found that the desktop sites and the mobiles sites have the same content.

That's not in itself a bad thing.

So it may just be that you are very good at targeting things.

But in a lot of sites, and some other research that our group has done, so the Responsive Images Group has done, actually shows that there's massive savings to be made.

And I'll come back to that, because it's a really interesting project, if I don't run out of time.

Anyway, so there is a lot of potential there to reduce the impact, particularly with images.

So what are responsive images, now that we've had like two years to try to understand this problem? They're images that correspond to the environment.

So what the hell is the environment? So it's always not the physical environment.

It's the browsing context-- so basically the browsing environment.

We're very used to interacting with that environment using CSS and JavaScript.

But there's other parts of that environment that as developers, we don't get to interact with-- things like user preferences, and, you know, things that the browser knows that you don't.

So it may know about bandwidth.

It knows about connectivity.

And it's keeping a lot of stuff from you.

So images-- responsive images need to work within that context.

Part of it you control as a developer.

Other parts, the browser controls.

So in order to get kind of-- this is very abstract, so let's get a better sense by looking at some actual use cases.

So use cases generally represent what we see in the wild.

So a lot of what we do at the Responsive Images Community Group is actually go and look for this stuff.

So we do this because the browser vendors asked us to do this.

It may have been a little bit of a delay tactic.

It's like, yeah, go and do some research and, yeah, let us know what you find.

But it's actually really important.

So the most common case-- you all would have done this as developers-- is what I call the stretchy images responsive case.

This is basically where you have a design and you're going to fit an image to some layout.

It's pretty straightforward.

We've all done the CSS max width, max height, whatever, and it fits.

So this is the fundamental one.

You need to keep this one in mind.

Don't discard this one.

This one's not even in our use cases and requirements document, but this actually underpins everything.

So when I mean that it underpins everything, you need to look at this image.

So this is an image that was put together by Paul Robert Lloyd, and it basically represents, let's say, three viewports, different devices.

So we have one layout with images.

So this could have been like a flexbox layout or whatever.

So notice in the first set, we have a small image, then we have like a medium-sized image, and then in the last one, which could represent a very small mobile phone, we have a single column going down.

But the images are essentially different sizes.

We're going to keep coming back to this image, because this one kind of holds the key to the solution of responsive images in some ways.

So this one is basically the thing that's going to allow us to tear apart solutions.

Can we build a solution that works with this? Then we have like, moving onto where all this started is, device pixel ratio, the iPhone.

So what is device pixel ratio? So it is the-- device picture ratio is the ratio between CSS pixels and physical pixels.

So shout out if you know what is the DPR, device pixel ratio, DPR, of the iPhone.

So how do you know that? Retina.

Exactly.

So I didn't even plant this.

It's perfect.

So we don't know about the Retina display, and Apple's done a great job marketing this stuff, but most people won't be able to tell you, for instance, what is the DPR of this projection here? So like, it's like, what, like 4 meters long by whatever.

It is a 1024 by 768.

Can anyone tell me what the DPR is? 1? From where you're standing? Yeah, that screwed you.

So it gets tricky, because you're actually dealing with multiple things.

So even though the DPR may be 1, when you actually start zooming in, the DPR changes.

So you need to basically understand the difference between physical pixels, the device independent pixels, and the CSS pixels, and how they interact with each other.

And that's actually really hard to do.

And it's very annoying.

So as I said, we have to factor in, if I have zoomed in, the DPR is not 1, what is reported at least by the browser.

And you can test this in any browser.

You go in, check the DPR at 100% and then zoom in and have a look.

It will tell you, like, 1.1, 1.2.

Turns out that something like-- Google did some research and it turns out that over, like, 10 to 16% of people browse the web with a DPR setting of, like, 1.1

because they've zoomed in.

So we know from the iPhone, we know about Retina, we know that because of Retina, we get kind of crappy-looking images sometimes.

OK, so the next one is viewport matching, which is essentially, if you have a screen of a particular width, you want to send an image that will match the pixels that you have.

So if you have, like, a giant Apple display, you know, one of these cinema displays, you might want to send an image that matches that size.

So desktop, of course, it's easy.

On mobile devices, it gets a little bit more funky, right, because you have to take into consideration the meta viewport tag.

And what does that actually mean? So again, DPR comes into play, because you're dealing with a virtual viewport.

So if you have like, let's say, let's take an easy example, so let's say you have a screen that has-- it's 200 pixels wide.

It has a DPR of 2.

How big should the image be? Come on, simple math.

400, maybe? Times 2? I could be wrong.

It's complicated.

You have to work it out.

You have to think about it.

And that's kind of annoying.

So where we see this having a negative effect-- so here we have a screenshot from BBC News.

So BBC News does the viewport matching on portrait.

So you can already see that it's a fairly low-quality image.

So the BBC guys do a fantastic job.

They deliver content very quickly.

But they know that most people look at this stuff in portrait.

They're holding their phone reading going down.

I like to get up in the morning and basically do this and hold the phone this way, kind of close to my face and read the news.

So when I do that, the image stretches and the compression artifacts-- you probably can't see it very well, but it becomes very obvious, you know.

So for nerds in their responsive image groups who are more concerned about this rather than, you know, global warming burning down Yosemite National Park, that's important.

You know, that's something we should fight for.

So the next case is art direction.

You all would have heard this.

This is like the popular one.

This is like, well, if you can't do art direction, you're not really doing responsive images.

So it's true.

So again, this one is kind of the yardstick by which we measure the solutions as well.

So there's two kinds of art direction.

Here's a simple one where you take the focus point, so the dog being the focus point, and depending on what screen you have, you're going to try to basically send something that represents the image quite well.

So you're art directing the thing.

So this is, like I said, the crop.

Then you have the swap, which is, given some condition, as you'll see, you swap the image for something else that represents the same subject matter.

So there-- this is from the Nokia website for the MiniGo web browser.

You saw that it swapped from-- I'll just play it again, just in case you missed it-- so it swaps the image.

The designers felt that it communicated better in landscape to actually swap the image out, even though it represents the same subject matter.

There are other use cases-- print and e-ink.

So printing-- we had all these talks about typography, you know, on screen, but we also sometimes like to print this stuff.

And when you print-- like, particularly images, when you print from the web, they'll print the 72 DPI image, and it looks like crap.

My printer can print something like 1,440 DPI.

So generally, like, a printer will print 600 DPI.

So why can't I print beautiful documents on the web? It's kind of annoying.

Another thing is e-ink displays as well.

So if you take a color image and you put it on an e-ink display, like particularly the chart that I was showing you before where it was all blue, you're not going to be able to differentiate the regions because of the shading.

So in those cases, you might want to swap out the image.

Another big one is file format support.

Part of a responsive images solution, reacting to the environment, is sending the image that is smallest for a particular setting.

In those cases, you might want to use WebP over JPEG over PNG.

So that would be nice.

Anyway, if you want to know-- if you want to know more about this, it's all written up all neatly in the use cases and requirements document for standardizing responsive images.

So we've really tried to solve this problem.

I think we've like, smashed our heads 100 times against the browser.

You saw Steve Souders talk.

He talked about images, and if you load them with JavaScript, it screws up preloading.

We've hacked the hell out of this.

What has ended up happening is, like, we just end up with solutions that are all just dibs.

Dibs-- there's no semantic markup there.

You know, we can't use image because it screws up.

We need to use dibs because we need to do the replacement JavaScript.

So it just ends up a huge mess.

So what can we do? We've hit the glass ceiling.

The next step is what? We go to C++, right? We got to go and fix this in the browser.

We can do that as a proprietary solution, or we can go and cry to the standards bodies.

Yeah? So the big scary things-- you know, so the WHATWG and W3C.

So this is really painful.

So I watched the guys from responsive images group fight with Hicksey, fight with lots of people there about particular solutions.

And it wasn't great at first, but we did make progress.

So this was like 2 years ago, and we've come a long way.

And people have paid attention to the needs of developers.

So it was the case that it was very difficult to convince them, but now I think we've managed to convince people at the standards bodies that we can do better.

The problem was that we-- browser vendors-- I've worked for Opera Software and I also worked for Mozilla.

Browser vendors-- putting on my Mozilla hat for a second-- we see the world a little bit differently to developers.

Developers are very used to having fine-grained control over everything they do, so pixel-perfect layouts and so on.

So in the browser makers' world, we see HTML as a declarative thing, that you declare your intent, and the browser will help you achieve some design.

So it's not a bug, like, that there is CSS, but it's kind of a weird thing in the intermix of how browsers work and declaring things a markup and how they get-- end up on the screen.

But we've come to an understanding.

By communicating, by talking, by first fighting it out, we've now come to some kind of understanding.

So enough rambling.

Let's talk about the solutions.

So it's been a pretty amazing journey getting to the solutions.

As I said, it's been a long journey.

And it's really a story about people.

So I'm going to tell you the story in-- I'm gonna walk through the solutions, and I'm gonna look at the good, the bad, and the ugly of each one of the solutions.

So the reason I'm doing that is because there isn't a perfect solution that meets everybody's needs.

It's just not going to happen.

There's always these annoying edge cases.

There's always going to be weirdness.

But we can see where they are, and then I'll it leave it to you to voice your opinion about which ones you like.

So, picture element.

So this is the spec that I edit.

It has a dubious origins.

Some claim that Bruce Lawson, who's up here, who will be speaking later, he made a straw-man proposal about it.

I'll just show you an example so I can explain a little bit, and I'll zoom it.

There's going to be not a lot of-- well, there's quite a bit of code, actually.

We're talking about markup solutions, unfortunately, so we're going to have to look at some code.

How it works is very similar to the video element.

So you have picture, and then you have a source, and then you have a media query that allows you to match a particular source.

So we're using the srcset attribute, which is another proposal we'll talk about.

So we basically merged two proposals together to come up with this.

And then inside that, we have this fallback image.

Pretty simple.

No rocket science there.

And it's nice.

So the good-- it covers the use cases.

It covers art direction, it covers DPR selection.

It does the advanced use cases, or the kind of niche use cases, too.

It does the print use cases as well.

And I think people get it.

Like, the response from the community about the proposal was very positive.

People got it straight away.

It makes sense.

The bad-- turns out that using multiple elements is not ideal.

It actually turns out to be quite hard to process, because you can put other stuff inside the picture element, and then it just gets confusing for browser makers.

And there's all these weird edge cases that I'm not going to go into.

Let's say that it's kind of bad.

It extends the source element by introducing srcset.

So what if you use srcset set inside a video element? Uh-oh-- confusion.

So that's kind of bad.

You can't have specialized semantics just for one element.

That seems kind of silly, in one context.

So that's not great, although, you could use in other contexts, but not ideal.

The other problem is that it mixes the media queries into the markup.

That's not very dry.

And it also causes problems.

So if your media query selects a source and you don't have an image width and height set and you pick a really big image, it can change the layout, which will cause the CSS-- a different CSS media query to match, which will change the layout, which will cause a different CSS media query to match, and you get into an infinite loop.

So that could be bad as well.

So it doesn't quite consider the problem as well as it could.

So now for the ugly-- my favorite bit.

I told you this image we going to be important.

So let's see what you would need to do in order to basically create this width picture.

And here's the lovely markup.

So this is a solution worked out by John Mellor.

So he went out and tried to find out, what would you actually need to do? So the reason he made the solution is because he works for Google.

This is what they do, and this is why I don't work for Google.

So he went out and did the hard work and worked it out.

So he got out his calculator and actually did the thing.

Now, this is an extreme example.

I show it half trollingly, because it's not-- like, you don't get into the situation really, because usually you target either just DPR or the viewport width.

But if you're going to do, like I said when I started this talk, you're going to do this, like you do normally in web design, you need to basically have this kind of markup, and that kind of sucks.

So you're probably wondering, how did it come to this-- you know, have your little Lord of the Rings moment.

I was going to play a video, but it was kind of like, ew, no sound.

So let's blame the srcset attribute.

Why not? So, srcset-- note that in the code that I showed you, the media query wasn't really the problem.

I'm not going to go in and look at this in detail, but the media query is not the problem.

The problem is really the srcset attribute doesn't quite understand what developers are trying to solve.

So let's take a look at it.

So with srcset, we have-- you set the device pixel ratio that you want, or you can target on the width of a layout, or you can do both.

So for 100 width, give me a times 2.

Pretty straightforward, you know.

That's great.

So the good-- it's an incremental improvement to the platform.

We like incremental improvement.

We don't like revolutionary things in the platform.

We like to take baby steps.

It reuses existing element.

It uses image.

It doesn't introduce a new element for us to learn.

The browser makers really like it.

So Apple proposed it and everybody seems to like it.

It's already implemented in WebKit, with the help of people from the KD project, and also somebody from the Responsive Images Community Group actually helped implement it.

And it also-- they also landed it in Blink.

It hasn't landed in Blink, but the code is in Blink as well.

And it does nice things, like if you don't set a width and height, and it knows the DPR and it selects an image, it will automatically resize it for you, so it won't get overblown to the wrong size.

The bad-- it doesn't support the art direction use case.

It doesn't support media type selection.

And now for some of the ugly.

So when it was first proposed, people really didn't like it, because it's really confusing.

What the hell does the W actually mean? So if you have just a normal layout, then the width of the device make sense.

What if it's in an iframe? What does the width then mean? Also, the algorithm that's used to select things is really counterintuitive.

I'm not going to explain it to you, because I actually coded it and I still don't get it.

It's kind of like a rubber band that goes like, OK, if you're bigger than this, then do this, and then go backwards, and then go forwards.

And it's like, whoa, it doesn't make any sense.

So it's kind of hard.

And if-- I'm not a particularly smart guy, but I go and code the specs, and if I don't get it, then I generally think, like, most people won't get it.

So I'm like average developer.

So, and like I mentioned previously, users have their browsers zoomed in, which screws with the algorithm.

So what if I told you that you didn't need any of that microsyntax nonsense-- no srcset.

See, because what we really want is just this.

We just want to declare some image, and we want to load it from the source.

So let me introduce you to client hints.

So to get to client hints, we need to take a conceptual leap.

Like I said earlier, as developers and as designers and so on, we see the world in terms of JavaScript and CSS.

We know the markup, but there's one layer that we forget about.

What's that layer in the platform? HTTP.

We don't think about HTTP because it's one of these things that's a little bit outside of our domain that we manipulate.

We use, like, XHR to get stuff, but we usually don't screw around with the headers and so on.

So this is a solution that tries to solve this problem using HTTP.

Here's an example.

When you make a request for an image-- so foo.org/image--

what you do is you attach to that image, or the browser will attach to that image, the DPR on its own, and the width and height.

So you don't need to put it in the damn markup, because the browser already knows his stuff.

It can just send it.

And it's also extensible, so then you can send other variables too.

So the good-- there's no need to change the markup.

It has a nice opt-in mechanism.

So you might have been thinking, oh, that would suck, because then every request would contain this information.

No, it actually has a nice mechanism that the server says, hey, I support these client hints, please send them to me.

So it does that on the first request.

The browser goes, yeah, that sounds cool, I can support those, so I'll send you this information that you need.

Fantastic.

Three questions.

Can you set the HTTP headers on your site? All right, for you people, do you use GitHub pages or a CDN, let me ask it again.

Can you set the HTTP headers on your site? So the number goes down dramatically.

So that's the bad.

Having a server-side only solution just defers the problem somewhere else.

It requires content negotiation, which kind of blows, because you basically, like I said, you're just deferring the problem to another place in the stack.

It doesn't really address the core fundamental problem.

Content negotiations is hard.

One of the use cases is not met.

It doesn't meet art direction.

And I thought I had one more, but I don't.

Oh, it's in the ugly.

So it gets real ugly when you start sending that data back.

So for people who have experience with a little bit of HTTP, proxies can strip your data.

So if they're like, oh, what's this weird thing, I'm just going to strip that out-- gone.

So weird things can get stripped out.

So in order to make it work, you might only be able to do it reliably over HTTPS, so using SSL.

And it relies on vary a lot.

How many people know what vary is? So a few people do.

So that's great, but the rest of you who don't know, I guess the browser would handle most of it, but it gets kind of weird.

So vary says, like, this is-- you send me a request, I send you this back, but I changed this thing.

But then it doesn't change-- it just tells you that it changed something, but it doesn't tell you what it varied on.

So then you need more headers to be able to actually do the-- understand what actually changed.

And that gets problematic if you requested a 2 times image, it sent you back a 1.5 image,

so it varied on the DPR, but then it doesn't tell you by how much.

So then when you go to lay out the image, when the browser goes to lay out the image, it's like, I don't know what size to make it, sorry.

Because I need to reconvert it.

So it's kind of like, uh.

So, yeah, it's all pretty much fucked.

Thanks.

[APPLAUSE] No, come on.

It's not so bad.

So, but, yeah, we're not in a good way.

So let's go back to the drawing board, yeah? We know we like picture.

Picture's pretty cool.

So let's take picture and see if we can fix it.

We're gonna try to do this interactively.

So let's start up here.

We already have an element that is that a damn picture, don't we, Bruce? What is it? (bruce) SVG.

SVG? No.

What is it? It's the image element.

Shut up.

That's enough help from you.

All right, so image-- we can't have an image opening tag and closing tag.

Image doesn't have a damn closing tag.

So screw that.

OK, so now we're down to image alt, which is good.

Width and height, yes, that's great.

But we have a source here.

We also have, like, another one down here, another image, which is our fallback image.

That's not great.

So we have repeated things.

So let's trash this we don't need.

Alt we need.

So we're going to keep alt. So like that.

And trash that.

Cool.

So what the hell are we going to do with these damn sources? They're like-- we have an image.

We still need to do the selection, right? So what do we do? We can't somehow shove them in into the image.

Or can we? So let's try that.

Let's just take this and, like, put it down here.

Let's put the-- wrap the tag in the tag.

OK, that's not really going to work.

So what if we just like, get rid of these tags, because tags are annoying inside other tags.

That doesn't really work.

And now they're kind of looking like what? Like attributes.

Cool.

OK.

So we have source, srcset, media.

What can we do with that? Anybody? Can you see a way of collapsing them? Yeah, let's kill the media.

So let's do that, and just say, like, source equals this.

And we have srcset.

And we're already saying that it's a source.

We don't need to say source, srcset.

That's kind of stupid.

So let's just put a delimiter there.

And we'll say that the source is selected from this media query.

And also we got medium 1, medium 2.

Everyone's happy.

So do that quickly again here.

And now it's looking a little bit more interesting, right? So is this going to work? We have two source attributes.

What's going to happen? What are the XML folks going to say? XHTML nuts, what are they going to say? (NASAL VOICE) Ehh, doesn't parse.

That's how they talk, I swear.

[APPLAUSE] So let's just put a number on that.

So we're on source 1 and source 2.

Holy shit, we solved it.

Yeah? I mean, that's all valid.

Everyone's happy.

We need to-- we knew that we had the source elements.

They have to be evaluated in order.

We've sorted the ordering problem.

Looks good, yeah? (audience) [INAUDIBLE].

Huh? (audience) [INAUDIBLE].

You're ruining my boom moment.

[LAUGHTER] So yeah, if you have 30-- we're getting to the 30 moment, all right? So we've Steve Jobsed the thing.

It's fantastic.

So we've simplified it.

So I wish we were that clever, but it actually took two guys from Google to kind of reduce it down.

So Tab Atkins and John Mellor got together over lunch or something, and they went, we can fix this, and took out everything and fixed it.

So let's have a look if it actually works.

Does is hold up to scrutiny? So here it is again-- simple example.

We'll zoom in so we can see it.

So pick one.

So the source picks this one.

Source one will pick the one in the middle.

So our main width 400, and then the large one at the end.

Fantastic.

So this is where it gets a little bit technical.

So we go-- we know that source-- that source and attribute is really a media query, which is optional.

Then we have these x-based URLs that we saw from srcset-- so foo, then two times.

And then we have these new URLs, which are the viewport URLs.

So I'm saving that last one for the ugly.

So the good-- it addresses picture's problems.

It solves most of the use cases.

It handles multicolumn layouts, as you'll soon see.

The bad-- it's actually not easy to polyfill.

We've been trying to polyfill this in the Responsive Images Community Group.

It doesn't solve for media query-- or sorry, for media types.

It has a scary new syntax, which I'm about to show you.

It's not very forgiving as a solution.

If you screw up a character or something in one of the attributes, in the source and attributes, it gets forgotten.

That's currently how it's defined.

And it's a little bit complicated.

So as whoever shouted out, like, what if you have 30? Yeah, what if you have 30? It's going to be complicated.

So let's now use our yardstick there.

So viewport URLs-- simple example.

Zoom in so you can see it.

So what this says is, if the image is 100% of the viewport, pick any of these available solutions.

So 160, 320, and what these values here represent is the width-- the real width of the image.

So pic 1 is 160 pixels wide, 320 and so on.

And then the browser can actually choose the one that best matches the size available for 100%, whatever that may mean-- so a full layout.

Here's where it gets a little bit fancy.

What this also allows you to do is you can basically set-- look at where your break points are in your CSS and express what images fit into what viewport.

You ready for this? So here-- I know this is like, if it looks weird to you, trust me, it's weird.

So we say that for-- we use 100 for a 30EM breakpoint.

We use a 400 image for a 70EM breakpoint, and then for anything else, like above 50%, go nuts, OK? And here's our bag of available images.

So I call this the pray-- so the spray and pray algorithm.

Just to go back so you get it-- you just set like a whole bag of things available, and then you just, like, let the browser sort it out.

And you just go b-b-b-b-b, and the browser will hopefully pick the right one.

So spray and pray.

So, yeah.

So-- but wait.

So that seemed like a really good solution, but then it got kind of complicated.

If the browser really knows everything, what if we had a container format that we could use? Again, we just want the freakin' image element with a source attribute.

What if we could solve this, like I said, differently, just using an image format? Let me introduce you to the magical image format.

So this is actually a thing that we're actually working on.

So how does it work? It's a file-based approach, where we put one layer per-- so we have one layer per resolution.

And it's able to handle our direction and resolution switching all in one.

So this is how we lay it out.

And you have advice from the Responsive Images Community Group to actually build it.

I'm going to show you, like, real outputs from this.

So this is not in reality.

This is just how it works fakely.

So you basically say that from 0 to 100K-- kilobytes, it represents the times 1 image.

From 101 to 400K is the times 2, and then the times 3 is the larger one.

You could put more information in there to help the browser make a decision.

But that's just the basic concept.

So let's see how resolution switching works.

So you take a very small image-- wait for it.

You upscale it.

You take the delta of the upscale, which looks basically like this.

This is me exaggerating the colors.

And then you upscale the smaller one to fit the space-- boom, and you get back the image.

It's crazy.

And you can do it again, and it only adds a little bit of data each time.

So you compose and just take the diff.

You can see it here with art direction.

So you start with this one.

Then you fill in the space.

So what you're seeing very lightly, hopefully you can see, like, a little bit of distortion.

That's actually the alpha there that will get filled in and stretched.

It's the same again here.

You can see it a little bit more clearly.

And you compose it, and you get back the image.

That's right.

[LAUGHTER] [APPLAUSE] So, yeah.

Responsive Images Community Group-- smart freakin' people there.

The good-- so the markup is left untouched.

Single file per image.

It's, of course, better for postdownload dimension changes.

You can basically let the browser select the bits that it needs and compose the image.

The bad-- it's a new file format.

We've seen the problems about getting, you know, even WebP adopted.

Mozilla's to blame, too.

We are very resistant to it.

We have our reasons, whatever.

It requires-- to get this to work will require quite significant changes to the browser.

We basically need-- you know, there's implications about he coding performance as well.

The really cool thing about this format, and you obviously actually tested this, is that you can take, like, WebP and JPEG XR and use them to interleave in between them.

It doesn't care.

The file format doesn't care.

It's able to use both.

But it does require a custom fetching mechanism.

So the ugly will be that you need new tools.

It will probably rely-- or require HTTP2.

It's probably going to be an IPR nightmare.

This is too cool an idea to not have a patent on it.

Hopefully it doesn't and the video will go out and no one will patent it.

But we've already represented this in a few places, but it's probably an IPR nightmare.

I saw like the MPEG LA, it already has a patent on this.

And it needs to be standardized.

This is just one developer hacking around.

So now, crystal ball time.

What's going to happen? So as you know, I'm doing all of the standardization here.

We're getting really close to getting agreement about srcset.

Like I said, we landed it in WebKit.

We've been pushing strong at Mozilla.

We don't know where we're going to go.

Client hints seems really cool, and I'm sure people agree.

If you have a CDN, you run a CDN, that's a really good idea for you.

And images are a huge part of a web page.

We know 60% of a web page is an image.

They serve a very important communicative purpose.

So it might be, or my gut feeling is that we might need all these solutions and people will be able to pick and choose the one that they want to use, and I think that would be ideal.

So hopefully we will get to the point where we actually do have the image format as well.

But it's going to take a freaking long time.

So right now we're just looking to increment-- to do the baby steps.

Maybe we'll get srcset.

Maybe srcset will win.

So I quite like, and the developer community really likes srcset.

There's not been push back on it, apart from the worry about not being able to polyfill it.

So, enough science fiction-- a little bit of practical stuff.

Don't wait.

There's things you can do today.

So one is, of course, compress your images.

It's not that hard.

Pick the right format.

Don't be too lazy about it.

I'll get to laziness, because there are lazy solutions.

But go back.

This used to be a real art when I started out doing web development.

So back in like 1996 when I was at uni, we used to like, obsess about, I'm going to use GIF here, I'm going to use JPEG here.

And we used to go and tweak all the things because we were in the middle of a transition that's similar to the one we're having now, where instead of devices, we were transitioning from dial-up modems to broadband.

And there was a whole range of different bandwidths.

So we had to be very careful that we were still serving people who had dial-up modems.

So those were slow, comparable to mobile connections today.

So this is a great tutorial.

I encourage you to go and have a look at HTML5 Rocks, "Image Compression for Web Developers."

If you haven't read this kind of thing, this is a really, really good article.

It's a good place to start.

If you're lazy, just like me, investigate using a CDN or some server-side solution to handle this-- so like a CDN like Akamai or CDN Connect.

I'm sure there's others.

I'm not here to promote other people's things.

But I know, like, for instance, they guys from Akamai, they really care about this problem, and they've done some amazing work.

And there are free solutions like Google developed PageSpeed, which does also a really good job at optimizing images.

Another thing you can just easily do is download Imageoptim and just drag and drop your files in, click a button and it will compress the crap out of them.

So here's the presentation slides.

Before it was like 20, like 30 megs.

I just ran it through these and now it's like 20-something megs of images.

So, sorry if you're on mobile-- the presentation, it's going to suck for you and it might cost you a little money.

But it's an image-based presentation, so I'm sure I'll be forgiven.

Compressive images-- next one.

This one's controversial.

So this guy-- I'm going to butcher his name-- Daan Jobsis-- he went out and did some experimentation where he took-- I'll just walk you through the secret recipe.

Before you do this, take a hard, look at yourself, because I'm going to talk about the performance implications of this.

You take a full-size image.

You set the compression to 0, and then you set it in place in the browser and let the browser resize it.

Yeah? Think about that for a minute.

So once you've done that, go back to step one.

Take a hard, long look at yourself and run through it again, and see if you can get the image down to the quality that you want.

And then go and actually test it in the browsers.

Make sure you're not screwing up your website.

So here's a side-by-side comparison of what I'm talking about.

So this was done by the Filament Group.

They're big participants in the Responsive Images Community Group.

So they tried the technique out.

They took the 1024 by 768 image, set the compression to 0.

It got down to 44 bytes.

You take the same image at 400 by 300, it's 95K at 90% compression.

So it's nearly half the size.

And-- huge screen, obviously you won't be able to see it, but they look freaking the same.

It looks great.

And you saved 40 bytes, 41 bytes.

So here, like, the full-size thing.

You can see, hopefully, some of the compression artifacts.

It doesn't matter.

When the browser shrinks it down, it looks good.

So-- skipped ahead too much.

But use it with caution.

You're basically sending a lot of data.

That data needs to be reprocessed.

It goes into a bunch of caches.

It uses a lot more memory.

So you can use this technique, but it's not that great.

But it can be used.

Another solution to do a lot of what picture was doing is to use picturefill.

So this was developed also by-- this was developed by Scott Jehl, who also works for the Filament Group.

And other solutions, go and check out CSS Tricks.

There's a great list of responsive images solutions there.

And I encourage you to come and join the RICG.

If you're interested, we really want people who are kind of passionate about the subject and want to hang out with a great community of developers.

Come and join us.

Thanks very much.

[APPLAUSE] Thank you, Marcos.

What do we do-- what do we do today? How do we-- in the present moment, do you have any guidance? I think the polyfills are OK.

The thing that will really push browser makers is, like, to see more content out there that's trying to do responsive images.

And-- but I mean, we've all acknowledged-- all the browser vendors have acknowledged that this a problem.

So we are at that kind of critical phase now.

Is there any-- is responding to the network conditions and the available bandwidth a hint that has been considered in any of these proposals? Yeah, so that one's super difficult to do.

It's like, almost impossible to do in many ways, because, like, even as you walk, like, from here to the end of the room, your network conditions will change.

It's just too dynamic.

Yeah, I heard that too.

This is one of the reasons why the network information API has not really had strong adoption.

Yeah.

It's just too tricky to say, oh, they're 3G-- Yeah.

--go for it.

That's-- doesn't work.

And also, like, a 3G connection may be faster than, for instance, a Wi-Fi connection here because of all the people using it.

So you actually may get better performance on 3G.

Right.

What is-- how do you manage multiple images-- image downloads per device, per orientation? In-- so I guess that really depends.

If it's a script-based solution, then, I mean, you're just going to have to go with image tags and do it.

But hopefully you will do it with markup as much as you can.

But just let-- you shouldn't manage it.

Like, just let the browser do it.

Don't try to optimize for this stuff.

The browser is, like, going to do it in a much smarter way than anyone else, basically.

Yeah.

The image format, the new magical image format-- I've heard it referred to as magical image format a few times, which I guess is magical image format-- mif? Mif, or we also have the responsive image container, or ric.

Ric.

So those are the working names? Yeah, those are-- we're going to find a better name for it.

And as far as, like, the immediate concern would be, with single container, wouldn't want to ship down the entire thing to-- [INTERPOSING VOICES] No, that's the beauty of it.

So you basically just grab the manifest.

So you would like, say, bite in the first few bytes, you see what you've got, and then get a little more.

So this is where HTTP2 comes in.

And then you go, OK, what do we have? Scan it, and then you can go, OK, we're at times 2, so we're only going to grab this, this, this and this, and then compose it together.

Oh.

Is it dependent on HTTP2? It's not dependent-- well, it kind of is, but it isn't.

It doesn't have to be, but you need a way of actually reliably setting, or saying I want this byte range.

HTTP1 will do that for you, but, like I said in the talk, you need like a special thing to actually get it to do that.

Like, you need a special downloader, in a way.

Gotcha.

OK, cool.

All right, thank you, Marcos.

No problem.

Appreciate it.

Thank you.

[APPLAUSE]

Post a comment