Alex Feyerke - Offline First: faster, more fun, and more robust (web) apps
Fronteers 2014 | Amsterdam, October 9, 2014
Let's assume for a minute that the technical side of building offline-capable (web) apps was really, really simple. Because it's getting easier every day, and it's obviously desirable: you’d rather have a robust, reliable app instead of one that turns into a wonky disappointment when it's disconnected for a moment. Once we take that as a given, we can think about what building apps Offline First means in terms of interfaces, in terms of your application's structure, and in terms of the experiences your users could have with it. What are the benefits, opportunities and challenges of Offline First development? A talk about patterns for offline UX, thinking differently about user data, confident wording, pre-empting people's needs, notions of the future and ordering things in lists.
A huge welcome for Alex Feyerke.
[APPLAUSE] Hi, thanks a lot.
I know I'm the last thing standing between you and a party.
But I still want to take the opportunity to thank Flurin and the Fronteers team for having me here and for doing a phenomenal job.
This is really, really cool.
This is my first time at Fronteers.
And I'm very impressed.
Also, Jake Archibald-- I've been sitting next to him.
And he's been constantly working in a flurry of activity since this morning.
And I'm also very impressed.
This is offline-first web apps, which sounds stupid.
But bear with me.
This makes sense.
My name is Alex.
I'm part of the team that makes Hoodie.
And this thing doesn't actually do anything.
There we go.
So what's Hoodie? It's an open-source server that lets you build complete data-driven web apps with users and everything from the client side.
So it's like jQuery with the data storage in it.
And the general aim is making web apps really, really easy, because we think it's too hard.
But Hoodie is a bit special, because all these apps are offline capable by default, which means that they won't break when the connection goes down.
And that's basically why we're interested in the whole topic.
So first of all, what this isn't? So it's not going to be a step-by-step guide how to do this.
I don't want to stand here and explain to you how this one thing works at this moment in time, because everything changes all the time anyway.
And I'd much rather show you what it all means and get you interested in the whole idea in general.
So it's more about the concept of the interfaces, the experiences, and the problems you have with an offline-first paradigm.
But before we get started, let's talk a bit about how we work.
So we make web apps or native apps.
It doesn't really make a difference in this case.
But we make, and while we make, we imagine those things in their working, complete or usable state.
I read once that the hallmark of a true professional is the ability to see into the future with what they're doing.
And that's kind of what we want to do in a way.
We imagine the stuff we make in the future where all the people are using it and making us rich and famous, or at least internet famous, or Kickstarter famous.
So you get stuff like this, for example.
I don't know if you've seen this.
It's called Plastc.
It's 20 credit cards in one thingy.
And like most pre-order crowdfunding people, they have a vision.
And of course, they have a video.
And it's basically a tiny slice of science fiction that says, here is how we imagine really human beings behaving in the near future, and, of course, all the technology that goes with it.
It's small and specific and mainly related to how people pay for things.
But there are other ways of imagining the future.
So Microsoft, for example, is a really big company.
And they love imagining the future in more elaborate ways.
So they regularly release so-called productivity future vision, which sounds exactly like something Microsoft would release.
It sounds terrible, but let's just take a look at this.
[MUSIC PLAYING] Anyway, let's don't talk about the content.
But I think this really feels kind of wrong, doesn't it? It's too smooth.
It's not normal enough.
It's a funky choreography of wealth, and status, and permanent productivity.
And it's utterly improbable in its perfection.
Every surface is a flat screen, of course.
There's information everywhere.
But maybe you'll notice the devices still have recognizable pixels.
None of them have a battery indicator or a signal strength indicator.
Half of them obviously don't contain batteries or antenna unless they managed to make them transparent somehow.
And so we have a world with transparent batteries and transparent everything.
But people still drive cars, obviously.
And in a culture so optimized for efficiency that you're practically working while waiting for the subway, people still fly to meetings instead of teleconferencing with their walls and their windows.
And that's just-- I don't know.
That doesn't make much sense in my eyes.
So everyone's happy-- not happy.
But everyone's happy.
Nothing is broken or smudged.
There's infinite Wi-Fi with infinite bandwidth-- no batteries to run out, no traffic, no delays.
And by the way, this is from the fantastic lecture by Tobias Revell, which you should definitely read.
It's really eye opening and very well done, which is why I'm stealing some more from it.
So you probably know the next one.
This is the Google Glass promo.
[MUSIC PLAYING] (male speaker (on video)) OK, Glass, record a video.
FEMALE SPEAKER (ON (video)) This is it.
We're on in two minutes.
MALE SPEAKER (ON (video )) OK, Glass, hang out with flying club.
Google photos of tiger heads.
[DOG BARKING] You ready? You ready? FEMALE SPEAKER (ON (video)) Right there.
OK, Glass, take a picture.
[CHILD PLAYING] [MUSIC PLAYING] (male speaker (on video)) Go! Woo! Holy [BLEEP].
That is awesome.
(child (on video)) Oh! Look at that snake! (alex feyerke) And on that creepy man with the 10-foot in the basement with the kid, we stop.
This is marginally better, because there's normal stuff going on like playing with dogs or kids or flights being late.
But the basic symptoms are kind of the same.
So the people in that video and the circumstances they live in-- they are mostly outliers.
They are very privileged people in this kind of weird post-scarcity scenario.
Where as Revell puts it, it's a classic case of everyone suddenly being an ice sculptor and flying around in balloons.
So that's also not normal.
It's a fantasy.
And as far as goals and dreams to work toward go, it's all a bit weird.
But it's also more nuanced.
It has some degree of normality.
And it's implying something interesting.
So people are using Google Glass in places without network connections like the sky.
And unless that balloon in the beginning was one of Google's funky internet balloons providing Wi-Fi to the sky diver in the flying club, I think the implication is that they are expecting Glass and the software that runs on it to work offline.
So it's still a bit improbable that it has hints of normality.
And now, we go back to it.
So the next video is from 1969 with a vision of the 1990s.
So I'd like you to just picture that in mind for a second-- the future as seen from the '60s.
You've all seen the movies.
Now, consider this was made by the British Post Office.
Let's take a look at the sort of facilities we can get him-- worldwide communication, of course.
Hello, Bill, this is Henry.
(bill (on speaker)) Who? Who did you say? (henry (on video)) Henry, I'm calling from London.
(bill (on speaker)) Gee, that's a pleasant surprise.
How are things in your part of the world? (henry (on video)) It's a dark, wet morning here.
I suppose you've mean basking in the sun all day.
And when we want it, view phone in color using digital transmission by pulse code modulation right into the customer's premises.
Let's have a look at you Bill.
(bill (on video)) Right, Henry, thanks for coming up.
Gee, it's good to see you, Henry, you're certainly looking fit.
(henry (on video)) How's Betty, Bill? (bill (on video)) She's right here now.
Come over here, Betty, and say, hello, to Henry.
(betty (on video)) Hello, Henry, long time, no see.
But we can do much more than that-- library services, transmission of documents, share prices, bank statements.
Let's take a look at the subscribers terminal unit.
To make a call, we take our service card.
A code will be scanned electronically and established the caller's identity for charging the call.
[CALLING] As she keys.
The digits appear on her view phone screen.
Suppose she makes an error, she can clear and start again.
[CALLING] (alex feyerke) And then a ship comes into the harbor.
So this seems a bit more normal.
Why is that? People aren't wearing futuristic clothing and doing things with flashy lights at transparent desks.
You see a normal man, a very normal man, in a normal office.
And the first thing that happens in this video demoing this new technology is that it doesn't work properly.
They can't understand each other.
The guy at the other end doesn't know who's calling.
And the weather's bad because it's London.
And it's always bad in London.
That's just normal.
And listen, suppose she makes a mistake.
Apart from the fact that it's obviously the woman that makes a mistake, which I find pretty appalling, people don't mistakes in Microsoft productivity future vision.
Nobody in the Glass video goes, Glass, hello? Did you hear me? It just works.
And the topics-- leases, bank statements, mortgages, the list goes on.
It's terribly dull.
But it's normal.
It's a bit wonky, but it's everyday life.
And so thinking about the future is obviously a bit hard, because for every miraculous iPad, there are countless broken realities-- Wi-Fi passwords, connectivity, battery life, privacy, compatibility, and a lot more.
And the real skill of creating a compelling and engaging view of the future lies not in designing all this gloss, but seeing beyond the truth-- the gloss of the truth behind it.
So let's go back to this thing-- what I just said about this product sci-fi.
Here's how we imagine real human beings behaving in the near future.
But I really mean, and what you really want, is here's how we imagine something absolutely normal happening in the near future.
But there's a problem because we're not normal.
I'm sad having to say this here as a well-educated, well-paid person, as you are, in the tech industry in a super-wealthy country, we are not normal.
The circumstances we work in aren't normal.
And they usually bear little resemblance to how and where our products are going to be used.
We build from behind shiny Macs with full size keyboards with well-equipped offices, fast connections, and the target environments are often radically different.
It used to be so that if your device could consume the web, it could also make it.
Remember, the next box from previous talks.
That's no longer true.
Most people who start off with the world today have a consumer device.
They can't make websites or web apps with them.
So first, you had to learn that people's devices were different than your dev machine.
And now you realize that the circumstances are also quite different.
And that requires a bit of an emphatic leap.
And that's hard because our circumstances define our mindset.
We have the best tools in the best infrastructure.
And of course, that informs how we work and how we think about all of the new, shiny future things that we make.
And so the extreme example would be if you take the Google Bus to work, you don't get the usual public transport experience.
You've got free corporate Wi-Fi and air conditioning and everything.
And your commuting experience is far, far removed from that of basically everyone else.
That's your baseline now.
And you seem aloof to everyone else.
But in a way, we're all on Google Bus.
And we frequently forget that we have to build stuff for a normal world, because normality is where most of life happens.
And that means building for a mobile world, because of the normal world, the normal people in it with average budgets, average resources, average attention spans-- that world from our perspective as developers is overwhelmingly mobile by now.
Phones are mobile.
Tablets are mobile.
This thing is mobile, and there are probably right now hundreds of university students sitting around in cafes in the city with their little laptops with USB 3G sticks plugged in.
That's a mobile device.
It doesn't really matter how big the screen is.
They're going to have the same problems as people with phones-- connections, battery life.
So the big grey box is really a relic.
Mobile is normal.
And normality, and you really probably all know this, is imperfect.
The web is very, very imperfect.
It can do amazing things.
But it also breaks quite easily.
The experience of building is going to break for people-- I guarantee you.
It's also imperfect.
And it's imperfect in ways you can't influence, mainly connection speed and reliability.
And then these kinds of things happen.
What you can do now though is build experiences to accommodate this normality-- experiences that go beyond your apps just breaking, becoming unresponsive, showing interviews, losing days, and making people nervous with panicky error messages.
You can now build apps in ways that acknowledge that the web is in imperfect and that your connections to it are too.
And you can provide a good experience regardless.
And that's the core idea.
So offline is not an error.
Offline is a fact of mobile life.
And because of that, it's a fact of normal life.
It's just another state your app can be in.
And it's a fairly normal one at that.
It's not exactly an edge case.
So this is the foundational idea behind offline first.
You treat the network as an announcement.
So in comparison to something you know already, mobile first is as a design strategy to help cope with the vast amounts of screen sizes your app may encounter, like all of these are Android devices.
In the same way, offline-first is a design strategy to help cope with the unknowable circumstances and connection states your app might be in and provide a good experience regardless.
So offline-first is simply an acknowledgement that this lack of certainty about where our app is being used extends a bit further than we previously thought.
So we assume nothing about the user's connectivity.
Offline is the default state.
Connectivity is a bonus.
Treat the network as a potential enhancement.
And in this case, potential enhancement is really the way to go, because if you want full offline capability after the fact-- tacking it on after you've already built something-- it's a bit hard, because the architecture is necessarily quite different, at least we think it should be different.
So before we continue, it's more foray into offline-first architecture.
And because I said this wouldn't be overly technical, I'll try and keep it short.
So if you have an app, what needs to be offline? Two things-- first of all, obviously you want the apps to work offline.
So you want the app and its assets and all the images, or SBGs, and fonts, and everything.
And you can do this through App Cache or service worker or a native wrapper such as PhoneGap or Node Webkit.
Or in a native app, this is just a given.
So you just want the basic stuff to be there.
But you also want the data to be there, because otherwise, the app is a bit worthless.
The app must handle data in a way that doesn't require a connection.
So this is best done by reading and writing through a local store in the browser which syncs, when possible, to a server.
And you can use local storage, IndexedDB, Web SQL.
PouchDB, or on the high abstraction level, something like Hoodie.
And this, to us, is the more interesting bit, actually.
So let's see how Hoodie does this just as an example.
So you have front end and a back end to your app.
And in the front end, you have, well, the app that runs in the browser.
And it only ever talks to the Hoodie API.
It never talks to server-side code.
It never talks to the database.
It never talks to the in-browser storage.
It only ever talks to the Hoodie API, which in turn, talks to your local browser storage.
And you can feel free to replace that with any technology you favor.
It doesn't have to be local storage because it doesn't really do very much.
So this by itself is enough for an app.
You can wrap that in Node Webkit and sell it to people who don't like the cloud.
But let's assume you want a bit more like management of users or syncing.
So the store of the Hoodie library you load into your browser.
It can sync stuff over a REST API to a CouchDB.
And we can do syncing because each user has their own little database, which only they can access.
So it's very simple to decide what gets synced.
It's user's private data.
And because it's the user's private data, it obviously should be on their machine as well.
And because we have all these little pockets of data, we can keep people's stuff separate on the server very easily as well.
And on top of that, you have what is basically the real back end.
I guess more a core of Hoodie, basically, and a whole lot of plug-ins that do stuff.
And they listen to events that get them into like CouchDB.
So when something new pops up there, the Hoodie plug-in is now, ah, I've probably got to do something.
So as an example, just assume you're sending a direct message to someone.
And you tell the Hoodie API, I want to send a direct message to Jake, for example.
The Hoodie store acknowledges that, puts it in local storage, and as soon as it can, which may not be immediately because you're in a tunnel or an airplane, it syncs that message over to the CouchDB.
That emits an event.
At the back end, there's probably something like a direct messaging plug-in that gets this even and says, look, I probably have to do something.
They will then take the message, find Jake's private data store, put the message in there, and put a confirmation in this one.
And then whenever possible, those things get synced back from the server to the client.
The client emits an event.
The UI can react to it.
And that's it, basically.
So we never have elements talk directly to each other.
They only leave each other the messages and tasks, a bit like passive-aggressive roommates.
So it's all very loosely coupled and event-based.
And that means it can be interrupted at this stage without breaking.
Messages and tasks will just be delivered whenever possible.
It's designed for eventual consistency.
So the nice thing is the front end doesn't really care, in most cases, whether the back end is actually there or not.
It will hang onto your data, and it will sync when it can.
And if your UI and your use case allows it, you can just keep working on the front end.
So I find first apps are a bit like the web's honey badgers.
They don't care.
They just do their thing.
If there's bees, or tunnels, or bad connectivity, the offline badger just keeps on going.
It's how things should be on the web-- robust and fault tolerant.
But anyway, the point was if you want true offline capability, your app shouldn't try to talk to the server directly if possible, only through a syncing engine.
And you always want to keep a local copy of the users data in browser storage.
That's another really essential point.
The user's data only uses device always.
Now, sync is pretty hard we found out.
And you probably don't want to be implementing this itself, because there's a whole lot of stuff to get wrong.
So we use CouchDB, which is cool and relaible and which is basically a database that replicates, which is exactly what you want when you're syncing stuff.
There's other ways to do.
There's that's Pouch, which we want to use.
There's something like remote storage, which has an interesting one-back-end-per-user approach that also works offline and syncs.
And there are funky things like JS Git and all sorts of cool ways of doing this.
But whatever you do, I implore you to try and find someone who as already solved that particular bit.
But in our opinion, sync is the keystone of a full offline first experience.
So getting into the offline mindset requires only one key realization.
When you leave the world of timely, reliable communication, the local database, not the servers, must be the gateway for all persistent changes in application state.
So then the big question-- how do I make all this offline magic work? Magic.
Anyway, no just a line of code.
So this is what Hoodie looks like.
You use the API to store stuff.
And when you add things, it expects a type and an object with data in it.
So it's a document-based storage in a way.
If you want to sign up a new user, we do Hoodie account sign up.
You give it a username and a password.
It's fairly straightforward.
We literally could not make this any easier.
We spend quite a lot of time formulating this is what we want, the simplest thing we could think of.
And then we implement it on that.
And on the return trip, you listen for store events.
So if something comes in the store, you listen with on.
And then you handle new objects appearing-- put them in your UI.
How do you make this work offline? That's it.
It's not a feature.
It's just how the whole thing works.
All you need to do is embrace this decoupled architecture.
You talk to the API.
You let it sync for you.
And you listen to events from Hoodie to see if new data has arrived.
And as I said, it's not a special feature you invoke.
It's how the architecture works.
You get it for free.
So let's just see that in action.
So I'm going to start a new Hoodie up and do Hoodie New Fronteers.
And for a very long time, nothing items because MPM wasn't particularly fast in the hotel network.
But I cut it together.
So it shouldn't be that bad.
There we go.
And then we can cd into that directory and do Hoodie Start.
Now, we set an admin password, and off we go.
The browser opens.
And there is our default app, which of course, is a to-do list, which I'm very sorry for.
It's a to-do list.
So I'll just show you what happens when you type into this text field.
In the background, the only thing that's actually done is a call to Hoodie store add.
And as I said, it expects two things, a type, which is to-do, and an object with some stuff in it-- in this case, just a title.
And the list listens to the event from Hoodie and shows it.
And that's all that happens.
It's the only thing that happens when you enter stuff into that text field.
It goes straight to the database.
The database emits an event.
And the list listens to the event.
And then it pops up.
Now, we'd like to sync that.
And to sync it, we need to put it on a server.
To put it on a server, we need to know who you are.
So we sign up.
We have a super-secure password.
And now we're signed in.
And we have a username as well.
So now let's do some syncing.
We're going to open this thing in another tab after a bit of layout dancing.
There we go.
So obviously, no data in it because I haven't signed in yet.
So we're going to do that now-- sign in with the same user we just signed up over here-- username, password, and everything magically appears.
So now we write something.
It goes to the local storage.
It goes to the server.
An event gets emitted over here, and it shows up.
And it also works in both directions.
It's not particularly exciting.
But the thing is in the background, you're still using the exact same code.
There's nothing in addition there.
So now, let's stop the server.
The server is stopped.
It didn't refresh.
And of course, since these two browser windows don't know that they-- they don't know of each other.
They can't talk to each other directly.
They want to talk to the server.
The server isn't there.
So they just hang on to the data until they can eventually sync it.
So let's turn the server back on.
And, again, I'm not refreshing the browser windows.
Everything's there-- cool.
And again, when you type into that, it's only that one line of code that gets executed.
The rest happens automatically.
[APPLAUSE] So you don't really get offline for free.
You get multi-device sync for free as well.
And this kind of illustrates the underlying strategy.
If you can make it unobtrusive, make it unobtrusive.
The simpler it is, the more people can use it.
We thought that is pretty sweet.
Now what do we do with it? We felt like we solved a bunch of technical aspects.
And it kind of worked really well for us.
But we didn't really understand what opportunities this gave us.
So we thought we'd ask some people.
So Gregor from the team went to Africa on an Africa Hack trip to get to know the tech seen in Africa.
He went to Kenya Uganda, Rwanda, and Tanzania.
And he met a lot of people with very interesting offline problems.
I ended up writing a blog post that got fairly popular and got some good feedback and some more good feedback, and some surprisingly civilized hacker news discussion.
I wrote another for A List Apart.
And it slowly became a bit of a thing.
We went to Mozfest in London and held a workshop-- held another one in Berlin.
And we just started talking to developers about these issues.
And we were surprised how many suddenly opened up about their offline troubles.
It was like having an offliners anonymous meeting suddenly.
And they'd all realized they'd all had similar problems in the past.
But they had never spoken to others about it, because it's just such an obscure topic.
And there's no language for it.
And nobody knows what anything is called and battled it out alone, gave up, or put it off.
But they all secretly wished they could have talked about it more.
And people had very interesting and specific problems ranging from building, and I quote, runkeeper for fish-- for anglers in the Norwegian wilderness to keep track of what they've fished from rivers and stuff to several people trying to build robust point-of-sale systems in places with bad connectivity like mountains selling ski passes and things like that.
But we identified some central themes.
And one of them was trust and reliability.
The first and most pressing issue people have is that they can't trust their apps to do the right thing when the connection gets dodgy.
And you can summarize this with the following user story-ish thing.
As a app user in general, I should strike that mobile.
Anyway, I want to be sure that my data is actually available when I need it.
And this is quite fundamental, really.
It's so fundamental, it probably gets forgotten most of the time.
Of course users want to have access to their data whenever they need it.
But it used to be a bit of a painful problem to guarantee this.
And it was easy to just fail, show errors, and let people to blame it on the Deutsche Telekom.
But you can do a lot better.
And we think offline-first is the way to go.
But first, let me illustrate what the problem is.
So here's the German railway advertising that you can book tickets and use that app under any circumstances, even when free soloing in the Alps.
And remember, in the tech future, everyone's into extreme sports and ice carving.
So that makes perfect sense.
Meanwhile, in the real world, there's this guy who can't access the ticket in his app because the app won't start up when the connection is bad.
It'll start up fine when there's no connection.
But on an actual high-speed train where the phone can connect to the occasional cell tower, but maybe not talked with properly, it fails.
And it feels really, really, painfully slowly, while the ticket inspector is breathing down your neck.
And German ticket inspectors are very unpleasant.
So you don't want that.
So the solution will obviously be to store personal data and stuff like tickets locally and not make the initial communication with the server blocking so you can't access the thing which you have on your phone.
And the symptom of this is that people take screenshots of their apps, because you can't trust the app to retain the state.
So who here does this? That's quite encouraging.
Anyway, lots of people do this.
And I went through the photos on my phone and found quite a few of these screenshots where I was afraid that the data0 I had just fetched would vanish the next time I opened my phone, because that kind of thing happens a lot.
So all of those views were just full of data moments before I took those screen shots.
But then I made a crucial mistake.
I put the phone in my pocket.
I switched an app or something.
Then the apps schedule the data, and they fail at loading new data.
And then you have no data at all, and that's just stupid.
So we can use offline-first to get more trustworthy and reliable apps.
It's just generally less frustrating.
Another thing that happens is that many apps are designed to work while connected, even if there's absolutely no intrinsic requirement to do this.
And again, I think this is a relic from the edge of the big, gray box when you had a terminal, and a thin client, and where all the computing power was far away from you, but probably not on your desk.
But now, it's in your pocket.
And we can do things differently.
So when I'm offline, I want to be able to use features that don't require a connection.
And we have some empathic agreement on hacker news with that.
If you're only viewing your own data or adding data yourself, why shouldn't this just always work online or offline? You should be able to add a meeting, market task is done, write a message, check in, post a picture, create things on the device, and see the stuff you have in your device.
Store it now, sync it later, but don't get in the way of getting things done if it isn't necessary.
And again, this is an opportunity.
You can have more useful and more usable apps.
And this is directly linked to the next one.
My data isn't with me.
It's probably on a server in the United States somewhere.
And if you're lucky, it's on a CDN on the same continent as you.
But I'd really like to have my personal data on my device at any given time.
I mean, it's my personal data.
Why shouldn't be on there.
I might need it.
And you probably don't know when.
Plus, in most syncing scenarios, I'm the authority over my data.
So let me just keep the definitive local copy.
And granted, this gets problematic if your app is highly collaborative.
But if you have a use case when your user data is fairly well isolated, you can easily do this.
So the potential benefit is always accessible personal data-- super-fast, always available, and simply reliable.
One more thing people came up with is that apps sometimes require advanced planning to be useful at all, which strikes me as very strange and them as well.
Apps simply not being smart about what they store locally.
So I want apps to preempt my needs in a sensible manner, because I don't know in advance when I'll be offline.
If I knew that, I wouldn't have to bother with all of this.
So once out of bed, internet and apps are used almost always constantly peaking during the daily commute with 70% usage.
What that means is people use their mobile devices most in the situations where connections are worst.
As an example, an RSS reader should already have data in it when you open it.
So you have the new articles you'd like to read before you notice that you're offline and can't fetch the new articles that you'd like to read.
So that's just one of the experience-driven facets of offline-first.
You just want to make a good offline experience in general.
Something like obviously important data should probably be stored locally.
So DropBox is good in this regard.
You save something on your phone.
It's now on the phone, always.
And that's a good assumption to make on their part.
It's obviously important to you.
So they put it on the device you always have with you.
And you could hypothetically take this a bit further, just to go a bit crazy with this.
You could have smart offline maps.
Sure, you can have offline maps.
Most of you are from here.
So you don't have this problem.
But usually, you need to know in advance that you want to have maps offline when you travel somewhere.
So you can get an offline maps app.
Or you get Google Maps to cache your surroundings.
But Google Maps knows where I live, right? So imagine if it would just preemptively download your surroundings as soon as it notices you're on Wi-Fi in a different country than usual-- technically possible-- a spooky, but possible.
But what this demonstrates nicely is that there's a lot of new issues now.
You can technically do this, of course.
But you need new UI elements.
And you need to answer some pressing questions how long does the map gets stored? Does it get removed at some point? Can I remove it when I don't need it anymore? Will it just gunk up my phone? Can I cash more than one at any time? And if, yes, is there some sort of limit? In short, how does it work? And can I trust it? And I think that's going to be much harder than the technical aspects.
And maybe for you, the question is more can you make it trustworthy while hiding all the complexity.
Interesting questions, but also a great opportunity to make smart apps that preempt user needs.
And this already sounded a bit complicated.
So let's get straight to the challenges.
It's becoming obvious that there are a couple of new things you could be.
Doing and these new things require new interfaces, and in some parts, new design language, because you have to communicate stuff you've never had to communicate before.
The really simple, basic example is saving versus syncing-- the horrible disk icon and the horrible cloud icon.
How do we communicate the status? It's syncing.
It's possibly out of date.
All sorts of things could be happening.
And possibly out of date is a state you'll get quite often in offline-capable apps.
And you'll have to make decisions about how crucial the age of data is and how you communicate it, because all data can still be useful while potentially being wrong.
Think of train schedules.
They could be wrong.
But they still communicate intent.
And that's valuable.
That's more valuable than not having any train schedule at all.
So we all know the spinner is a lie.
It means a lot of things.
But in the end they only means, I'm not ready.
And it's too complicated to explain why.
But we want to reassure users, right? We want to make the apps trustworthy.
And we want to find a good language, possibly visual, to do this.
So let's take a look at some examples.
If your app behaves differently depending on whether it's connected or not, you're going to have to make this clear somehow, probably.
So Threema does this by displaying an always visible connectivity stripe at the top.
It's fairly obvious.
If it's red, it's not connected.
If it's yellow, it's trying to.
And if it's green, it's connected.
And it'll work as expected.
And it teaches you that the first time you use it.
But you can go further.
You could just as well selectively disable, or hide, or rephrase features.
Imagine a Save button that knows whether it's connected to the server and changes accordingly.
It could save and sync later.
Or you could just let users save either way and inform them about the outcome afterwards.
I saved your stuff locally.
But you're offline.
So we sync it later.
There are lots of ways to solve this problem.
You can be super-secretive, just handle everything in the background.
Or you can inform users beforehand during or after they've done something.
It depends on the use case.
It depends on who you're developing for.
It depends on how crucial their data is too.
That brings us to another big problem, which is informing users about sync outcomes and not just whether they succeeded or not, which is potentially simple.
But how and possibly why data has changed that they might be looking at this very instance.
That's one of the things that happens when you sync data asynchronously.
It's probably one of the hardest UI problems related to sync, especially when it comes to chronologically sorted data like chats or streams or threads.
So I said previously that if users only add data, why not always just let them do that and sync later? Well, it turns out it's a bit more complicated because context really matters.
So here's a fairly simple chat example that illustrates some of the things that you'll have to keep in mind.
So you have a chat between A and B. A says, hi.
B says, yo.
A wants to meet on Thursday.
B answers, yeah, sure.
And then A says, oh, wait, I meant Tuesday.
Now, imagine that B's second message, sure, I'm free, was written while one of the two was offline, because they're in a tunnel, or on a train, or something.
And it's just not there.
But your fancy offline-first technology just works in this case.
The user reconnects.
The messages sync.
And A receives B missing message.
But the question is where do you put it? You could do the chronologically logical thing and put it where it belongs in time.
But the chat may have changed to the point that it's way up there now.
And nobody is ever is going to see it again.
It would make sense to put it there because the order of messages carries meaning.
But nobody's ever going to see it.
So it doesn't really work that way.
That's a new UI challenge in itself.
But you could do the other thing.
You could display it in the flow according to the time it arrives at.
This guarantees that the message will be actually seen by the other people.
But this approach has the potential to change meaning because the order of the message isn't meaningful.
And this is only text-based, one-dimensional data.
It's simple chat-- one of the simplest examples I could think of.
What do you do with deleted items? Things that can't be organized in lists-- objects that aren't immutable.
There's a whole lot of potential for complexity.
And it's definitely something to be aware of, because of web platform is becoming more and more powerful and capable.
We just saw with web RTC.
Here's the photo editor.
It's not Photoshop.
It's not trying to be.
There's no installation.
It's cross platform.
It auto updates.
And it does most of the things you want from a simple photo editor, except you probably also want it to work offline as well.
This is an experimental, collaborative multi-track recording app.
It does a lot of things at once.
It has synthesizers, and drum kits, and audio recording.
And it runs in the browser.
And this is part of a master's thesis by a guy named Jan Monczka.
It's really bleeding edge, and it just about works.
But imagine where all this tech will be a year from now.
You can have a cross platform garage band with other people in other countries.
And yes, I know Audiolense is complicated.
But it works.
And that's pretty damn cool.
But again, it can't be used offline.
And that's a disadvantage in comparison to the native apps it competes with.
Now, there's much more complex stuff on the horizon already.
And we have no proper strategies for dealing with it.
And it's all a bit complicated.
But there's a second point to all this escalation in complexity.
Offline first isn't just a nice-to-have feature for commuters.
When you've got web apps that rival native apps in functionality.
They really have to be offline capable to be competitive in many cases.
It's going to be a necessity more often than not.
But there's more to offline first than bad networks and competing with native apps.
There's a lot of advantages and opportunities.
Let's talk about those for a bit.
So first performance-- so we pay a lot of money and put stuff in CDNs to move it closer to the user.
But the closest thing to me right now is my phone in my pocket.
Put your app in this.
Put your data in this.
That's zero latency for me.
This is the way to the snappiest experience.
And the data may be old, but at least it's there.
And in many cases, time doesn't even invalidate the data.
So there is still a benefit to having it.
Another thing-- offline apps are robust to protect against service interruptions.
And this is something we hadn't really anticipated.
So we have fairly large servers running on Hoodie and had to briefly take it down for reasons.
And most people using it at that very instant didn't even notice, because the app doesn't care.
It doesn't make a difference if the client is in a tunnel somewhere and can't reach the server or if the server is down and the client can't reach the server.
It'll still work within its capabilities.
And you get better experiences.
Apps don't lose data.
Apps are more trustworthy.
Apps are more usable and useful and cause less frustration.
And remember, you're saving to a local store first.
You can save after every single keystroke if you want and sync to the server every couple of seconds.
You can forget Save buttons altogether.
You can't do that if you're saving to server.
So there's a lot to gain from an offline first architecture that's not completely obvious at first glance.
And you can start asking yourself new, exciting questions like, do I even need my user's data? It gets you thinking a lot more about the data you're collecting since you have to decide upon how you sync it.
And you may find that you don't even actually have to even want to store your users data anymore.
And now you can.
It's a possibility now.
And this can do wonders for scalability and privacy.
There may even be a business benefit for you.
So this is just a start-- image manipulation, maturing web audio APIs, all the web RTC stuff we saw.
There's a lot more to come.
At JSU this year, there was a lot about service worker client-side storage and the web crypto.
And it's all just coming into existence, the maturing, but it's wildly promising.
But by this time next year, we'll already be talking about these tools, because we have them.
We'll be able to build apps that encrypt user data before it gets backed up to the stuff.
You can do that now probably somehow.
But then it's going to be easy.
And there'll be new business opportunities in markets for that.
And it will build apps that use Service Worker for fantastic offline experiences.
And if it's in the browser, it's going to be mobile.
And if it's mobile, it's going to be offline.
And it's not even about increased mobility anymore.
At some point recently, the browser transformed from being an awesome interactive document viewer into being the world's most advanced, widely-distributed application runtime.
It's a very attractive platform for many reasons.
And users will simply expect to a and solid experience from it.
And waiting for more cell towers to be built won't help them.
Your apps have to work regardless of the circumstances they are in.
So let's recap.
You may be looking at this and thinking this seems like an awful lot of work for a really specific issue.
And networks are getting better all the time.
Why bother? We could just keep on doing what we always have done.
So networks-- think of your every day mobile experience in your wealthy, well-connected cities.
Think of your mobile experience in the countryside while traveling in other countries.
And then realize that global, mobile traffic is on this kind of growth curve.
And we're roughly in the middle.
And half of everyone is already somewhat unhappy with their coverage.
And I live in the capital of one of the richest, most-developed nations on earth.
I have a really expensive phone I still feel a bit bad about.
And I frequently only get unresponsive edge connections or none at all.
And that's before I even get on a train and go anywhere, my provider wants to charge me $0.80 per megabyte while
So I'm offline.
And yeah, that will probably change in the coming years, thanks to EU regulations.
But not every is the EU.
There is a whole world out there where you can be charged all sorts of amounts for really ridiculously tiny bits of traffic.
So don't just wait it out and hope it'll solve itself, because networks aren't the only thing.
You have this aspect of competing with native apps.
And it's attractive, because if it's a web app, you can interact quickly, you can cover more platforms, and people are moving their stuff over to web services a lot.
Look at Google Drive and what it's done to Excel.
It's only ever going to increase.
And the benefits, to recap those, high performance, more robustness, better experiences, more trustworthy apps.
And while you can optimistically hope for more cell towers, you can't really argue with that list in my opinion.
So in closing, we can't keep building apps with the desktop mindset of permanent fast connectivity where a temporary disconnection or slow service is regarded as a problem and communicated as an error.
We can and should do better.
And at the same time, this whole offline first idea-- it's not magic.
It has a spectrum of usefulness.
It's not a magic solution that will work for everyone.
It starts with simply not forgetting data and ends with the full syncing thing that I showed you in the middle of the talk.
And there's a lot in between that you can do as well.
But it's early days.
There's still a lot to talk and think about.
There are still many technical challenges with Hoodie.
We have a solution to some of them.
It may not be what you need.
To be honest, what you need may not exist.
People want really weird stuff.
But as time passes, more people will find new and different ways of solving the problem for different scenarios.
And you might be one of them.
I'd encourage you to be.
But it basically works right now.
Last week, I finished a prototype of webapp that works completely offline.
It has offline maps.
You can take photos while offline, install them until you're back online.
And it runs well on my wonky, two-year-old Samsung Android thing.
It's only ever going to get easier, faster, and more stable from here on out.
And it works right now.
But we need to worry about more than the technical side.
There aren't really any UI pattern libraries or established design metaphors for saving and syncing in sync states and connectivity states.
And you'll have conflict resolution and all sorts of things.
None of that really exist in any significant volume.
And finally, we need more awareness this a thing.
People probably need to get used to their browsers working while offline.
That's counter-intuitive, I know.
But email works like this already.
Nobody thinks about that anymore.
But it's also awareness among developers-- more awareness of problems, especially those problems we ourselves aren't exposed to-- more awareness of the technical possibilities, and more awareness of the opportunities of embracing offline first.
So we went ahead, and we set up offlinefirst.org.
And we wanted to start bringing people together on this.
We've got some discussions going-- GitHub.
We hold workshops and talks and travel around the world and talk to people.
We've started collecting other resources and projects as well as people's feedback.
And we'd like to add more.
And we'd like to enlighten you to add your own ideas and issues and just come talk to me or us on the various platforms or in person.
So please join in.
And also when in doubt, don't panic.
Use Hoodie, and please come talk to me about Hoodie or offline-first.
And I've also got some really sweet Hoodie stickers.
And the slides are up there.
And thank you very much.
[APPLAUSE] OK, you can come over here and join me in my lair of chat.
And you're free to go.
You can escape now.
Oh! Oh, my god.
Sorry, you got stuff there.
That was my fault.
I'm so sorry.
It looks like there's stairs there.
It's exactly the kind of thing I would do.
Oh, this is nice.
Yes, lovely, isn't it? [APPLAUSE] I think was me either trying to crawl out, I'll be honest about it.
Can I take my shoes off too? Is that OK? So you're saying, there are not enough cell towers of the moment.
But surely, there is going to be more built.
This problem of connectivity is going to slowly go away.
We're going to have connectivity everywhere.
Are we both working on a dying art? I don't think so, because there's probably going to be more-- the caller population coverage-- the potential coverage of the networks.
That doesn't really account to the peak use.
So you'll always get points in time or places in space, I guess, where there's too many people for the network.
And it just will slow down or not work, or something will happen.
You'll get Wi-Fi on airplanes at some point.
But yeah, it's probably going to get better.
But it's never going to be perfect.
And as I said, there's more to it than just dealing with the network problems.
There's a lot more opportunity you can have as well.
I completely agree.
I think one the many problems I've had is being a train station.
And the train is canceled.
And that's it.
You get a whole platform of people trying to check the next time.
And the whole thing just fails.
But I know that the app I have checks train times of around a time area every 15 minutes.
And even though it's innate and it has the capability, it won't give me the last data it had.
It's look, I will settle for that right now.
That would just be enough.
In my eyes, that's already a great step in the right direction.
So when you're dealing-- Hoodie is doing a lot of the data synchronization stuff.
How are you dealing with conflicts? Is this something you expose to the user? No, Hoodie is very unopinonated in most other things apart from that event-based thing.
So we leave the conflict resolution to you, because we can't possibly know what you want to do with it.
But in stuff you've built experimenting with it-- how have you handled that in the apps that you've built? It depends.
You can just say things like the client wins.
We've never really had a case so far where it got to a point that we had something so collaborative that we couldn't handle it with a simple mechanism.
But I can easily imagine that you'd have a thing where you'd actually have to expose the user to a conflict resolution UI somehow.
And because it's Couch and keeps revisions of every document, which you could then get, you can't actually do that yet through Hoodie, but they're there.
And we're going to make it a thing, I suppose, because people are going to need it.
You can build the conflict resolution UI yourself and just make a new revision and store that, because really, we have no idea what you may want in your specific scenario.
So we try to stay agnostic.
I think off the top of my head, I think that's what Google Docs does.
They get last winds as a conflict.
But you have a history revision that you can get yourself an iPhone.
So how do we get users to realize that stuff works offline? I really don't have any idea yet.
Because I don't know either.
Yeah, it's still so counter-intuitive.
Even when Chrome pops up and shows me apps while I'm offline, I say, ah, really? But I build these things myself.
And it still feels strange to do it.
You're on a phone.
You obviously don't have any connection.
But you can refresh.
And you can go somewhere.
And it works.
And maybe it's a two-part thing of communicating it in a really simple app like something like a festival timetable or something where you can get people to use it.
It has a really simple use case.
And you can have a small onboarding learning thing at the beginning saying that this app will not break if you're on the same square kilometer as 10,000 other people.
And so in the beginning, I think there's going to be a lot of this explicit, hey, this thing will not break.
And then at some point, I hope it's going to become the expected behavior in a way.
At least all this horrible empty view thing won't happen anymore because people just expect it to work.
Yeah, I wonder if there's something the a pseudo-standard like the RSS icon-- like an equivalent to that you can put on your page.
Even then, the RSS didn't really get beyond tech users.
But the answer is if anyone's got a good idea, this is a problem we're going to have very soon.
(alex feyerke) Yeah, I think so too.
(jake archibald) So when you were making Hoodie stuff work offline with the current technology, is it appcache? Have braved the haunted forest of appcache? Yes, Gregor and our team has been extremely brave and build something some helpful things that make it a lot less painful.
It's still super weird, but it works.
The apps work.
And yeah, it could be cooler.
But it does what it's supposed to.
So someone else asked this.
It's not just my own interest.
But how does this tie in with service worker then? Can you separate offline from front end logic completely? Well, we haven't really tried.
I don't really know if anyone on the team has actually tried doing anything with service worker yet.
We're kind of postponing it a bit because we have other stuff to do.
But we think it's just going to make the whole thing even more pleasant and easy to do.
And what was the second part of the question again? Can you separate a lot of the front end logic like the syncing database part? Could that sit in the service worker? This is what someone asked.
I actually don't know the answer myself.
I haven't thought about what the best way of doing this would be.
I guess the way we want to do it-- currently, we have a slightly convoluted way of doing it.
The way we want to do it is we want to use Pouch to be in the browser and use Couch to be on the server.
And then we just replicate back and forth.
It's just a one-line command.
It's really simple.
So I don't really know if service worker has any benefits for us in that case, because we currently know when we have connection or not, But I guess we could do it somehow through service worker, because you don't have to do the long polling and see if the server is still there.
Yeah, there is two options with it.
I guess if you try and send some data to the server, you can have the service worker pass that through to the network.
And when it comes back as a failure, you can do your database storage locally or whatever.
Yeah, we do it the other way around anyway.
So we store it first in the database.
So we're never going to lose it anyway.
So yeah, I think having the database in client side makes-- not on the service worker side makes the most sense.
Someone tweeted-- and find it very interesting.
So I would not say offline-first is the new progressive enhancement.
But the progressive enhancement just has a new starting point.
And, yeah, it feels a bit weird considering the way we've usually done it.
But in our case, we feel it makes sense if you really have more of this app context and less of this reading content consuming context.
(jake archibald) Yeah, I if you're going to have it work in a traditional progressive enhancement where you're going to have to renders, right? You're going to have your render on the server.
But you're going to have the same one on the client.
Which, I guess, it's not too bad if you're building something such as TweetDeck, which is behind the login window anyway.
I think something as Twitter, which still has endpoints that want to display a tweet.
I guess ti's going to bit a of a juggle.
(alex feyerke) Yeah, it is the change in thinking about this too.
And a couple people approached about this as well.
And because everything gets generated on the client side, it's a bit tricky because you could probably do the dual render approach.
But we haven't really tried it yet.
I suppose it's entirely possible.
Can e-commerce sites-- could they be offline first? What challenges are there? What do we need to solve there to make that work? Somebody asked that on the boat yesterday.
I'm not really sure about this, because, as I said, it's not really a solution for absolutely everything.
There's a lot of instances exactly like the e-commerce where you do a lot of moving about on a site.
And you're constantly browsing and opening new tabs.
And it's very much about finding things.
And to find things, you need to be online.
I suppose there are some super-specific cases where you could do like in an airplane, for example.
You cash the entire shop near the client, for example, and then make it still usable for consumption.
But if you can't communicate back to the selling authority at all, it gets kind of difficult.
And the example we had, or I talked about briefly, with this distributed point of sale thing, there were several people who came along who had distributed points of sale who sold limited resources-- so something like very rare bottles of wine in a restaurants.
And occasionally, the servers would not be able to communicate with each other-- the table serving servers, not the computer servers-- and would not be able to find out what the actual stock is.
And that's the point where you have to acknowledge that there is no technical solution.
At some point, you have to know what the stock is in order to be able to sell it.
You can only find a communication solution between the guy standing at the table selling the bottle of wine and the person sitting at the table buying a bottle of wine-- how they communicate about this potential lack of stock.
But in general, I don't know-- it's a big topic.
Yeah, I suppose you couldn't really sync Amazon.
Yeah, I know.
(jake archibald) Because all the images alone would be too big.
(alex feyerke) Exactly, yeah.
With web-of-scale or something you can store immense amounts of stuff.
And it's large.
But yeah, as I said, I don't think it works.
At the least, it should be possible to just keep what you're looking at in local storage.
So if you found out and said, hey, look I found this amazing chair I'd like to buy.
Oh, look it's gone, because the cache cleared.
That, as I said, for me, is the baseline of an offline-first approach.
It's not losing stuff you've just loaded.
And tracking orders would like you to be able to go to your order page and see the status of your order as something that has background updated.
Yeah, exactly personal information like that should easily be doable, I guess.
And it's still relevant to you.
It may not be current.
But at least you can say, wait, did I actually this or not? And when is it supposed to be here.
It may not be current information anymore.
But it's somewhat valuable.
And those parts you can easily do, I guess.
(jake archibald) So Hoodie is back on CouchDB, right? But on the client, it's Pouch? Not yet.
It's currently still local storage, because most of what we do is text-based.
But obviously, it's not an approach for the future.
And local storage is really weird.
So we're switching to Pouch.
And that's currently the ongoing mega project we have internally is switching the local div to Pouch.
So what's Pouch backed by.
Is that IndexedDB under the hood? It actually switches between-- they did a really crazy thing.
I don't know what they called it.
But you can basically switch between all sorts of databases now.
So they used to be able to support Web SQL, IndexedDB, and I think, I may be lying, local storage as well for some things.
But now, you can plug in all sorts of stuff and have in-memory storage.
And it's totally bonkers.
But it basically takes all of the pain out of storing things locally in a browser, because it just provides a unified API for everything.
(jake archibald) So do you know what the differences are between CouchDB and PouchDB in terms of capabilities.
Or is this an ongoing discovery? To be honest, the actual database work isn't exactly my area of expertise.
I think PouchDB is basically CouchDB in the browser, because apparently it's possible.
And the API is cool.
And it's a perfect tool to sync data to a couch, because it's already all in the same format.
And it's a really nice way of doing the replication, not just between servers, but also between clients and servers and clients and clients.
So a lot of people were asking about storage limits.
How do you think the browser should handle that? How can we ensure that a website visit can't fill up my device, or worse, fill it up to a point where it removes a lot of-- I'm about to be on a flight-- it deletes all the movies I've cached offline? I worried about this for quite a while.
And then I thought about how native apps handle it.
And they don't ask you anything at all.
And nobody seems to mind.
I think actually is the plan that we're going for.
And it makes me-- I guess the difference there is that with a native app, there's an explicit, installed step.
So it's a fine to expect an explicit uninstall? I don't know what we can do with-- if there's an imbalance where I can use up a lot of data, but it gets down to the user to have to get rid of it.
I actually think that's on opportunity to do better than native apps do it because I recently cleaned up my MacBook and found that the native Twitter app stores absolutely every image I've ever downloaded.
So I had three gigabytes of year-old images on my drive.
And we don't really intrinsically fault it for doing that, because we installed it.
And we gave it permission to use the disk intrinsically.
But it's still rubbish in a way.
And it could be a lot better.
So I think this is actually an opportunity to say, let's think about the whole problem and try and design a good solution how to communicate this to people and exactly how much do you keep, what do you delete-- all of these things.
It hasn't been handled well in the past, I think.
The difference is, of course, in your normal operating system, you can just go and find the file and delete it.
It's a bit harder in the browser.
So it's a problem.
But I think it's also actually quite a good opportunity to make it better than it currently is.
So as this technology becomes available in service worker already, and people start to build their stuff to work offline, are we going to see it be line accessibility and performance in that the offline first effort happens last in the development process? In part, it probably could happen because as I said, you have the simple approach of storing stuff for display.
But in the other instance, you can't avoid not doing it.
You can't avoid doing it.
If you do it in the architecture, the way Hoodie does it, it's built in.
You can't not have it, in a way, which is one of the reasons why we did it that way.
We believe firmly-- if you want to change the way people do stuff, you have to make it as easy as possible or as invisible as possible.
So it just happens automatically.
And that's kind of the gamble we're doing here.
We're making it so simple to the point that you don't have to think about it at all in the hope that will make it popular, because it will just happen to apps in a way.
If you designed the UI around it, the technical aspects are solved to you.
But you only have worry about the UI only and how you display and communicate things.
And you kind of want encourage people to deal with it for that reason just because we take a lot of the technical pain out of it.
I wanted to end on a slightly softer note.
But it's something you referred to a couple times in your talk.
So as developers, many people in the room-- we're using high-end Macs and high-end phones.
And we're not seeing the stuff we build the same way our users do, even down to font rendering.
I thought it was a crime that Chrome and Windows took so long to get DirectWrite font rendering, even though it's a huge benefit for so many users.
And I think it's indicative that so many of the engineers are on different machines other than Windows.
Should we be spending more time as developers on real-world devices, even something to simulate lower connectivity? Lower connectivity is probably a good idea at this point.
It's gotten a bit easier with a couple of desktop tools.
I know a couple of startups and offices that have a dedicated router, actually, that throttles HTTP connections and loses packets on purpose.
But, yeah, it's another extra testing step.
It's impossible to automate I think.
I'm not really sure how you would do that.
Yeah, it's an extra burden, definitely.
But depending on how much time you want to invest and how important this experience is, your user's experience in this case is to you, I definitely think it could be worth it.
As I said, it's not a bulletproof solution for everyone.
But yeah, the testing thing-- it's another thing on the list of the many things that we have to think about.
And I fully appreciate that that's not going to be in everyone's mind.
No, it's absolutely fascinating.
Again, just to make it work and have people not have to worry about it that much.
I've let my about my own biases let this Q&A overrun quite a lot.
But that's been absolutely fascinating.
Ladies and gentlemen, Alex Feyerke.
(alex feyerke) Thank you.