[APPLAUSE]
Hello, so as Bruce has said, my name is Sally Jenkinson
and I'm definitely not a synergy leverager whatever the hell
that is.
Um, if you could all avoid that, that'd be great.
I'm also sjenkinson on Twitter if you want to get in touch.
And I do, basically, work as a freelance consultant
and solutions architect through my company.
And we help people like this from big to small with project
discovery and research, and solutions architecture,
and doing transformation around all the systems and processes.
And we basically try to help people understand
what they need, the choices they have,
and how to, technically, make that happen.
And that's the kind of stuff that I typically talk about.
But one of the great things about my job
is that it is really broad and it constantly
challenges my perspectives on things that I take for granted.
And last year, I worked on a project
that was really different for me and that kind of
influenced this talk and its this
that I want to talk to you about today.
So, I want to talk to you--ooo sorry-- yeah,
I want to talk to you about data and specifically I want to talk
to you, as Bruce has said, about this idea of open data.
And everybody has gone absolutely nuts
for data over the last 10 years and I'm
sure the big kind of buzz words like this
are no surprise to you.
And last year, IBM actually published
a stat of 90% of the world's total data
has been created within the last two years,
and it's something that's really become so ingrained as part
of our projects.
And if you did on places like dribble,
for instance, for terms like dashboard,
everybody's at it, right?
So just a quick show of hands to kind of validate
this a little bit.
Who here collects data, whether that's in your projects
or perhaps you've got, you know, wearable
and you collect personal data?
Yeah so most of you, I'd say.
Who here kind of ingests it as part of the project?
So you might use something like an addressed base service
to do things like address completion or verification,
or talk with standardization, or you might do some geocoding
or something like that?
Kay, a few-- fewer.
Um, and who actually exposes it?
So does anybody create an API, or to put data out
there at all in any way?
Getting a bit less again.
And does anybody create content around data?
So does it inform the content that you put on your sites,
perhaps?
Even less again.
So there's interesting different ways
that we can actually engage with data.
And one of things I want to do today
is to hopefully kind of expose you
to a few more ways that you might
be able to use it in your work.
So personally, I really love data.
I'm a bit of a nerd, and it's not always directly
part of my job, as I've said.
But I really do find it fascinating,
and it touches so many different aspects of my life.
Now I obviously use Google Analytics--
as I'm sure many of you do-- on my own site.
Professionally, I kind of use it to identify
any issues people might be having and stuff like that.
But personally, I like to just look
for really weird things in it.
So up here that massive spike was when
people found my cookies page.
You can have a look at that yourself if you want to.
If I wanted to I could see which of my tweets were most popular.
So lots of services are building in this kind of ability for us
to analyze how people are reacting to us-- how
we're engaging with them.
Now my last FM profile is quite an interesting one
because what this is meant to do is
it's meant to show all the music that I've been listening
to over the last 10 years-- except what it actually does
is to show me how I've been consuming music,
whether it's been tracking or scribbling it or not.
And as you can see there's some massive dips there
which is when I've, sort-of, been
doing different things in my life
or using different services.
And I also make stupid stuff with all the people's data.
So this is using the Marvel API and D3
to help me to find out which the most popular characters
by the number of times they've appeared
in individual comics, or series, or stories, or events,
and things like that.
And I make stupid stuff with my data too.
So this is my-- utterly pointless,
soon to be retired because I can't handle the pressure-- tea
tracker page which details all of my tea drinking
habits since November 2013 and in case you're wondering,
today I have had cups of jasmine tea with the last one
(being at 9)40, which you could checkout live.
If you so desired.
But I'm sure, like me, when you make stuff
and you're not working with your own data,
you probably found yourself being deliberately limited
by some of the data providers out there
or maybe being bound by terms that don't quite
meet your needs of what you're trying to do with it.
Many of the database services that we actually use
have these very strict terms and conditions around usage,
or they can be prohibitive in other ways.
So they can end up locking off data from us, deliberately,
or restricting access in certain aspects.
The Twitter API is, obviously, really notorious for this
especially with regards to kind of taking away the access
that it may have initially given out.
But if we look back at the Marvel API this, for instance,
says, you can only use the data for non-commercial applications
that don't portray Marvel in a negative manner.
You can't change, edit or augment the content,
and any time Marvel's data appears
on screen it must be accompanied by Marvel's copyright notice
and a link back to their own site.
And they all say limit the delivery mechanism.
So, it's free at the moment but they reserve the right
to charge in the future.
And it's rate limited in terms of the terms and conditions
to 1,000 API calls per day, but I actually
logged in the last night and the documentation there says 3,000,
so you get this kind of conflicting message.
There are also rules around caching and content retention
in that basically you can't.
Well, you can to an extent but not long term.
And also if they terminate the API then
they could come up to ask you to delete all of your content.
So it may completely ruin whatever product
you built on it.
And this is my favorite-- so you agree
to maintain your apps and your systems in accordance
with industry standard quality levels
because, as far as I'm aware, we're still all yet
to agree on actually what a web app is
and whether they exist at all.
So I'm not sure weather we've got kind of industry standards
quality levels to a definition just yet but good for them
to put that in there.
So anyway, we can't always get our hands on the data
that we want but, of course, getting your hands on it
is only one part of the puzzle.
So sometimes we're restricted by whatever, whoever
has put it out there-- in terms of what we can then
do with it as well.
So actually the Marvel API is an example of data sharing
which is the release of data for restricted
purposes to restricted people or organizations.
And it's doing something with data that really matters.
So through using open data, the goal
is that your consumption, and your analysis,
and your presentation of data shouldn't be controlled
by whoever's published it.
And this leaves you free to use it in ways that they may not
have originally anticipated and it also
allows people to subsequently build on your work as well.
But OK, what is it that actually makes data open?
We've heard this term a few times now.
So that we've got a shared understanding of it,
we're going to use a definition.
So this is on opendefinition.org and it's
one of the most popular definitions.
It's not the only one.
And as you can see there's a really detailed explanation,
should you wish to read it, but it can basically
be summed up as, open data and content
can be freely used, modified, and shared
by anyone for any purpose.
So this isn't a new concept.
It's just one that kind of goes against some
of the more traditional attitudes of how
we approach data on the web.
And it's also something which is being standardized
a lot more as time's going on.
And in fact, there was recently a great diagram
from the people at The Open Data Institute in the UK
of what they call the data spectrum.
So this is basically everything from open data
on that side, which is just what we've
talked about-- which has an open license
and allows people to use it freely--
through that spectrum of it being shared
and it being out there but maybe has a license that limits use,
through to things that have named access,
so only certain people can access it at all,
or that it's internal so things like employment contracts
and policies.
And if you use open data, or if you put data out there openly
then you start to give people freedom.
So they can work with this data without restrictions
from copyrights, or patents, or other mechanisms of control--
so by opening up this data you could potentially
allow people to maybe republish the content
or data on their website, or to derive new content
or data from yours.
Maybe to make money from selling products that use your content
or data or to republish the content or data whilst charging
a fee for access.
And these last two are quite important to understand
because the data itself should meet all of these criteria
about being open so that the data itself
can be freely used, and modified, and shared
by anybody for any purpose.
But you might potentially choose a delivery mechanism
with a commercial aspect like an API
that requires registration or has different,
sort of, pay for levels of access.
So open data does not necessarily
mean unlimited free access of viral APIs for everybody.
So this is something that's both incredibly simple
but actually quite confusing at the same time.
And one of the reasons that I wanted to give this talk
was because there's quite often a lot
of different interpretations or misinterpretations
about openness, especially in different contexts.
And I think that this is an area that, it's
kind of like, frequently misquoted
and misunderstood by certain people.
And there was a great article recently
which discussed naming conventions-- again
from The ADI who you'll see crop up a lot in this talk.
And as here, in this quote, open data isn't something entirely
different, in terms of its formats,
it's-- it's not magical in its raw form,
it's just more about the mentalities that lie around it.
So you're probably starting to get the picture,
that actually a lot of this comes down to licensing
and you'd be right.
So these are some of the licenses for open content
which you might be really familiar with,
and in terms of databases it's again, really, really similar.
There are other licenses that enable
reuse, in which you might encounter, particularly,
around public sector information,
s so it's not just creative commons.
The open government license is an attribution license,
for instance, that covers both content and database rights
and it's mainly used for information made available
by central government-- so the UK has one, Canada has one,
I don't know if the Netherlands has one
but you'd probably have something similar.
And the OS open license is extremely similar
except it just instills that attribution is back
to ordinance survey instead.
Licensing is really boring so I'm not going
to spend a whole talk on that.
And--but let's look at actually why this is all exciting
because there's some amazing data out there.
And when you talk about open data,
you tend to find that a lot of the same sets
tend to crop up again and again.
So you get things sited like DBPedia which is great
because it essentially makes the content of Wikipedia
available in rtf format and also incorporates these links
to other data sets on the web.
And then we've got something which isn't super exciting,
but if you want to overthrow the conservative government
in the UK like Bruce you might want
to go in here, which is data.gov.uk.
This is a collection of data sets available from all the UK
Central Government Departments and number
of other public sector bodies and local authorities.
But then there's also data out there
about pretty much anything you could possibly
be interested in.
So if you like music-- a lot of music yesterday--
there's Music Brains which is the open music encyclopedia
and you've got the Million Song Dataset.
If you like geography, you've got some amazing earthquake
data and this is some of the data around earthquakes
this September just gone.
And if you're a real nerd-- I don't even
understand what half of this means
but-- you've got current and voltage measurement sampled
at 30 kilohertz from 11 different appliance types,
present in 55 households in Pittsburgh--
if you want to get really specific.
But personally, I lived in Indonesia for a long time
so I have a, like-- I have an interest in the country.
I like to keep up with what's going on there.
But even then, I'm not entirely sure
that I'd want data on all the billboards in Jakarta including
that orientation, but that leads to people making info-graphics
like this.
So there's something out there for everybody
but for me it's probably more likely to be a data
set of 10,000 cat images and their attributes--
or there's 20,000 dogs, if you prefer.
Or I'm sure you can do some really interesting
social analysis using the 1.7 billion JSONobjects
representing public Reddit comments, which
I'm sure you could get some really strange and interesting
stuff out of.
But you want it the chance is that it
is out there in some form.
So when I started putting this talk together originally,
I started collecting together a few of these data
sets that I thought might be quite interesting for people.
But then, as is always the case, somebody
sent me this link, which is just a repro full of some really,
really interesting things from the more bazaar
ones through to some quite big portals and standard sources
of data that you typically find.
So, not everything in here is open in the true definition
but have a look at the licenses and have a look at the stuff
there because there's some really great things.
So, in terms, of kind of using all of this-- consumption
is pretty straightforward and this
is the bit where I kind of stand up
here feeling a little bit like a fraud
because as we've seen from earlier quote, the data
itself isn't special.
In terms of the technicalities, it's kind of like most data
that you're going to be using-- or it is.
The key points, if you are going to be using
open data in your projects, are that it
has an appropriate license, that it's
of a good quality and level of detail for your needs--
and I'll come back on that latter stuff in a little bit.
But once you've found what you need,
you're commonly going to be working with data that we all
know and love, so formats like CSV, JSON, IDF,
XML-- different levels of love for each of those,
but pretty much what we're already doing
when we're working with it.
So there's nothing hugely special here
and you might be downloading it periodically
or you might have access to an API or something else.
Now one thing to watch out for is that some people actually
release data openly as they claim,
but it is actually stored as PDFs
and this is another example of this kind of misunderstanding
of the different kind of ways that you can go about things.
So as we all know, PDFs aren't really structured very well
or machine readable in, you know,
for passing in the same way that, say,
JSON is-- this wouldn't be considered desirable to people
so watch out for that when people do claim
to have it out there openly.
And then, in terms of, once you've
got it, once you've grabbed it, once you're doing something
with it, you obviously want to present it back
and there are loads and loads of different ways to do this.
So I'm not going to go into too much
but I'm sure we've all heard about libraries like D3,
there's Processing, uh, the Jam Sessions, a load
of ones that were name checked.
But of course, this is going to tie in hugely to what
you're doing, so what I'd encourage you
to do is to very much think about whether you need to just
slap a dashboard on there for the sake of it
and what end you're, sort of, you know,
you're trying to get to.
So think about the purpose again.
It may be that, actually, your interaction with open data
is much more functional and it's much more happening
in the background.
So think about whether we do need
to, sort of, have this visual aspects
or whether the data is hidden away.
But of course what we want to try to do here,
is to try and make it simple for people to understand the point
that we're making.
And of course, if we're making a point on the web,
we want to follow all the good stuff that we've heard
about yesterday, especially.
So we want to be using standard space,
web technologies, we want to make your performance which
can be tricky sometimes when we are working with large data
sets, and we want to follow progressive enhancements.
So all the good stuff that we've, helpfully already
had covered.
But these are still just websites and data
sets for the most part.
So they're kind of contained.
But I'd like to challenge you a little bit today.
So in order to do that, let's look at some slightly different
examples.
So I want to start doing that by telling a story about this guy
and this fire hydrant.
So this is Ben Wellington-- afraid
I don't know the fire hydrant's name
but-- Ben lives in New York and this
is a place where since 2012 they have had a comprehensive, city
wide open data policy.
And as you can imagine, all of that data being put out there
is great but it takes analysis to really maximize it.
So New York is also a place where
if your car is parked within 15 feet of a fire hydrant
it can be ticketed and towed away.
And now thanks to the availability
of the open data around fire hydrants in New York, Ben
ran a little experiment to see how much people were actually
being fined for parking near hydrants
like you do-- you know, a standard Friday night.
But he actually found something pretty interesting.
So he mapped the top 250 grossing hydrants
in New York, and our friend the hydrant on the previous slide
was found to be grossing $33,000 US a year in fines.
Now here's one of these ones down there-- so the big two
blobs.
And actually on the next block as well,
the second most profitable hydrant
was generating $24,000 US a year.
So that's a huge amount for two hydrants in close proximity.
So this was a bit of a puzzle as to why
and Ben did a bit more investigation
and found that these two parking spaces were actually
extremely confusing.
Basically, it felt like a trap.
So there is this wide curve extension between the street
and fire hydrant, as you can see--
which I think may actually have been a cycle lane.
So there's a bit of confusion whether it was actually part
of the road, part the pathway, or a cycle lane
and it's-- if that's different.
And additionally, the most confusing thing
was, as you can see where these cars are,
there are actually painted parking spots right
where you would be fined if you were parked.
So Ben kind of put this data out there.
The story was picked up by a lot of different people
and the Department of Transport eventually came back
and commented and they said, well DOT
has not received any complaints about this location,
we will review the road way markings
and make appropriate alterations.
And if you look at the fire hydrant now, on street view,
you get this slightly weird patching,
but you can see that it looks like this.
So they actually painted over a visual kind of restriction
to people so that they know they can't park there.
And now of course, this is one of the big challenges
against open data, that people kind of, you
know, on the-- maybe-- business end of the spectrum--
on the less good side-- they always raise this question of,
how happy people are to be losing out
on all their lovely revenue, in terms
of people like this coming along and going well actually.
But, of course, in terms of the public good, open data,
and Ben did a great thing here.
But it's not actually just about companies
losing money because the economic impact of opening up
data has been the subject of a lot of recent studies,
showing benefits in terms of both generating value and cost
savings.
So you've got things like the 2013 McKinsey and Company
report which estimated that open data could generate $3 trillion
in additional value per year across seven different sectors.
Now, obviously, the story in New York came out of the fact
that data was mandatory to release,
but people also want to be seen as being transparent.
And there's a great example of this
in a book called "Thinking In Systems"
which has a little bit about how in 1986 there
was new federal legislation which required
US companies to report all chemical emissions from each
of their plants.
And this became publicly accessible
through people applying for it through freedom
of information requests.
And actually, what happened was that many
of the emissions being generated weren't at illegal levels
or breaking any rules but you started
to get wonderful-- I suppose, the Buzz Feeds of their day--
reporters were getting the information
and starting to put out lists like, the top 10 polluters.
And so, actually, what happened was
that due to things like that, within two years
nationwide emissions had decreased by 40%
because transparency isn't just about giving people access
it's actually about giving them the freedom
to analyze your data and to share it and to reuse it.
And as we see with Ben's hydrants, often
to really understand problems, data
needs to be visualized, perhaps, or combined with other data.
And this requires that the material
is able to be freely used however you want to.
Now also with the polluting data,
freedom of information requests are fantastic
but it would save everybody an awful lot of time
if it was actually open by default.
So I want to share one more example of how
we can do great things when data is more open, when
we've got more freedom.
So, I'm a big advocate of thinking about the experience
that users have when you consider
any kind of technical or bill decision and one
of the benefits that open data can bring
is to be able to combine these data sets in ways that answer
human questions rather than being purely data led
and just putting it out there.
This is what Mapumental have done.
So they've created a tour that helps people
to find a viable part of the country
to be able to live in based on the amount of time
that it takes them to get to work or to do life based
things, rather than thinking in terms of miles which
is what we typically do.
So, they created a frame work to do this.
But then once you've got that, you can apply that
to much bigger problems than, you know,
say where people should live in order
to be able to get 15 minutes more sleep in the morning.
So you can maybe create a tool for planners
to think about public services as this quote shows really
well.
Because in Wales, as it really uses to illustrate,
the amount of distance, again, isn't
representative of the time that it
takes to get some of these key places.
And another one, another sort of question
that you might have that isn't easily answerable from raw data
is around emergency services.
So you want to be able to ask human led questions like,
how quickly could four fire engines get to a specific post
(code at 2)00 AM or, could we get a helicopter somewhere,
you know, within the certain amount of time.
And so tools like this can only come
about when there is openness and lack of restrictions
on the data's usage.
The more that we place restrictions,
the fewer these benefits we're actually likely to see.
So we can help to improve efficiency and effectiveness
and we can really, really measure the impact as well.
And data's this really key resource
in the modern age and the more that we can release it or have
the ability to consume it without limitations
then the more likely it will be that we
can to go out there and build, really,
some innovative products that answer these big questions.
But, well, it's most of my focus today is obviously on the web.
You don't just have to work with open data in digital ways.
So people are also using these open principles in conjunction
with the physical world and to create, actually,
pieces of art.
So by opening up your data to any usage whatsoever,
you can get these amazing interpretations
through completely different formats that
were never originally intended.
So most limiting digital terms don't actually
think about this.
So if we think back to the Marvel API,
how can I meet these criteria if I
want to use their data to do something physical.
So I'm sure that I'm probably going
to be breaking their caching limitations instantly
and I'm not sure really how I'm going to do a hyperlink back
to their homepage, if I'm making something physical but.
As a half way house, because I know that we all
like to cling onto the internet, you've
got schemes like opensensors.io which
aim to take this enthusiasm around the internet of things
and to in-- encourage the publication of open data
within your physical projects through the provision
of a platform that facilities this.
And they give examples of things like environmental censoring
projects from communities.
You might want to monitor their air quality or water quality.
But then you can put that data out there and also, maybe,
combine it with data from other communities on a real time
basis or on a historical basis as well.
But if you want to take it further, and completely
step away from the internet, there's
a chap called Doug McCune.
He does a load of artistic projects based on data
and I'm going to show you some in a sec.
But his particular slant, is that he
loves to take the most horrible, nasty data he can find
and to really make something beautiful out of it.
And I think that this is particularly interesting to me
because having come from a development background,
Doug really nicely illustrates this shift in perspective
because he started off really similarly.
So he was just using data in his projects
but then he found his relationship changing
and there's a great talk that he gives called,
Desperately Trying to Remove the Air Quotes
Around the Word Artist, which documents his shift
in identity from a developer through to a fully
fledged artist.
So here's one of his pieces.
It's called Drunk Traffic Map of Portland DUIs.
And it's a print-- the number of driving under the influence
arrests on each street in Portland
over the course of 10 years.
And this is a Haiti earthquake tree trunk
map which is a representation of the 2010 earthquake
that destroyed much of Haiti.
And this is the laser cut out of different kind of bamboo
and it shows the shake intensity data
that radiates out of the epicenter near Port-Au-Prince.
And this one is a 2014 South Napa Earthquake 3D print.
The source data here comes from the earthquake data
source I showed you earlier.
This, sort of, the contours of the 3D print
represent the peak ground velocity
which is a measure of how intense the data shaking was
which is really closely linked to damage, as you'd expect.
And this one's my favorite, I think.
It's the Bay Area Homicide Constellation Map
and it's a map of murders throughout the Bay Area,
what he's done here is he's taken the data for murders
in 2013 which are, sort of, connected in terms of geography
and he's drawn the lines between them
and then just to give it a bit more
context he's put the data for 2014
on just as individual points.
So lovely, lovely work-- horrible, horrible data.
And this is some work from Stefanie.
So, Stefanie works with a range of open data.
She's done some great talks recently,
which you should checkout, but through her project Air
Transforms she focuses on the question, what if we could
really see and feel the burden that air pollution places
on our bodies.
And so she's created, in her words,
a series of wearable data objects
which are pretty much kind of like to these necklaces.
And she's used air quality data from Sheffield in the UK
which as it says in the quote is a-- it's
a really sort of notorious city in terms of the history
there, around air pollution.
And she's taken that data and she's
applied a couple of different categorisations.
So you've got the physical kind of shape
and you've got a color scheme as well.
So that ranges from green being quite good through to, sort of,
reds and orange.
And so these necklaces end up looking
like this where you've basically got a good week here,
which is very nice, friendly colors, friendly, sort
of, shapes, nice and rounded and soft.
But it's really apparent, the message
that she's trying to convey, in terms
of if you look at Bonfire night you get something that
is very physically jarring and I'm sure that I'd
be stabbing myself in the neck every time
I tried to wear this.
Bu But it's really obvious, both in a kind of physical sense
and a visual sense the impact that bad air quality
data is having here.
And this is what things like open data can let you do,
so Stephanie, Doug, and Ben, in their own way,
they're using open data as a way to get their personal message
across.
So you might be somebody who wants to actually make
a statement, to get your message across like that,
but actually it could be that you just want
to solve your own problems.
So there's a Danish lady called Tina Muller,
and I really like that she saw the problem
that people with bladder weakness
don't always feel confident in actually
kind of going out and exploring the city, which for somebody
who's come to Amsterdam and likes to wander around,
I really take that for granted.
But what she's done is to use open data
to solve that very niche problem of allowing people
to have confidence and be able to get out about.
So if there is data out there without restrictions,
you can actually use it to help make your lives better
or to make a big impact.
Up to this point, the examples I've
showed you have pretty much been quite self-contained
and had a single focus, but there's a great Ted Talk
out there by Tim Berners-Lee, which
you can find on the data.gov.uk site amongst other places,
but he basically talks about the growth of the internet
and how originally he came along,
he asked us to put all of our documents on the web,
and now we have, and now we can't stop.
We're addicted to it, and it's just
more and more cats every day, and it's great, it's fantastic,
but what he also wants us to do is
to share the data that underpins those documents but to maintain
this original powerful concept of the humble hyperlink,
so the infinite navigable journeys
that we can go on that make the web so special.
Because it's not just about these individual silos of data,
it's about actually what you can do
with them and, more crucially, the relationships
you can force between them.
So by having this freedom to reuse and combine data,
we gain some great insight into solving problems.
And this is something that we've actually
done a lot through the ages, and it's not exclusive to the web
at all, but the web can really help facilitate it
and to make it more accessible to people.
So an example that I love, not from the time of the internet,
is the relationship between drinking water pollution
and cholera in London in the 19th century,
and this was discovered by Dr. John Snow, who basically took
the data around deaths from cholera-- we're back
to the really horrible data-- but he
combined with the location if water wells, specifically
ones around Broad Street in London, which are up there.
He did that, he worked out this relationship,
because previously they had thought
that this was an airborne thing maybe,
and the thing that I like about this story
is that, interestingly, there was one significant anomaly,
so none of the monks in the adjacent monastery
contracted cholera at all.
Actually, again, when you dig further into this
and you find out the human stories and the relationship
with the data, this wasn't actually an anomaly,
it was further evident, because the monks only
drank beer, which is how they protected themselves,
and they brewed this themselves.
And also residents in, or sort of near the brewery,
they also weren't affected because
of the result of fermentation of the contaminated water.
So the beer was safer to drink than the dirty water
from the street pump.
So this eventually led to the building of London's sewage
system, and it hugely improved the health
of the general population.
So we see the same benefits now.
When we can standardize, we're able to use data
in combination, the more easily patterns will be identified
and new knowledge can be gained.
So this principle of linking up data is a really powerful one,
and it's gained a lot of prominence
since it kind of started to get talked about in 2009.
And when you start to view data as being linkable,
you build up this picture of the amazing resource
that we have out there.
You may see some names in here from the previous sites
that I showed you, so things like DBpedia.
But, of course, as we can see from this diagram,
where there's great potential to combine data,
there's also huge potential for things to get very, very messy,
and I really don't envy the people that put this together.
But, as Lisa was telling us yesterday,
this is where standards come in.
So the concept of linked data itself is a standard,
so it's defined as a set of best practices
for publishing and connecting structured data on the web.
And it comes with guidelines to how
best approach it so key technologies-- obviously
HTTP for requesting or receiving data, URIs for being
generic identifiers for entities or concepts,
and probably most relevant, RDF which, if you don't know,
is a generic graph-based data model with which
to structure and link data that describes different concepts.
So the basic principles are reasonably simple.
So it's to use the RDF data model to publish
your structured data on the web and then
to use RDF links to interlink data
from different data sources.
So very simply, you could have one data set,
which is described in a really structured way, which
has a semantic relationship to another
and that, in turn, may have relationships to other data
sets and so on.
Now, one of things we need to be careful about when
we start linking up data is this potential
for misinterpretation.
When we think about the concept of linking,
some scientists have actually argued
against this, because their data may be collected
under very specific conditions, and you
don't want start combining that with data
that wasn't meant to be combined with it in the first place.
We have all, no doubt I'm sure, seem some wonderful examples
of just how wrong data can go when it's taken out
of its proper context.
So you start seeing things like this,
and one of my favorite sites-- cheap joke, sorry-- yeah,
one of my favorite sites is something called
Spurious Correlations, which teaches us
that correlation and causation may not be exactly
the same thing after all.
Who knew?
So Nicolas Cage is a very, very dangerous man
for a few reasons.
Just one more.
Ate a lot of cheese last night, could have died, didn't know.
Anyway, interoperability can be really important,
but this is also where licensing gets really tricky as well.
Like I said, licensing, boring.
Not going to cover it too much, but one of things
I really do want to stress is you just
need to think about this a little bit
when you do start using data with different licenses.
So you may have used some data in your projects
which requires attribution.
You might have done that because you're a good person, so you've
attributed it correctly, but you might then
want to build on top of it.
Then you might want to release your new project
as being something in public domain,
completely open and free.
But that, obviously, is going to contradict
the terms of the attribution that the other license might
have placed on you.
So if you think about how this can
start to work when you've got lots of different data
sets, all with different requirements,
it can get really messy sometimes.
So there's a really useful tool available down here,
which can help to guide you in these matters,
and I'll put these online later if you want grab any links.
But we've looked at people using open data,
we've looked at how we can link it up,
but this publishing aspect is really crucial too.
So additionally, I'd like to ask all of you here
today if there's data that you could be publishing
or whether it's something that you could perhaps think about
with the projects that you work on,
whether you could ask the question of other people
as well.
So for me, one of the best things
about the idea of open data is really the fact
that people can take it and use it in ways that you've never
dreamed of.
And this quote is one that's mentioned a lot,
but I always think of it in conjunction
with something called the Relevance Paradox.
This is basically where people are
unaware of certain information that
could help them, because they don't necessarily
know that it exists.
It's one of the great arguments for people putting data
out there openly.
So as we speak about the [INAUDIBLE],
data itself isn't necessarily that useful.
So unlocking that value and the relevance
is the really tricky part, and you might not
be aware of the relevance that your data even has,
but others may be.
The other good thing is, you might not
have the resources or the skills able to unlock that yourself,
but by putting it out there, you look really
great and altruistic and actually everybody else does
the work for you, so something to consider.
But to finish it off, I want to talk to you
about what sharing your data openly might entail.
So we all remember [INAUDIBLE] data, really useful stuff.
Remember also the pointless data sets
other people put out there, so the billboards in Jakarta.
Maybe once I stopped collecting this,
maybe I should release it openly,
because it might have some use to somebody else.
Who knows?
So if I was going to do that, the first step
is obviously to identify what should and shouldn't
be published.
And this typically involves an assessment of the data
you have, the quality of it, and also
to plan the processes that are going to be involved
in the date of publication.
So many organizations, when they come to this point,
when they're looking at what they have,
they start with the data that they
might be releasing already, but it may not necessarily be open.
So that might either be because of the restrictions
that they've previously placed on it,
or maybe because of the format that it's been in.
So things like PDFs are a great place
to start, because it's reasonably simple
to kind of take that, dig the data out, apply a license
and put it out there.
But you do need to be careful if you do this
because, as we've seen with the model API,
sometimes you get conflicting terms-conditions kind
of limitations, because typically-- I'm
sure everybody has seen the terms and conditions that
basically say, you can't do this, you can't steal this,
you can't do anything with it, and if people want
to put open data out there, they always forget
about the terms and conditions on the bottom,
so watch out for that.
The things that I would need to think about
would be putting some clear licensing and usage information
so that people know how they can use my data.
And I want to make sure the structure was good
and the quality was good, so data
should be cleaned and formatted in a logical way
and presented ideally as machine readable.
You'll need to think about the support.
So data that's simply put out there which isn't maintained
will likely not be very desirable.
And it's the gift that keeps giving, because last night I
actually went to fix a bug that I've noticed in my MarvaLytics
page, because I thought, oh, two people are
bound to look at that and tell me that's it's broken.
This is just something that I made about a year and a half
ago when I was bored at my lunch breaks,
but actually it was quite useful because I
went there and realized that the API had an issue,
and so I did the thing where you go into the community page,
and you start to see reams and reams of people complaining
about stuff, and now it's really sad.
It looks like that they're kind of just letting it slip,
and that's a really, really important point.
So if you are putting information out there,
people are going to be building products
on that, so you need to think about how you're going
to support it and what happens if it does fall over,
and is it going to be out there forever
or are you going to communicate that to people?
But anyway, on the structure and quality front,
data may also not be accurate, it might not be complete.
So my data, I know, has some issues,
namely because I either forget to track things,
because I always meant to build in offline capabilities
and just never got around to it, so I
might have some timings being completely out,
and I didn't factor in things like time zones.
So this wasn't something that I was thinking about when I first
started collecting data, which is another important point.
But there's some [INAUDIBLE] of you
as well that if data is kind of out there in the open
and especially if it's able to be fed into by people,
then reliability is going to be at stake.
But you need to balance that against the fact
that if it's a closed silo entirely, then that opportunity
isn't there.
And actually by opening it up, you
might be able to grab data that you would never
have possibly gotten before.
So once the data has been identified,
this extract compilation might be needed,
and this particularly happens if personal details are held.
And the data might actually need to be cleaned up as well.
And so this point about personal data
is a really, really important one,
because many of the data gathered online or held
by companies is actually around individuals,
and this especially has become the case
in the last 5, 10 years.
When you start dig into data about individuals,
you get into anonymization issues,
and anonymization or true anonymization
is notoriously really difficult to do safely.
And as such, this usually means that open data that does
relate to individuals is typically
kind of aggregated statistics, so you might have
like the results of the census rather than Bruce's
favorite animal and how many cups up tea he's drunk today.
But the exception is here when data is made available
by the relevant individuals themselves
or where there's explicit consent of the affected
individuals, or where there's actually a mandatory reason.
An interesting example here is in Scandinavia.
Everybody's tax return data is openly available,
and this includes everybody from the government
to celebrities to your neighbor next door,
and if you think about it, this kind of openness
can actually do some real good.
So it can help to alleviate pay gaps between the genders,
or it lets you know that your colleague is
on a similar wage to you and that nobody's being screwed
over, or it just gives you enough data
to kind of know your worth if you're doing research generally
into what might be available in different careers.
But I know that in the UK, this is something that we would not
be comfortable with at all.
It's also something that Scandinavia dialed back a bit,
so they've done more to protect the individual's rights.
So you need to be very careful with this balance.
When you've actually sort of done your extraction,
maybe done some kind of compilation aggregation of it,
you might need to do some cleaning as well.
There are lots of tools that are available to help you clean up
data.
So OpenRefine is one that's really widely used.
It can clean, it can transform your data,
and you can also use it to extend it with web services
should you want to do that.
But as a tool, it's really, really useful
and I'll actually recommend it if you're
working with data in any way, not just thinking
about releasing it openly.
And you've got other guidance as well,
so there's a resource called Clean Sheet, which
is useful for giving you guidance around how
you might want to put your data out there
in terms of spreadsheets.
Now finally, you want to share it,
so you want to put it out there, and this can take many forms.
So you've got putting the data out there,
and then you've actually got helping people to find it,
because that's a really important thing, right?
So in terms of actually putting it out there,
there are a couple of different schemes
to give you some guidance again.
The first one I want to talk about
is 5-Star Data, again, from Sir Tim Berners-Lee.
Honestly, the guy just doesn't stop doing stuff.
And these are some levels, so this is the 5-Star guidance.
So this goes from basically having
your data available on the web in whatever format
but putting it out there under an open license,
through to starting to make it better.
So make it structured, making sure it's not proprietary,
and then starting to use URIs in order to actually give it
a place to live and to allow people
to link through to your data sets as well.
So you might actually want to host this directly,
as we've talked about, you might want to create an API,
you might want to put it on a torrent site,
you might want to house it on one
of the portal sites that were in the repo that I showed you.
But one of the things that's really important
is that if you are taking data, if you're building on it
and you want to re-release it, think about the structure
early on.
So don't make people have to scrape your data out
of your beautiful visualization that you've built.
Give them a way to access it directly.
So aiming to give a less technical and more kind
of well-rounded view of what a good data set is,
you've got the open data certificates which
are pioneered by, again, the Open Data Institute, and these
also provide information around rights and licensing,
around documentation, and guarantees about availability.
So this is what we talked about in terms of the support
and the ongoing kind of information.
What this is, they basically act like a reference sheet,
and it contains information that may be of use to re-users
of your data.
So what we want to do here is we want
to show that it's easy to find and use and share data.
So in summary, we now live in a society where
data is so entrenched in everything tat we do,
and the information we can get from that is absolutely key.
So data now underpins much of how we actually
live our lives as well as our projects.
So we collect it, we use it, and sometimes we might share it.
But data, as with documents, is a really fundamental element
of the web itself.
So rather than data being something
which is purely hidden below applications,
we can really bring it to the forefront
and make it more powerful and use it to gain benefits
either indirectly or directly.
We can do this by following some stats like these.
So using data, publishing data, thinking
about how our data fits into the bigger picture,
whether it can be linked with sets to enhance its context,
by removing these unnecessary restrictions
and promoting freedom and creativity of usage
and by embracing standards and open web technologies
with better, more freely available data,
society can hopefully start making these better decisions,
improving costs, being more creative,
and sparking new innovations that would not otherwise
have been possible, so some of things that we've seen today.
By consuming or publishing open data,
you too can help promote this.
It'll really help to make the data grow as well.
So thank you very much.
[APPLAUSE]
That was epic, Sally.
Thank you very much.
We're not going to do Q&A now, because many of the questions
are going to be equally applicable to the following
talk, and maybe we'll combine it then.
There were so many different conversations going
on during that talk, Sally.
Something I learned is that you should drink beer,
because that prevents cholera, and that because cheese eating
seems directly correlated with getting tangled in bed sheets,
now I know why in the Netherlands
I only ever get a quilt rather than sheets,
so that's a useful thing to be aware of.