Remixing web data for your hacks the easy way

1
Remixing web data for your hacks the easy way

• YQL the can opener of the web

2
Hacking together systems in 24 hours is a lot of
fun.
3
But you want to spend that time thinking about
the interface.
4
And not how you get to the right data in the
right format.
5
The web is full of juicy and long lasting data.
http//www.flickr.com/photos/clspeace/162336973/
6
And there is a lot of it around.
http//www.flickr.com/photos/tudor/2981410947/
7
However, our attempts to get to it can be clumsy.
http//www.flickr.com/photos/lumachrome/2140368742
/
8
What we need is an easy way to get to that data.
http//www.flickr.com/photos/careytilden/115435226
/
9
We had a way to do that for quite a while now.
10
http//pipes.yahoo.com/
11
Pipes, however is high end technology
http//www.flickr.com/photos/axio/2346342672/
12
Were developers, not interface users.
http//www.flickr.com/photos/codepo8/2278641937/
13
So for a long time people asked Yahoo for a
command line version of pipes.Can we have one?
14
Yes, we can!
15
The Yahoo Query Language, or short YQL is a
unified interface language to the
web.http//developer.yahoo.com/yql/
16
Using YQL, accessing the web and its services
becomes as easy as SQLselect what from
service where condition
17
Say you want kittens in your hack (who
doesnt?)select from flickr.photos.search
where text"kitten"
18
Say you want kittens in your hack (who doesnt?)
19
Say you only want 5 kittensselect from
flickr.photos.search where text"kitten" limit 5
20
Say you only want 5 kittens
21
Nice, but where can you get this?
22
YQL is a REST API in itself, and it has two
endpoints.
23
The public endpoint does not need any
authentication.http//query.yahooapis.com/v1/pub
lic/yql?qqueryformatformat
24
The private endpoint needs oauth
authentication.http//query.yahooapis.com/v1/yql
?qqueryformatformat
http//developer.yahoo.com/yql/guide/authorization
-access.html
25
Output formats are XML or JSON. JSON also allows
for a callback parameter to use the output
directly as JSON-P.
26
http//query.yahooapis.com/v1/public/yql?qselect
2020from flickr.photos.search20where20text22
kitten22limit205formatjsoncallbackmeow
27
(No Transcript)
28
You can mix and match several web services using
the in() command.
29
Guess what this doesselect from
flickr.photos.info where photo_id in (select id
from flickr.photos.search where woe_id in (select
woeid from geo.places where text'london,uk') and
license4)
30
Find photos in London, UK with a Creative Commons
By license and give me all the information you
have about them.
31
select from flickr.photos.info where photo_id
in (select id from flickr.photos.search where
woe_id in (select woeid from geo.places where
text'london,uk') and license4)
32
Using a command like this and some out-of-the-box
UI elements like Yahoo Maps and the YUI carousel,
you can build something very quickly.
33
http//isithackday.com/hacks/cantine/index.php?loc
coventgarden
34
Instead of selecting all the information you can
also limit the resultsselect name,url from
upcoming.venue where metro_id in (select id from
upcoming.metro where search_text"stokey")
35
select name,url from upcoming.venue where
metro_id in (select id from upcoming.metro where
search_text"stokey")
36
The diagnostics part of the returned data shows
you what happened and how long it took
37
Heres what Yahoo offers you in this format
flickr.photos.exif flickr.photos.info flickr.photos.interestingness flickr.photos.recent flickr.photos.search flickr.photos.sizes flickr.places flickr.places.info geo.places geo.places.ancestors geo.places.belongtos geo.places.children geo.places.neighbors geo.places.parent geo.places.siblings geo.placetypes gnip.activity local.search music.artist.id music.artist.popular music.artist.search music.artist.similar music.release.artist music.release.id music.release.popular music.release.search music.track.id music.track.popular music.track.search music.video.category music.video.id music.video.popular music.video.search music.video.similar mybloglog.community.find mybloglog.member mybloglog.member.contacts mybloglog.member.newwithcontacts mybloglog.member.newwithme mybloglog.members.find mybloglog.stats.find upcoming.category upcoming.country upcoming.events upcoming.events.bestinplace upcoming.groups upcoming.metro upcoming.state upcoming.user upcoming.venue
38
Heres what Yahoo offers you in this format
search.images search.news search.siteexplorer.inlinks search.siteexplorer.pages search.spelling search.suggest search.termextract search.web social.connections social.contacts social.presence social.profile social.updates weather.forecast yahoo.identity yap.setsmallview search.images search.news search.siteexplorer.inlinks search.siteexplorer.pages search.spelling search.suggest search.termextract search.web social.connections social.contacts social.presence social.profile social.updates weather.forecast yahoo.identity yap.setsmallview search.images search.news search.siteexplorer.inlinks search.siteexplorer.pages search.spelling search.suggest search.termextract search.web social.connections social.contacts social.presence social.profile social.updates weather.forecast yahoo.identity yap.setsmallview
39
You want even more?
http//www.flickr.com/photos/verylastexcitingmomen
t/3123597774/
40
Alright, how about this?
json microformats rss xml
atom csv feed html
41
Telegraphs headlines anyone?
select from html where url'http//www.telegraph
.co.uk/' and xpath'//h3/a'
42
Telegraphs headlines anyone?
43
You can dynamically create YQL queries to collate
several sources
44
Then use cURL to pull them off the web with one
single http request!
45
YQL caches and compresses the results for
you.Pretty easy, isnt it?
46
Heres another fun partAnyone can be part of
the YQL interface.
47
All we need the data provider to do is to create
a schema that explains their data
structure.http//developer.yahoo.com/yql/guide/y
ql-opentables-chapter.html
48
And people do
amazon bitly delicious dopplr etsy friendfeed github Greader guardian imdb iplocation lastfm nestoria netflix nmm nyt opensocial search shopping social twitter update.groovy weather wesabe whitepages yahoo yelp zillow
http//github.com/spullara/yql-tables/tree/master
49
For example the national maritime museum
select from nmm.archive.search where
searchtermhoratio nelson'
50
select from nmm.archive.search where
searchtermhoratio nelson'
51
Thats pretty cool allowing anyone to be part
of this interface.
52
The only shame is that you cant do all the
things in YQL that you can do in Pipes for
example string manipulation.
53
We wondered how to make this possible.One thing
we didnt want to sacrifice is the simplicity of
the language itself.
54
So instead of inventing an own language, we
decided to piggy-back on one you already know.
55
YQL execute allows you to embed JavaScript in the
open table schema that runs on the YQL server and
converts the data for you.http//developer.yahoo
.com/yql/guide/yql-execute-chapter.html
56
For example you can augment an existing service
to do something different.The following example
adds a rank to search results.http//www.yqlblog
.net/samples/searchrank.xml
57
(No Transcript)
58
(No Transcript)
59
Stored as XML this can be used in a YQL
queryuse 'http//yqlblog.net/samples/searchrank
.xml' as searchrank select from searchrank
where query'pizza' and dispurl like 'pizzahut'
60
(No Transcript)
61
Anyways, the easiest way to start with YQL is
to use the console.http//developer.yahoo.com/yq
l/console/
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
(No Transcript)
70
(No Transcript)
71
Of course, you can also spend half the hack day
reading API docs ?
72
Check out some code examples on.http//isithackd
ay.com/hacks/ohd-london
73
THANKS!Chris Heilmannhttp//twitter.com/codepo8