Flags and Lollipops - Bioinformatics Blog

Aggregating activity from Twitter

2009-06-13T22:50:00.004+01:00

Update: you can't follow a specific set of users using GNIP any more - their feed is equivalent to the 'spritzer' method in the official Twitter API.

Interested in building a real time aggregator for Twitter? Who isn't? You have lots of options:

Just the vanilla API

Simply call user_timeline for each user that you are interested in every x minutes.

The standard rate limit on the Twitter API is 100 requests per hour e.g. checking 25 users every 15 minutes is pretty much the best you'll be able to do. If you're a lazy chancer you can try and get your application whitelisted which removes rate limits.

Good:

Very simple

Not so good:

Too simple - won't scale.
Slow update time (while the number of calls you can make per hour is limited)
Seeing so much redundant data returned for each call makes the internet cry.

Vanilla API + robot

Create a new Twitter account, log in and follow the people you're interested in aggregating tweets from. You don't have to follow people manually - you could do it programmatically using the friendships/create API call.

Now just check the friends_timeline for that user as often as you like (up to the hourly rate limit, obviously). Page through results if necessary.

Twitter has some (sensible) rules about follower / following ratios. Once you're following ~ 800 people further follow requests will be blocked; you have to wait until you have more followers before adding anybody else. You can't whitelist your way out of this.

Good:

Again, pretty simple.
Better update time (aggregation within a couple of minutes of a tweet)

Not so good:

Can only follow ~ 800 people before Twitter starts blocking your follow requests.
Users will know that you're aggregating them (is this a bug or a feature?). Can't keep following / unfollowing people - they'll get spammed by emails telling about it.

GNIP

GNIP works activity streams from a bunch of different web 2.0 sites. Here's how it works in a nutshell:

you set up a GNIP account
you add rules to your account ("give me all tweets by @twalf" "give me all tweets by @ianmulvany") and set up a web hook (a script on your server). You can have up to 25k rules per site for free.
GNIP receives data in real time from Twitter
If any data matches your rule set then GNIP POSTs to your web hook with some metadata about the matching tweet (a unique id, the tweeter's username, a URI for the actual message)

Now you'll get pinged whenever anybody in your rules tweets - in close to real time.

Rules can be added programmatically or by hand. GNIP's API docs are pretty opaque but it's actually a fairly simple, efficient system once you've gotten to grips with it.

Unfortunately the metadata that gets POSTed to you doesn't contain the actual tweet. For that you have to go back to Twitter using the supplied URI, which points to the message in XML format. Remember that there's a rate limit on the Twitter API so by default you won't be able to aggregate more than a hundred messages per hour. This sucks. Whitelisting is pretty much the only way you're going to overcome this.

Twitter on GNIP is unique in this respect; none of the other services require you to call the originating site to get messages. It's especially annoying as tweets are only 140 characters long - it's definitely not a space / bandwidth issue!

Good:

Fast update time (pretty close to real time)
GNIP infrastructure can help you aggregate from other sites (Digg, Delicious...) in the future.
Follow up to 25k people for free and without scaling issues.

Not so good:

Relatively complex.
GNIP can be a bit flaky - occasionally it goes down and you lose updates for a few hours.
Requires whitelisting by Twitter once you're collecting more than a hundred tweets p/h.

Twitter streaming API

Twitter has a streaming API in alpha.

You can follow up to 200k users by POSTing their ids to http://stream.twitter.com/birddog.json - after you've been approved by Twitter and signed a usage agreement.

You can follow up to 2k users for free using http://stream.twitter.com/shadow.json which is similar.

You can follow up to 200 users for free using http://stream.twitter.com/follow.json which is similar.

Once you've opened a connection to shadow or birddog it'll never close. When a followed user tweets it'll come down the wire as a line of JSON (ending with a carriage return). Think Comet.

Good:

As fast an update as you're ever going to get.
Don't need to rely on third parties (like GNIP)

Not so good:

Still in alpha.
Need an agreement from Twitter to follow more than 2k users.
Complex (in that it requires you to move away from reactive, asynchronous scripts towards an app that can keep an HTTP connection open for hours)

Postgenomic hiatus

2009-03-20T14:12:00.003Z

A couple of weeks ago I switched off the Postgenomic aggregation pipeline.

This is mainly because the pipeline scripts were hogging disk / memory resources on the server which it shares with a bunch of other applications. I'm not sure exactly where the process is sticking; but to be honest it's not a complete surprise.

Writing a blog aggregator is actually pretty easy; the hard part is dealing with all the weird edge cases. I haven't been paying close attention to the Postgenomic pipeline recently; I think what's currently going wrong is a combination of slow queries across what's now a very large database and one or more odd posts or blogs clogging up the pipeline (I'd post more details if I had them).

NPG doesn't officially support Postgenomic any more, though it does host it ably. Patching the code is something I do 'on the side', which is why it hasn't been fixed yet - I'm really pushed for time with other projects that need to take priority and will be for at least another three or four weeks.

In the meantime, no new blogs will be picked up and posts won't be aggregated at postgenomic.com. The site itself and the API will continue to work.

If you use postgenomic.com for any mashups or scripts then I apologise for the outage - sorry! It will be fixed, it's just the timing that's an issue.

In the meantime, please consider switching to Nature.com Blogs - the user facing features aren't as complete but the backend is. What's more it's fully supported by NPG developers and IT staff.

Paperview

2009-02-23T14:11:00.002Z

About to put some software here, need to know the permanent URI.

Er, check back later?

Sony eReader on OSX

2009-01-30T12:00:00.003Z

For future Google reference, the Sony PRS-505 is perfectly compatible with OSX (just like the Kindle). If you plug it into your mac's USB port it should show up as a new disk image ("Untitled", but still); just drag and drop EPUB, PDF or text files into the database/media/books directory et voila.

Unfortunately it won't charge through USB while plugged into Macbook Pros (don't know about other laptops or desktop Macs) - seems like there's not enough power. I had to find a PC to recharge at.

Graham Lawton and Darwin was Wrong

2009-01-25T14:31:00.003Z

New Scientist this week has an eye grabbing cover.

The cover sports a big green tree with the words “Darwin Was Wrong.” I hope they sell a lot of magazines with that load of tripe, since they certainly were not thinking about the generations of school kids and church-goers who will now be treated to that cover in every creationist power point presentation between now and the Rapture. How many people do you think will actually read the article to discover what it was, precisely, that Darwin got wrong?

(from EvolutionBlog)

There's some fair (I think) coverage in a couple of places like Sandwalk and lots of not so fair coverage everywhere else.

I don't really understand what the big deal is. How dare a mainstream publication use a sensational cover to help sell copies? How dare a journalist cover a story that might be quote mined selectively by creationists?

It doesn't really matter if you're on a magazine front cover or tucked away on pg 127 - if somebody wants to quote you out of context then they can. Surely the thing to do at that point is to confront the person doing the mis-quoting, not to berate the original author.

The cover does make a lovely image for ID proponents to include in powerpoint presentations, yes. But why should New Scientist care? Why should they pander to creationists and sell fewer copies of a magazine that probably does more than any number of science blogs to get schoolkids interested in science?

Graham Lawton is not the enemy.

(New Scientist is full of crap sometimes, though)

Unforgiving UTF-8 to ASCII conversion

2009-01-24T18:04:00.002Z

The bulk loader for App Engine doesn't support unicode (?). Irksome.

Here's a quick and dirty solution if you've got iconv installed.


iconv -c -f UTF-8 -t ASCII utf8_data.csv > ascii_data.csv

Drops unacceptable unicode characters (i.e. anything that doesn't have a direct ASCII match). Did say it was dirty...

A specialist OpenID service to provide unique researcher IDs

2009-01-22T20:31:00.006Z

I was going to write a post re: this epic Friendfeed thread (and Cameron's original post) but then Geoff Bilder left some comments that pretty much covered everything:

Let me just address a few points:

a) Yes, CrossRef is exploring this space.
b) For those afraid of CrossRef being in the thrall of "traditional publishers", I will note CrossRef members include PLOS, PubMed Central, The Encyclopedia of Life, Hindawi, Jove, OECD, World Bank, some IRs... In short, we are catholic in our definition of "publisher". I should also note that we are a non-profit. When/if we charge for things, it is only so that we can sustain the service.
c) It is true that CrossRef could go under. Any place could go under. But because so many depend on us already, a central concern of our members is to make arrangements so that we can pass-on data and systems should something happen.
d) CrossRef is looking for something that will work across disciplines. We represent the sciences, social sciences, humanities, etc.
e) Cameron is right- the author ID problem is "much bigger than publishers". We are talking to researchers, librarians, funding agencies, etc. about what they would require from a service. We were at the CNI meeting and Cliff Lynch is on our advisory board and is aware of our project.
f) We too see OpenID is a critical component of the system, but we don't think OpenID and the Contributor ID are one and the same. As Richard says, OpenIDs are pretty fragile. There are also complicating issues that would arise from multiple institutional affiliations, etc. (OpenID delegation is only a geek solution to this).
g) Gumunder described our approach pretty well. We envision creating a repository of profiles. People could use open-ids (they might have a few) or shibboleth ids to authenticate with the service in order to edit their profiles. OAuth and MicroID might be used for other aspects of the service (e.g. profile exchange, blog signing)

I'm definitely up for getting off the ground quick and fast - and arguably the big disadvantage of CrossRef is that it's not always very good at that simply because it represents so many interests - but basically they're the people best placed to do this and they have the will and technical ability to see it through. Why compete when you can cooperate?

You could still start small, be unafraid to fail and try things out before any CrossRef sanctioned solution arrives though. It might be cool (and useful) to see unique author IDs across particular datasets or disciplines and if things were set up properly you could potentially just import the unique author ID / person pairs into CrossRef later to help seed the system.

Connotea Ian, today

2008-12-09T13:51:00.002Z

Look at that focus! The dedication!

New hardware is here

Code is all set up

Database files are being moved over now

Testing is imminent

don't lose hope

Strip HTML tags from a string, Ruby edition

2008-12-05T10:13:00.003Z

Get Hpricot.


require 'hpricot'
page = Hpricot("<b>some marked up <i>text</i></b>")
puts page.to_plain_text

Interestingly the Hpricot FAQ says:

Q: How do I strip all HTML tags from a page?
A: Use regex replace!
A2: The regex is ok, but will break in some cases, even with valid html. Try the to_plain_text or inner_text methods instead.

Strip HTML tags from a string, Python edition

2008-12-03T17:36:00.004Z

Obtain Beautiful Soup.


from BeautifulSoup import BeautifulSoup

''.join(BeautifulSoup(page).findAll(text=True))

where 'page' is your string of text and HTML.

I'm not a pythonista, there might be a nicer way of doing it (Beautiful Soup is a lot of overhead). Might want to expand on this a bit to make sure spacing is handled OK, you can keep certain tags etc. etc. Feel free to post corrections or better suggestions in the comments.

Just don't use one line <(?:.*?)> regular expressions. No, really.

Academia.edu

2008-10-25T23:43:00.007+01:00

I like academia.edu. The academic family tree idea is pretty cool (I know that the concept has been around for a while in various guises but their implementation is pretty slick) and I like the fact that new visitors can arrive and be interacting with the site within minutes. It's also nice to see an academic networking site that, well, doesn't look like Facebook.

I'm also impressed by the speed at which they've been throwing up refinements and bug fixes... and by the adverts on PhD. Canny marketing (good work poorly paid but well fed intern)! The academia.edu team are a smart bunch of people which is probably how they got funding in the first place.

For balance what's not good about it? The flash freezes my mac on an empty cache... and the .edu TLD is really only for educational institutions, not commercial enterprise (vetting only started in 2001, academia.edu was first registered back in '99). Tsk! Ironically my other bugbear is that I can't join properly because I work for a commercial enterprise and not an accredited educational institution.

Disappointed with Popfly

2008-05-24T22:46:00.002+01:00

Popfly is the mashup editor that Microsoft released last year. The idea is good. The 3D graphics are good. Silverlight is a bit buggy in Firefox (sidebars don't always redraw properly) but that's OK.

If you're going to create a web 2.0 mashups builder, though, don't you think it's be a good idea to provide some Atom support?

Meta-analysis

2008-05-19T17:08:00.006+01:00

The journal platform team here at NPG just rolled out machine readable metadata for the papers we publish in Dublin Core, PRISM (good PRISM, not to be confused with evil PRISM) and Google metadata formats.

No more scraping to automatically get the citation for a paper, it's all in the HEAD:


<meta name="citation_journal_title" content="Nature" />
<meta name="citation_publisher" content="Nature Publishing Group" />
<meta name="citation_authors" content="Paul Schenk, Isamu Matsuyama, Francis Nimmo" />
<meta name="citation_title" content="True polar wander on Europa from global-scale small-circle depressions" />
<meta name="citation_volume" content="453" />
<meta name="citation_issue" content="7193" />
<meta name="citation_firstpage" content="368" />
<meta name="citation_doi" content="doi:10.1038/nature06911" />

Useful for apps like Zotero and Connotea (which before now downloaded two files each time you bookmarked a Nature paper: the page itself and then the linked EndNote file to parse).

The metadata will be there for all papers going forward and back through some of the archives.

For fulltext indexing of papers behind the paywall you can use the linekd OTMI file (I only just saw Twease, which does just that) although there's only OTMI for Nature papers at the moment, I think.

Lastly at some point in the future we're aiming to put XMP metadata in our PDFs, which should make it much easier for scripts and applications (like Papers) to look at PDF files on your filesystem and work out what they represent.

Nice work Pedro!

2008-04-18T11:41:00.004+01:00

Noticed while leafing through today's Nature that Pedro has a paper out (Isalan et al., Evolvability and hierarchy in rewired bacterial gene networks).

There's more on this over at Public Rambling.

Ian owes me a pint

2008-04-17T16:34:00.001+01:00

(update: Gavin Bell at Nature gave up one of his app spots so that I could put this live, which I did: only to discover that Google App Engine is even more unforgiving of timeouts than Facebook. Currently trying to work out how to make the bookmarking process, for now it doesn't work very well. Also the search is broken, though that's Google's fault and not mine.)

I bet Ian earlier that I could rewrite Connotea on App Engine in six hours. I can't remember why. Probably ego (mine, I mean). He didn't actually bet me a pint but he should have done...

... because the original estimate was a tad optimistic (ahem). After twelve hours I've produced pycite, though, which is pretty good going I think. I'll admit it: Python is actually very cool.

pycite is three hundred lines of logic and a set of html templates that implements a (very simple) social bookmarking service. Sadly I don't actually have an App Engine account so it's not live on the web anywhere (I'll buy whoever does have an account and puts it up first a pint - let's spread the love), you'll have to download it and run it locally to see it in action.

What you can do with it:

run it without owning a server of your own

add new bookmarks (the citation will be collected automagically)

view everybody's bookmarks

filter bookmarks by user:
```
http://path.to.pycite/users/bob.smith
```

and by tag:
```
http://path.to.pycite/tags/diabetes
```

and by user and tag:

http://path.to.pycite/users/bob.smith/tags/diabetes

and by keyword (the full text of each bookmarked page is searchable):
```
http://path.to.pycite/users/bob.smith?q=t2d
```

get atom feeds for all of the above

What you can't do with it (yet):

edit or delete bookmarks

anything else

I've put it all up on Google Code. It's fairly straightforward stuff so if you've got any brilliant social bookmarking ideas then go for it. Send me an email and I'll give you write access to the subversion repository.

Gaggle

2008-04-07T14:26:00.003+01:00

I hadn't heard of Gaggle before but both Deepak and Sutee Dee (who needs a homepage.. ;)) from the ISB mentioned it last week so I figured it was worth a look. It's a system built by Paul Shannon at the ISB in Seattle to share data between different bioinformatics applications on the fly. It has been around for a while, I think - there was a BMC Bioinformatics paper describing the system in March 2006.

A small server program (the ´Gaggle Boss´) provides communication among analysis and display programs (the ´geese´) which are modest and minimal adaptations of existing (or novel) bioinformatics and computational biology programs, and web resources. The Boss and the geese all run as separate programs on the user´s desktop computer, communicating with each other, at the user´s behest, by passing simple messages.

(from the ISB's 'about Gaggle' page)

I ran through a tutorial showing data sharing between (modified versions of) Cytoscape (also developed by ISB), R and a data matrix viewer no problem. Quite cool.

You can't share data from an arbitrary application (I don't think?), they need to be modified to send messages to the Boss goose. Having said that there's a Firefox extension called Firegoose which lets you pass messages to and from web apps, Entrez etc. I couldn't get it working properly but suspect that's something to do with my install rather than the extension itself.

Anyway, it's good to see stuff like this. Truth be told it's not the slickest thing ever, but it's still pretty cool - and it works. I wonder if you could turn it into a simple lab notebook - could you write a brief description of what you're going to try and do for the Boss app every time you send data to another app or something?

Why you should try online dating

2008-04-04T00:22:00.006+01:00

(you can jump to the short answer here, if you're feeling impatient)

Onto the psychology of social media. Kristin Stecher of the University of Washington and Dave Evans of Psychster LLC both gave interesting talks about profile pages.

Psychster is a consulting company dedicated to "the social science of social networking". Recently they've been looking at interpersonal perception (how does person A perceive person B? How close is that to B's self perception?). Most research into this uses 'fake' people - i.e. A is given a detailed written description of B and works off of that, rather than meeting anybody face to face.

To try and get a large 'real people' dataset Psychster created a Facebook application (and later a website) where users could fill out a questionnaire that rated their personality on a variant of the big five personality inventory (the big five being openness, conscientiousness, extraversion, agreeableness, and neuroticism). They then had the option of rating the personalities of other people (not just their friends), the idea being to collect how users saw themselves, how others saw them and the correlation between the two.

On the standalone website users created profiles to reflect their personalities. Profiles could contain any number of elements (name, location, gender, favourite movie, most embarrassing moment...) chosen from a large list.

The results in general:

people do 'get' each other (where to 'get' a person means to guess a personality close to their actual, self-rated personality).
people on Facebook get each other better (this kind of figures - you'd want to go rate your real life friends).
women are better guessers than men - but only when guessing random strangers.
women are easier to get.

Psychster looked at different profile elements on the standalone website to see if the presence or any in particular were correlated with higher rates of accuracy.

Profile elements that make somebody easier to get:

A link to a funny video (the number one predictor of personality)
What makes me glad to be alive?
Most embarassing thing I ever did:
Proudest thing I ever did:
My spirituality:
A great person:
I believe this:

Profile elements that make you harder to get:

Profile picture (but only if it is of a non-person)
An awful website:
An awful person:
A great book:

That last one (naming a great book making it harder to guess your personality) is pretty interesting. Dave did say that he hadn't yet done any proper analysis of why it might be. I wonder if there's any research into how much (or little) reading habits have to do with your personality? Here's a tangent (why do some people get interested in science fiction?) if you're interested. Here's another (people who read lots of fiction aren't socially awkward, in fact the tendency to get absorbed in a story correlates with empathy scores).

OK, anyway...

Why were women easier to read? Because they tended to fill out the profile elements that were good predictors ("my most embarrassing moment").

At this point you might be wondering (well, I wondered) who cares how well an online profile reflects your true personality. One answer is the online dating industry who have a vested interest in not setting you up with anybody plainly unsuitable. If profiles were set up the right way then maybe you could tell in advance if the guy or girl messaging you is worth seeing in the real world.

Sticking with the online dating theme, it turns out that the levels of agreement (between actual and guessed personalities) you get by looking at Facebook profiles approach those you see in long term acquaintances. They're certainly better than what you get after a short face to face meeting (like a date). In fact, short f2f meetings are particularly bad at helping you gauge levels of agreeableness and neuroticism - not good. I think this means that stalking potential partners online actually makes good, practical sense and should be encouraged.

In case you needed any reassurance.

Do you use language differently when you're depressed?

2008-04-03T19:32:00.004+01:00

Can you tell if somebody is clinically depressed by analyzing their use of language? I'm not a psychologist, so take the background info below with a pinch of salt but the topic came up at ICWSM (more on how later) and I thought it was fascinating.

In 2001 Stirman et al compared the collected works of nine poets who eventually committed suicide and nine poets who didn't (as a control set). Their theory was that the depressed (and eventually suicidal) poets would use more first person singular (I, me, my) and words related to hopelessness and desperation (hate, worthless, death, grave) and that was supported by the data.

Rude et al later found something similar when they compared essays (on a common topic - "coming to college") written by college students. Depressed students used "I" and negative words significantly more often than controls.

Interestingly Oxman et al has found that spoken language patterns can be a good discriminator for classifying patients as depressed or not, so it's not just written language use that may be different.

Anyway, at ICWSM Nairán Ramírez-Esparza from the University of Texas presented a language analysis of some depression discussion boards on About.com. She ran a two part study: the first to confirm Stirman and Rude's findings and the second making use of the fact that the About.com boards are bilingual (there's a Spanish section too) to see how different cultures talk about depression.

Her approach was pretty simple - she collected ~ 400 posts from the depression forum and 400 posts from a breast cancer forum as a control, broke each post down into single words and then used off-the-shelf software to classify them (as verb, adjective, pronoun, positive emotion, negative emotion, etc.). She did this for both English and Spanish sections of the site.

Her results seemed to confirm the earlier studies: first person pronouns were found three times more frequently in the depression forum posts than in the controls and words relating to negative emotions occurred four times as frequently. This was true for both English and Spanish datasets.

The second part of her study was to see if English and Spanish speakers approach depression differently; what do they talk about? She studied this by using normalized word frequency counts then grouping different words into themes.

The top five themes discussed in the English dataset:


Treatment (medicine, doctor, therapist...)
Disclosure (tell, discuss, talk...)
Family (mom, dad, brother, sister...)
Symptoms ...
School

And the top five themes from the Spanish dataset:


Family
Relationship history  
Hopelessness
School
Treatment

I'm a bit suspicious of results that are so intuitively appealing (family and romance are more important to Spanish people?). One thing that I did wonder was how much the results are skewed by different community expectations: if you visit a discussion forum where people are sharing stories about their depression and everybody else mentions their family maybe you feel compelled to mention your family too. Maybe the English language forums are dominated by a younger age group and so older visitors shy away, or v.v.

Anyway, it was interesting stuff. Somebody in the audience wondered aloud if this means that you could build a system to identify people at risk of depression (or perhaps more to the point suicide) by analyzing their language online. Maybe this could be built into the next version of the anti-plagiarism software used in high schools and colleges (I'm not advocating that, just saying)...

Analyzing MySpace profiles

2008-04-02T02:26:00.011+01:00

This morning James Caverlee presented his study of almost two million (well, two sets of ~ one million - one set of profiles picked at random and one gathered by traversing the social graph) MySpace profiles. It was interesting stuff. Some bits and pieces below.

MySpace users live up to gender stereotypes, rather disappointingly:

Words most frequently appearing in MySpace profiles

Women	Men
love, people, dancing, life, shopping, can, girl, family, hearts, being, have, notebook, are, dance, favourite, things	dating, sport, networking, metal, serious, football, relationship, sht, single, wars, straight, band, video, fck, guitar, gay

And geographic ones (didn't manage to write all of these down in time):

users in Oregon	users in Alabama
camping, hiking, pixies, snowboarding, wine, vegans	football, jesus, gospel, nascar

Demographics wise ~ 50% of the profiles that they picked at random had one or no friends (i.e. weren't active). Age wise the peak is at 24, with smaller peaks at 69 and 100. The 69 peak is a secret MySpace code, apparently - it means that you're interested in, uh, one-handed typing (this wasn't made clear, but I'm guessing). By having a common age - 69 - you can use MySpace's advanced search to find others looking for the same thing. 69 year olds on MySpace are most similar (in their use of language) to people in their mid thirties.

Younger users are overwhelmingly female. There is a 2:1 ratio of girls to boys at age 14. This difference decreases as age increases. The flip over point is at 20 - after that you start seeing more men than women.

About 20% of the profiles in the connected dataset were marked as 'private'. Over time this percentage is rising. Having privacy preferences set is negatively correlated with age.

He had a fantastic slide showing top terms wrt to age... will post it and a link to the slideshow when it's online.

Tossed Salad and Scrambled Eggs

2008-04-01T06:43:00.003+01:00

I'm in Seattle for the ICWSM. The first day just finished and I'm going to blog about the more interesting talks tomorrow when I'm more awake. In the meantime:

Crowdvine for conferences is actually pretty useful
Lions for Lambs is terrible
St Trinians is actually quite good
The Century of the Self is brilliant
Seattle looks really nice from the air
Note to self: US Milky Way bars = UK Mars bars, tricksy bastards
Steak + beer + bay views = awesome (thanks Deepak!)
More Starbuckses than normal
Everybody is disconcertingly friendly. People keep offering to take me skiing. And to see waterfalls. People here big on waterfalls
MS Live Labs are hiring
Brad Fitzpatrick is a great speaker but I found his talk disappointing - too much hand waving about OpenID / OAuth / XMPP / XRDS. Dude, it's a room full of social network developers, you're preaching to the converted
Sadly Bernardo Huberman has cancelled. Marc "most unGoogleable name ever" Smith is talking instead. Marc is either founder of Poetry Slam (cool) a Happy Hardcore DJ (not cool) or a senior research sociologist at Microsoft Research (as yet undecided)

Dawkins officially bigger than Jesus - datamining Scienceblogs.com

2008-03-19T11:31:00.008Z

I've run all of the posts from Scienceblogs.com in 2007 through the ClearForest API. ClearForest extracts entities - people, places, organizations - from plain text.

I'm in the process of pulling things together for a visualization, but here's a quick answer to the 'who are Sciblings talking about?' question. The 'count' is the number of times that each entity was seen (could be multiple times in the same post) across 2007.


+-----------------------------------------------+-------+
| term                                          | count |
+-----------------------------------------------+-------+
| Michael Egnor                                 |  1855 | 
| Richard Dawkins                               |  1737 | 
| Bush                                          |  1669 | 
| Congress                                      |  1430 | 
| Charles Darwin                                |  1226 | 
| Michael Behe                                  |  1031 | 
| Chris Mooney                                  |   927 | 
| FDA                                           |   920 | 
| DCA                                           |   765 | 
| National Aeronautics and Space Administration |   745 | 
| National Institute of Health                  |   741 | 
| Bush administration                           |   721 | 
| Google                                        |   700 | 
| Guillermo Gonzalez                            |   691 | 
| White House                                   |   658 | 
| Supreme Court                                 |   655 | 
| Thomas Jefferson                              |   632 | 
| John Edwards                                  |   614 | 
| Casey Luskin                                  |   605 | 
| George W. Bush                                |   603 | 
| Jesus Christ                                  |   601 | 
| Discovery Institute                           |   596 | 
| the New York Times                            |   587 | 
| Larry Moran                                   |   576 | 
| World Health Organization                     |   543 | 
| Hillary Clinton                               |   517 | 
+-----------------------------------------------+-------+

Bear in mind that ClearForest extracts entities, not key terms. It can't tell us how often blog posts are talking about mammoth DNA, supernovae or dicyemid mesozoa. That's a different dataset entirely...

.... this one, in fact, generated using the Yahoo! term extraction API which pulls out important concepts (terms) from text. The dataset is about half the size of the above as I'm only including ScienceBlogs indexed in Postgenomic. Here 'count' is the number of distinct posts containing a term:


+---------------------+-------+
| term                | count |
+---------------------+-------+
| evolution           |   963 | 
| carnival            |   923 | 
| global warming      |   640 | 
| intelligent design  |   543 | 
| new york times      |   542 | 
| blogosphere         |   468 | 
| religion            |   460 | 
| brain               |   437 | 
| climate change      |   432 | 
| creationist         |   420 | 
| birds               |   415 | 
| creationism         |   409 | 
| creationists        |   398 | 
| pz                  |   378 | 
| darwin              |   367 | 
| discovery institute |   354 | 
| atheists            |   351 | 
| atheist             |   333 | 
| biology             |   314 | 
| richard dawkins     |   301 | 
| skeptics            |   290 | 
| love                |   289 | 
| genes               |   288 | 
| job                 |   286 | 
| money               |   283 | 
| orac                |   281 | 
| god                 |   276 | 
| atheism             |   266 | 
| animals             |   261 | 
| bush                |   258 | 
| google              |   258 | 
+---------------------+-------+

In light of this data it's tempting to revisit that Bayblab post suggesting that Sciblings spend too much time discussing ID. That'd be a mistake, though: the numbers above are absolutes. 963 posts had 'evolution' as a key term but that's only 2.4% of all posts that year (my 2c: I think that Sciblings do talk about Egnor, ID and creationism too much, but hey, it's their blogs - I just skip over those posts).

I also had a look at linking patterns - who do ScienceBloggers link to the most? Here 'count' is the number of unique posts that have a link to a particular domain.


+-------------------------+-------+
| domain                  | count |
+-------------------------+-------+
| www.scienceblogs.com    | 15966 | 
| en.wikipedia.org        |  2016 | 
| www.technorati.com      |  1797 | 
| www.nytimes.com         |  1388 | 
| www.amazon.com          |  1078 | 
| www.sciencedaily.com    |   661 | 
| www.washingtonpost.com  |   478 | 
| feeds.feedburner.com    |   467 | 
| www.nature.com          |   453 | 
| news.yahoo.com          |   401 | 
| news.bbc.co.uk          |   333 | 
| www.youtube.com         |   305 | 
| www.del.icio.us         |   297 | 
| www.cnn.com             |   260 | 
| www.eurekalert.org      |   260 | 
| farm3.static.flickr.com |   259 | 
| www.sciencemag.org      |   231 | 
| www.ncbi.nlm.nih.gov    |   225 | 
| www.pandasthumb.org     |   224 | 
| www.google.com          |   219 | 
| www.latimes.com         |   213 | 
| www.gnxp.com            |   208 | 
| sandwalk.blogspot.com   |   197 | 
| www.dailykos.com        |   196 | 
| www.donorschoose.org    |   194 | 
+-------------------------+-------+

Presumably the technorati links are from tags. Sciencebloggers link to scienceblogs.com far more than anywhere else - but I'd guess that this is simply because there are a lot of good science blogs on one domain there.

Wikipedia's reliability might be in question but it's interesting that almost everybody uses it to define terms.

Drilling down, where do ScienceBloggers link to papers?


+--------------------------------+-------+
| domain                         | count |
+--------------------------------+-------+
| www.nature.com                 |   241 | 
| www.sciencemag.org             |   194 | 
| www.dx.doi.org                 |   177 | 
| www.ncbi.nlm.nih.gov           |   111 | 
| www.pnas.org                   |   104 | 
| www.plosone.org                |    89 | 
| biology.plosjournals.org       |    76 | 
| content.nejm.org               |    67 | 
| medicine.plosjournals.org      |    65 | 
| www.sciencedirect.com          |    43 | 
| www.arxiv.org                  |    33 | 
| genetics.plosjournals.org      |    22 | 
| www.jneurosci.org              |    15 | 
| www.cell.com                   |    14 | 
| compbiol.plosjournals.org      |    10 | 
| pediatrics.aappublications.org |    10 | 
| www.jcb.org                    |    10 | 
| mbe.oxfordjournals.org         |     9 | 
| www.ajp.psychiatryonline.org   |     8 | 
| www.current-biology.com        |     8 | 
| www.journals.uchicago.edu      |     8 | 
| www.plosntds.org               |     8 | 
| www.blackwell-synergy.com      |     7 | 
+--------------------------------+-------+

Nature and Science are at the top, perhaps unsurprisingly - but if you add up the counts from the different PLoS journals it'd be up there too.

Science streaming

2008-03-19T00:39:00.003Z

Michael Barton has a nice post up:

I currently use Subversion to back up my project files, and I noticed Twitter status updates are very similar in length to subversion log messages. I created a short script so that every time I do a subversion repository check in, the message is also sent to Twitter.

I'd like to see activity aggregators accept arbitrary updates - sort of like Facebook's Beacon updating people's News Feed, but done properly.

Nature archive visualized - draft

2008-03-18T03:08:00.006Z

I'm using up my annual carry-over vacation days by taking some time off work this week. Normal people probably use this valuable breathing space to bond with their loved ones, play badminton and learn exciting new hobbies. So far I've sat alone in my flat for thirty six hours straight writing Processing sketches *.

So... here's a draft visualization (14mb MP4, should play in your browser with Quicktime) of the key words and phrases found in Nature journal over the past thirty years.

The video starts with the phrases from 1970 and continues until 2007.

Phrases appear on the right in the year that they were first seen, then travel leftwards, disappearing in the year they were last seen.

The size of each phrase is related to how often it was seen relative to all the other phrases.

The hue of each phrase is related to how many distinct journal issues it appeared in - green / yellow phrases are relatively transient while red / brown phrases are stable, appearing in many different contexts.

The data is incomplete (it's a bit sparse after '88) and I took lots of shortcuts to see how things might look, so don't read too much into which phrases appear and when for now... a better version will follow - this is just a release early, release often draft.

Eventually I'd like to have a sort of Pop-up Video timeline of science from the 50s till today, with major events (and relevant terms) flashing up on screen.

If you're particularly impatient here's a version from Vimeo. The quality is rubbish, mainly because I munged the file with iMovie (which is crap) to add some rockin' beats. I still suggest you get the mp4 instead, though.

Tommorrow I'm going to the park.

* I watched Rendition, too, it was quite good.

Seattle

2008-03-11T10:44:00.005Z

I'm going to be in Seattle the first week of April for the ICWSM.

There are a whole bunch of awesome looking talks and one or two wild cards. Wild cards like:

Spontaneous Inference of Personality Traits from Online Profiles
Kristin Stecher, Scott Counts

Which sounds interesting, anyway.

Let me know if you're in the area and fancy meeting up for lunch or a drink. I'm in town from the 29th of March to the 5th April.

New JoVE blog & commenting on papers

2008-03-10T23:11:00.003Z

Anna Kushnir's new blog for JoVE is up and running (actually it has been up and running for a while, I'm a bit behind with blogging. Those January Open Science posts are coming at some point, too). It's a nice mix of content.

Of particular interest are a couple of interesting entries talking about the online participation - or lack thereof - of scientists. See also Noah Gray's take on neuroscientists and web 2.0 and David Crotty's 'why web 2.0 is failing in biology' post.

Did you skip over all those links? You shouldn't, really. At least read David Crotty's.

So, yeah, anyway, why scientists don't comment on papers - my take is that being too busy and being afraid of the consequences don't come into it.

Sure, they're valid concerns - but everybody is busy at work and everybody realizes that what you say on the internet is recorded forever by Googlebot. People still write ranty forum posts and blog comments.

IMHO the main reasons scientists don't leave comments are:

There's no point - who's going to read it? Will you get any feedback? Will you get any credit for it?

and

It's too much work - writing a comment should be a one click operation. Well, two clicks, one to get the focus in the textbox and the other to press 'submit'.

Science publishers can address both of these issues, but we've been failing to do so.