Using Amazon Mechanical Turk to Extend Wikipedia Base

In my previous post, I described how we use Wikipedia to extract main topics from a Web page filtering out noise content.

The beautiful thing about Wikipedia is its very good coverage of many different domains. But still sometimes I get questions, are you sure Wikipedia is enough to understand any Web page in WWW? Ok, of course, Wikipedia has its limits. Sometimes, if you need to dig deeper into a domain, you might need more concepts.

To solve this problem we need to extend our Wikipedia-based knowledge base with some other, possibly domain-specific dictionary/encyclopedia. Take for example TechCruch' CrunchBase - a base of startups and investors descriptions, the good part of CrunchBase entries is not present in Wikipedia.

Merging Wikipedia and CrunchBase automatically is a bad idea, since it would spoil the Wikipedia benefits of being a clean human produced resource. So, me and my student Artem Chuzhmarov had a nice idea to process this task half-automatically using Amazon Mechanical Turk.

How exactly we need to merge the two encyclopedias. What we use from Wikipedia is its article titles and hyperlinks between articles. So to merge CrunchBase into Wikipedia we need to add CrunchBase entries and to connect them with Wikipedia concepts via hyperlinks.

The Turk gets CrunchBase entry description (title of startup or name of investor and text describing it) and fulfils the following task:

1) Check is there corresponding article in Wikipedia, if no

2) Read the entry text, identify key terms in it and find corresponding Wikipedia articles for these key terms.

The result of Turk work for a single CrunchBase entry is a list of key terms and their corresponding Wikipedia articles. So now we can automatically add the entry to our knowledge base.

CrunchBase entries become one-way connected with Wikipedia articles: they don't become referenced from Wikipedia articles. But that seems to be enough to compute semantic relatedness between two CrunchBase entries or CrunchBase entry and Wikipedia article.

When experimenting we paid ~15 cents for one CrunchBase entry. So, to include all CrunchBase into Wikipedia we would pay about 3000$ and these are once-only expenses.

Comments

[Elliot Turner]

An interesting approach! Amazon Turk seems to be increasingly leveraged as a source of machine learning training data, linkages, etc.

Another worthwhile variant of this technique would be to combine Turk-based human annotators with an automated suggestion tool, along the lines of ODDLinker (Used to interlink the LinkedMDB project with dbpedia).

Tools such as ODDLinker that leverage tuple data to generate potential linkages can alleviate much of the human legwork for "obvious linkages", leaving manual disambiguation/lookups for the more ambiguous entries. Combining these sort of tools with workflow systems such as Amazon Turk has the potential to bring the Crunchbase annotation/linkiong costs down significantly.

[Maria Grineva]

Thanks Elliot! It is a reasonable suggestion to make some more obvious things automatically in order to bring costs down.

Can you give me links to ODDLinker tool?

[Enrique Gomez]

I wonder if your idea of using human annotation could be used against the twitter data through twitter itself. Consider that a human could add content to a twitter post, especially when referring to a linked article, through the use of a new kind of twitter tag. The content added would raise or lower the relevance according to the user. You could then 'sift' through the twitter data and extract a metric of relevance based on the new tag. The Turk humans are rated for effectiveness and therefore more reliable but in the twitter scenario you would leverage the larger mass of annotators. Hopefully some good data would bubble out of the noise.

I'm still thinking on how the tags would look like and a simple syntax. Something like the RT tag, say ReleVance. One could RV + or RV - if you find the whole post relevant or not. Maybe RV +term if you want to add a term you think applies to the post, RV -term to remove, etc.

Maybe I just need more coffee...

[Maria Grineva]

I didn't understand what you mean "relevant"? Do you mean, if I post something to Twitter I would rate how is the posted link relevant to me?

[Volker from Germany]

Hi,
I randomly came on this side and would like to leave nice greetings. I would be glad if you visit my homepage also! Maybe you want to visit Sylt Westerland in Germany http://www.MeerblickSylt.de or the Baltic Sea http://www.OstseeblickHolm.de for vacation?! We have there very nice flats with a nice view. Maybe we'll see soon!
Kind regards

[volvo seat covers]

There's some padding between the front and back layers of the cover, and that held moisture.

[Bookkeeper Caloundra]

Wikipedia is one of the most important wbsites.Here, you will find wealth of information.Wikipedia contributed so much to the information and knwledge based world.