Trusting Differently: Diversity-aware search for people, content, events


Copy/paste this URL to link to this paper

Our goal here is to develop socially robust, usable as well as epistemologically and ethically sounds tools to support researchers in finding content, people and events that are new, but highly relevant for their research.

Keywords : diversity, content, search

Abstract : Abstract

Our goal here is to develop socially robust, usable as well as epistemologically and ethically sounds tools to support researchers in finding content, people and events that are new, but highly relevant for their research.

Central Claim : Trusting Differently: Diversity-aware search for people, content, events

'See only what you want to see, hear only what you want to hear, read only what you want to read' (Sunstein 2002: 238). Cass Sunstein has warned us that such a personalized usage of the Web, the 'Daily-Me' of accessing and filtering information according to our interests may be harmful for the democracy and freedom of speech (Sunstein 2002). Instead of filtering information in a way that only confirms our views and targets our attention to supportive additional information, a democratically sound way of informing oneself should ideally balance contradicting information, should be able to assess the pros and cons of different viewpoints. Hence, in recent years there have been attempts to develop recommender systems, which are not only or primarily based on similarity of information, but on diversity of information [Agraval2009, Hadjieleftheriou2009, Minack2009].

In science or research more generally, the tendency to suppress or ignore contradicting information may appear less severe, especially if falsification and critical assessment of others' claims are at the core of research activities. Nonetheless, the problem of being informed only about similar content seems quite prevalent in many disciplines. This problem seems to be caused by implicit trust and reputation relations between a scientist or a research group and other scientists or publication venues. More specifically, researchers know only a very specific group of people, attend only very specific events and ignore content, people and events that may be interesting, but are associated with or rooted in different communities. For instance, there are researchers working on trust in many different disciplines, in computer science and AI, in philosophy and sociology, in economics and literature studies. Maybe many of the research topics are unrelated, yet acknowledging how often ground-breaking research was inspired by thinking outside of the box, it may be well worth trying to identify research that is different, but interesting and relevant. This is the aim of this research project: to develop means to identify content, people and events in research that are interesting and relevant, but which provide different perspectives instead of being rooted in similarity.

In short, the project is split into two distinct, but related parts. On the one hand, we want to develop a platform, which helps researchers to

  • Find people, content and events that are relevant to them, but based on diversity increasing metrics
  • Find interesting papers for review and find reviewers which will provide different viewpoints on the same paper
  • Assign paper to panels/sessions with different degrees of similarity or diversity
We will develop this platform with researchers from different disciplines in mind from the very start. Our three paradigmatic communities of research, which we will use as test cases are computer scientists, philosophers and social scientists. These communities were chosen due to the background of the researchers, which should enable easier access to these communities. On the long run, our platform needs of course to be cross-validated with respect to their utility for researchers from different communities. Usability testing and formative evaluation will accompany the design of this platform. To fill our tool with content and make it attractive to use, we plan to involve publishers (Elsevier), public protocols used by pre-print repositories such as ArXiv.org, as well as other research-targeted Web projects (philpapers.org, Mendeley.com, Academia.edu, ResearchGATE) which provide access to their data.

Beyond this practical goal of developing a usable and used platform, our main research effort, will be focused on the development of diversity-based recommender algorithms for people, content, events and their utilization for research. While some research exists on recommender systems for Web-content such as web pages [Hadjieleftheriou2009], there is yet no approach that makes use of these insights for research content (papers, blogs, datasets, experimental procedures). 

There are three dimensions we consider for people, content, events: relevance, reputation, diversity. As the first step, we will fix the relevance and reputation metrics, using one of existing metrics and develop metrics for diversity of content. For instance, to estimate relevance of a person, content, even, metrics from information retrieval can be used, while for calculating reputation we can use the metrics based on links or citations. Therefore, our research foci will consist in:
  • Developing diversity-based metrics for recommending content, events, people
  • Cross-checking the viability and utility of these metrics for different research communities (computer science versus philosophy versus social sciences)
  • Integration of the developed diversity-based metrics into a recommender system that also has metrics of relevance and reputation
  • Evaluation of the added value of the diversity metrics
  • Applying the recommendation system in for matching papers with reviewers, i.e., finding reviewers with a different viewpoints on a paper
Our goal here is to develop socially robust, usable as well as epistemologically and ethically sounds tools to support researchers in finding content, people and events that are new, but highly relevant for their research.
Attribution Share Alike

Bibliography

Agrawal, R., Gollapudi, S., Halverson, A., and Leong, S., 2009: "Diversifying search results". Proceedings of the Second ACM international Conference on Web Search and Data Mining, R. Baeza-Yates, P. Boldi, B. Ribeiro-Neto, and B. B. Cambazoglu, , pp. 5-14.

Cass R. Sunstein, 2002: Republic.com. Princeton: Princeton University Press

E Minack, G Demartini, W Nejdl, 2009: "Current Approaches to Search Result Diversification". Proc. of 1st Intl. Workshop on Living Web,

Marios Hadjieleftheriou and Vassilis J. Tsotras., 2009: "Special Issue on Result Diversity". Bulletin of the Technical Committee on Data Engineering., 32, 4

Mike Masnick, 2010: Rethinking Peer Review As The World Peer Reviews Claimed Proof That P≠NP : URL: http://www.techdirt.com/articles/20100812/03472410604.shtml

Start a new discussion thread (are you logged in ?)

Write a review (are you logged in ?)

Moderator(s): Gloria Origgi , Judith Simon , Jordi Sabater-Mir 
  • Scope might need trimming (1 contribution)
    Marlon Dumas, Dec 3 2010 08:49 UTC
    There are definitely some interesting ideas here.
    Particularly, the idea of using diversity in the context of a recommender system for scientific articles is very promising. This, to me, seems the core idea of your vision,
    However, I feel that your vision is overshadowed by the fact that you mix this core idea with several other ideas including:
    - Saving, organizing and annotating research content
    - Write, co-produce and share [...] content
    - Estimate reputation of an event...

    I can agree with the fact that "diversity" alone is not a sufficient criterion for recommending a paper or an event to someone. The "relevance" also needs to be taken into account, as well as the "reputation" and possibly other related criteria. However, there are already many metrics of "relevance" (e.g. based on topic analysis) and there is perhaps little you could add in this area. The same applies to "reputation". So maybe you should focus your efforts, particularly your "viability" and "utility/usefulness" evaluations, on the diversity aspect. In other words, take an existing measure of relevance (e.g. from information retrieval), take an existing "reputation" metrics (e.g. based on links or citations), and add the diversity metrics on top of these. This would allow you to assess how much added-value your diversity measures bring on top of what the relevance and reputation metrics give you.

    Also, it seems to me that you are mixing the concepts of "interesting" and "relevance". I'm not a specialist, but these sound like very different concepts and it would make sense to focus on one of the them rather than both. My suggestion would be to focus on "relevance" rather than "interestingness" since "relevance" is perhaps less subjective.

    When it comes to the "foci", I think there are too many foci in your vision. Maybe you should consider dropping the following ones and concentrating on the ones related to diversity:
    - Developing trust and reputation metrics for content, events, people
    - Analyzing hidden research networks
    - Analysis and avoidance of conflicts of interest...

    Finally, I suggest that you frame your work in the context of "Recommender Systems". Basically, what you want to achieve is to develop a recommender system that takes into account diversity between the sources (e.g. authors) of the documents being recommended. This Recommender System of research articles has two targeted applications (among others):
    1) Recommending articles/content, events and people to a given scientist
    2) Matching papers with reviewers

    The second of these applications is very promising. I don't think that conferences are currently using anything close to a diversity measures when matching papers to reviewers. Yet, one might not want to have a paper being reviewed by multiple reviewers who on the end bring the same viewpoint into the paper.

    So you could structure your research as follows:
    1) First you motivate and propose one or several diversity measures for research articles/authors/events, etc.
    2) Then you show how these metrics can be integrated into a recommender system that also integrates metrics of relevance and reputation
    3) Next you evaluate how much added-value your diversity metrics bring on top of the metrics of relevance and reputation
    4) Finally, you show how the recommender system can be used in the context of recommendation of scientific articles, but also matching of papers to reviewers.

    marlon
  • Link to the paper on usage of citation data on assessment of academic performance (no contribution)
    Peep Kungas, Dec 14 2010 16:11 UTC
    During the workshop after your presentation Dr. Ethan V. Munson suggested the following article: http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf.

    From the executive summary: This is a report about the use and misuse of citation data in the assessment of scientific research. The idea that research assessment must be done using "simple and objective" methods is increasingly prevalent today. The "simple and objective" methods are broadly interpreted as bibliometrics, that is, citation data and the statistics derived from them. There is a belief that citation statistics are inherently more accurate because they substitute simple numbers for complex judgments, and hence overcome the possible subjectivity of peer review. But this belief is unfounded.