So, I really need to start getting some of my thesis ideas down on paper. I have been talking about them and researching them and even writing my literature review, but I have yet to be able to articulate in writing what I am interested in. This blog entry is an attempt to allow those creative juices to flow without hindrance from the rules and references. I probably won't cite anyone formally, but if I use someone else's idea, I'll try to put in a link to the paper or at least say "Some guy said...". Hopefully everyone will take this as some rough sketching and not a formal paper plagiarizing those sources.
Chapter 1 of my thesis is what I'm attempting to hack out today. Chapter 1 is called the introduction, and in the words of my adviser is supposed to tell the audience "what the problem is, and why it's important". I'll conclude it with my research questions (which are still pretty fuzzy in my head, but will hopefully be clearer after this writing). Unfortunately, a lot of the history of the development of this topic in my mind is going to come out here today. Chapter 1 would generally not include this information, but I think that putting it here will be instrumental in helping me discover how best to support my topic.
Ground Rules:
1) Blog style - first person is OK
2) Grammar/punctuation - use properly, but don't stress about them
3) Citations - they would be good...
4) Comments - definitely appreciated
Alright. Let's begin. (side note: I just found out about some rocking job opportunities locally... I'm trying not to be too distracted! I guess I know what I'm doing tomorrow.)
The US Army Aeromedical Research Laboratory (USAARL) has acquired an extremely large collection of irreproducible data. (Dear Firefox, Thank you for checking my spelling when I type in a text box on a website.) This data must be curated and organized for ongoing use. During the present attempt to wrangle this data, we have been working to develop a data repository or digital library. The use of the phrase 'data repository' is a misnomer as is the phrase 'digital library'. We are attempting to develop something somewhere in between. We plan to have more relationships between items in the collection than a digital library and we intend to have many more kinds of information than just the raw data. The word here is 'reproducible'. Though the data can never be reproduced, we are trying to build a data warehouse that is essentially a re-creation of the experiment so that researchers may use the data within its original context. For example, data collected may be influenced by the sensors used to collect it. We intend to provide our users with calibration information, model numbers and manuals for the sensors used to collect the data.
During the process of designing and building this data warehouse, we have chosen to store most of our information as files in a method similar to that used by digital libraries. This has led to the creation of a metadata schema so that we can label and categorize the files and their relationship to each other and their place within the grand experiment. During the metadata schema development we had many things to think about. We had to consider the value of each element and the time cost of applying each element to each file, among other considerations. One of the things that came up repeatedly was justification that applying a particular element would allow users to filter out what they were not interested in as well as to identify something that they were interested in. That is, if a user had a collection of items from the database he could say "I want ones that have label X" or he could say "I don't want ones that have label Y". I will call these ideas 'inclusion criteria X' and 'exclusion criteria Y'.
I found this discussion to be rather interesting because use or availability of use of inclusion criteria is extremely common in information retrieval, exclusion criteria is not. One might argue that they are the same. That is, exclusion criteria Y is equivalent to inclusion criteria NOT Y. Where Y and NOT Y are inverse collections. Interesting argument. Consider Google. Let's call each word in a search A1, A2,... AN where N is the number of terms you have (can you tell I am a mathematician?). Google interprets your search terms by looking for pages containing every single one. Okay. That's not totally true, but close enough for this discussion. What happens once you have the results? You look at the first page, or maybe the first three, but never really any farther (per some study--see, I told you I wasn't going to look them up right now). Then you likely (per some study) either have found the page you were looking for and clicked on it or you give up or you try again by adding another search term (A(N+1)). Bam! Inclusion criteria. "I only wish to keep the results that have term A(N+1)." Some people (< 10% of the population per another study, and only about 5% of the time per Google) would choose to edit their search request differently (more like exclusion criteria). These people recognize that the results include a specific set of pages that are about something totally different from what they are looking for. If the original request is for "dolphins", but many results are about the NFL team, and the user actually wants results about the mammal, he might choose to modify his query by adding "-NFL" (NOT NFL). Exclusion criteria! "I wish to remove results that meet certain criteria." Well, pretty much no one does this (per a study) or at least they didn't in 2006 when the study was done. The thing is, no one really does this in the general population. I am very interested in why that is the case. More specifics to come.
Consider Amazon with me. I know Google and Amazon may not necessarily be the best services to use as a demonstration, but you can't say they aren't culturally pervasive. Okay. When you shop on Amazon, let's say that you're looking for a bicycle for your kid. If you type in bicycle in the search box, you're going to get over 50,000 results. That is a few too many too look at. There are a couple things you can do to narrow your search results, but one of them is to choose the 'Department' you'd like to look in. That's cool, but the first two are 'Sports & Outdoors' and 'Toys & Games'. Which one are you going to choose? Eeeks. Searching for the perfect bike just got harder. Also in the list of departments are things like 'Clothing & Accessories' and 'MP3 Downloads'. It would be really great just to drop the results from those departments from the list. The ability to do this does not exist in Amazon's web interface. Darn. See what I mean? Exclusion criteria is not an option here. On a side note, neither is multiple inclusion, which would function similarly.
Let's return briefly to the argument that inclusion and exclusion criteria are the same. First, it would be appropriate to define two measures of "good" search results. Precision is the percentage of retrieved documents that are relevant to the search. Recall is the percentage of all relevant documents that were actually retrieved by the search. Don't forget these definitions; we'll probably use them again. Okay, back to the idea that multiple inclusion is the same as exclusion. In some cases this might be true. You might could add appropriate words to your "dolphin" search to get what you are looking for. What about 'mammal dolphin, ocean'? Well, that might not work, but I'm guessing that pdf of scientific data tracking dolphin migration is not going to use all of those words. Now you've missed out. You've likely decreased your recall. On the other hand, adding '-NFL' to your search is will likely (though not with 100% certainty) maintain the recall of your results and increase the precision. Precision increase is a really good thing. Remember back at the beginning when we discussed that you are statistically likely to look at three or less pages of results? That makes precision a significantly important metric regarding your ability to find what you are looking for. Let's look at a counter example before you start thinking that everything is butterflies and roses with exclusion criteria and that it is different and better than multiple inclusion. Sometimes it just doesn't work. I am proposing, however, that there are a good many cases where it adds value (I wish someone was doing a study on this so I could cite them!). What if I go looking to see if anyone else has done any research on 'exclusion criteria'? Most of the results returned at scholar.google.com are about excluding volunteers from human research based on certain criteria. Ugh. Now what? What words could I use as exclusion criteria? 'Subjects'? 'Volunteers'? 'Study'? Yikes. I can't think of one. This is probably a case where adding additional words like 'Information Retrieval' would probably help more than excluding a particular term.
Are you convinced, though, that exclusion criteria is a good, quick solution in some cases? Hopefully. At least some people are convinced enough to actually use it, so I guess it's ok to study it.
So, here's the thing. We have this database and we think that researchers are going to 1) know how to use exclusion criteria 2) recognize its value in general 3) recognize its value in a specific situation 4) remember to actually use it based on the interface or internal reasons when those situations occur. Though I'm not going to study exactly this, I'm anxious to study something similar.
I want to know:
1) Do researchers know how exclusion criteria works in Google searches?
2) Do researchers conceptually understand exclusion criteria in the context of Google searches? A related question is whether they understand the basic mechanisms of Google searches?
3) Do researchers recognize appropriate instances where exclusion criteria would add value to a search?
4) Do researchers feel like exclusion criteria is valuable in reducing result sets (in general)?
5) Do researchers actually use exclusion criteria as part of their Google searches? Both original searches and/or reformulated queries?
All of this could be summed up in the broad research question:
"What are the attitudes toward and knowledge of exclusion criteria when used in Google searches among researchers?"
One of my committee members suggested a brilliant article called "
That's Interesting". It's about what makes a sociological study interesting. The bottom line, is that you have to give your audience something they didn't expect. Well, right now, the information retrieval community doesn't think that people use exclusion criteria (reference that study). There are some authors that caveat their studies with the idea that some people with more complex information needs might use them more frequently. Some studies even espouse the view that exclusion criteria isn't valuable. I hope to show the community that researchers are one such set of people who actually use exclusion criteria, but I know that not all researchers use it. For them, I would like to know why. Researchers do have a complex information need and if they are not using something as simple and potentially helpful as exclusion criteria, we should know why! Perhaps we will be surprised to find out why. I guess I haven't yet presented a clear view of why this topic is important or of interest. I should note that all web search engines implement capability for exclusion criteria. We should definitely understand one of the prime user groups' perspectives on it! Maybe after I get the results I can tell you more of the exciting and interesting things that I find. I can definitely tell you that I haven't seen any studies get into researchers' brains concerning this topic. That is what I hope to do. The other thought I've had about why this study is important relates to the crisis that would happen if researcher's can't access previous generations' work. There is this sweet book called Lost Knowledge which is about how to combat the loss of knowledge due to organization turnover. This is a particularly interesting topic in light of the pending mass exodus of the baby boomers from the work force. Information retrieval has a similar problem called "Too Stinking Much Information" (very scientific name) as a result of the increase in technology over the last 50 years (probably should cite a bunch of convincing statistics here). On an individual basis the information retrieval community needs to understand what is going on between the ears of people attempting to use our information systems! If you're interested in some side reading about the necessity of this, check out Dr. Vannevar Bush's
Memex.
Okay. The sun is going down. I'm going to stop babbling. Commence discussion.
Posts
