Online evaluation for information retrieval:
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Book |
| Language: | English |
| Published: |
Boston
Now
[2016]
|
| Series: | Foundations and trends in information retrieval
volume 10, issue 1 |
| Subjects: | |
| Online Access: | Klappentext Inhaltsverzeichnis |
| Physical Description: | x, 123 Seiten |
| ISBN: | 9781680831634 |
Staff View
MARC
| LEADER | 00000nam a2200000 cb4500 | ||
|---|---|---|---|
| 001 | BV043838272 | ||
| 003 | DE-604 | ||
| 005 | 20161221 | ||
| 007 | t | ||
| 008 | 161021s2016 |||| 00||| eng d | ||
| 020 | |a 9781680831634 |9 978-1-68083-163-4 | ||
| 035 | |a (OCoLC)968135764 | ||
| 035 | |a (DE-599)GBV870101412 | ||
| 040 | |a DE-604 |b ger |e rda | ||
| 041 | 0 | |a eng | |
| 049 | |a DE-355 | ||
| 084 | |a ST 270 |0 (DE-625)143638: |2 rvk | ||
| 100 | 1 | |a Hofmann, Katja |e Verfasser |4 aut | |
| 245 | 1 | 0 | |a Online evaluation for information retrieval |c Katja Hofmann, Lihong Li, Filip Radlinski |
| 264 | 1 | |a Boston |b Now |c [2016] | |
| 264 | 4 | |c © 2016 | |
| 300 | |a x, 123 Seiten | ||
| 336 | |b txt |2 rdacontent | ||
| 337 | |b n |2 rdamedia | ||
| 338 | |b nc |2 rdacarrier | ||
| 490 | 1 | |a Foundations and trends in information retrieval |v volume 10, issue 1 | |
| 650 | 0 | 7 | |a Benutzerverhalten |0 (DE-588)4122898-4 |2 gnd |9 rswk-swf |
| 650 | 0 | 7 | |a Information Retrieval |0 (DE-588)4072803-1 |2 gnd |9 rswk-swf |
| 650 | 0 | 7 | |a Erfolgskontrolle |0 (DE-588)4015228-5 |2 gnd |9 rswk-swf |
| 650 | 0 | 7 | |a Internet |0 (DE-588)4308416-3 |2 gnd |9 rswk-swf |
| 689 | 0 | 0 | |a Information Retrieval |0 (DE-588)4072803-1 |D s |
| 689 | 0 | 1 | |a Erfolgskontrolle |0 (DE-588)4015228-5 |D s |
| 689 | 0 | 2 | |a Internet |0 (DE-588)4308416-3 |D s |
| 689 | 0 | 3 | |a Benutzerverhalten |0 (DE-588)4122898-4 |D s |
| 689 | 0 | |5 DE-604 | |
| 700 | 1 | |a Li, Lihong |e Verfasser |4 aut | |
| 700 | 1 | |a Radlinski, Filip |e Verfasser |4 aut | |
| 830 | 0 | |a Foundations and trends in information retrieval |v volume 10, issue 1 |w (DE-604)BV035495746 |9 10,1 | |
| 856 | 4 | 2 | |m Digitalisierung UB Regensburg - ADAM Catalogue Enrichment |q application/pdf |u http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=029248866&sequence=000002&line_number=0001&func_code=DB_RECORDS&service_type=MEDIA |3 Klappentext |
| 856 | 4 | 2 | |m Digitalisierung UB Regensburg - ADAM Catalogue Enrichment |q application/pdf |u http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=029248866&sequence=000004&line_number=0002&func_code=DB_RECORDS&service_type=MEDIA |3 Inhaltsverzeichnis |
| 999 | |a oai:aleph.bib-bvb.de:BVB01-029248866 | ||
Record in the Search Index
| _version_ | 1804176705913356288 |
|---|---|
| adam_text | Katja Hofmann, Lihong Li and Filip Radfinski
Online evaluation is one of the most common approaches to measure the effectiveness
of an information retrieval system, ft involves fielding the information retrieval system to
real users, and observing these users’ interactions in situ while they engage with the
system. This allows actual users with real world information needs to play an important
part in assessing retrieval quality.
Online Evaluation for information Retrieval provides the reader with a comprehensive
overview of the topic, it shows how online evaluation is used for controlled experi-
ments, segmenting them into experiment designs that allow absolute or relative quality
assessments. The presentation of different metrics further partitions online evaluation
based on different sized experimental units commonly of interest: documents, lists, and
sessions. It also includes an extensive discussion of recent work on data re-use, and
experiment estimation based on historical data.
Online Evaluation for Information Retrieval pays particular attention to practical issues:
How to run evaluations in practice, how to select experimental parameters, how to take
into account ethical considerations inherent in online evaluations, and limitations that
experimenters should be aware ok While most published work on online experimenta-
tion today is on a large scale in systems with millions of users, this monograph also
emphasizes that the same techniques can be applied on a small scale. To this end,
jf highlights recent work that makes it easier to use at smaller scales and encourages
studying real-world information seeking in a wide range of scenarios. The monograph
concludes with a summary of the most recent work in the area, and outlines some open
problems, as well as postulating future directions.
Contents
1 Introduction 3
1.1 Terminology............................................... 4
1.2 Motivation and Uses....................................... 5
1.3 This Survey............................................... 6
1.4 Organization.............................................. 7
2 Controlled Experiments 9
2.1 Online Controlled Experiments in Information Retrieval . . 9
2.2 Planning Controlled Experiments.......................... 12
2.3 Data Analysis............................................ 18
2.4 Between-subject Experiments.............................. 22
2.5 Extensions to AB testing................................. 24
2.6 Within-subject Experiments............................... 28
2.7 Extensions to Interleaving............................... 31
3 Metrics for Online Evaluation 33
3.1 Introduction............................................. 33
3.2 Absolute Document-level Metrics.......................... 35
3.3 Relative Document-level Metrics.......................... 38
3.4 Absolute Ranking-level Metrics........................... 39
3.5 Relative Ranking-level Metrics .......................... 41
IX
X
3.6 Absolute Session-level and Longer-term Metrics.............. 46
3.7 Relative Session-level Metrics.............................. 60
3.8 Beyond Search on the Web.................................... 50
3.9 Practical Issues ........................................... 50
4 Estimation from Historical Data 53
4.1 Motivation and Challenges............................... 53
4.2 Problem Setup........................................... 56
4.3 Direct Outcome Models................................... 58
4.4 Inverse Propensity Score Methods ....................... 60
4.5 Practical Issues ....................................... 69
4.6 Concluding Remarks...................................... 70
5 The Pros and Cons of Online Evaluation
5.1 Relevance............................
5.2 Biases ..............................
5.3 Experiment Effects ..................
5.4 Reusability..........................
73
74
75
76
77
6 Online Evaluation in Practice 79
6.1 Case Studies Approach..................................... 79
6.2 Ethical Considerations.................................... 80
6.3 Implementing Online Evaluations........................... 81
6.4 Recruiting Users for Reliable Evaluation ................. 88
6.5 Validation, Log Analysis and Filtering .................. 90
6.6 Considerations and Tools for Data Analysis ............... 91
7 Concluding Remarks 95
Acknowledgements 101
References
103
|
| any_adam_object | 1 |
| author | Hofmann, Katja Li, Lihong Radlinski, Filip |
| author_facet | Hofmann, Katja Li, Lihong Radlinski, Filip |
| author_role | aut aut aut |
| author_sort | Hofmann, Katja |
| author_variant | k h kh l l ll f r fr |
| building | Verbundindex |
| bvnumber | BV043838272 |
| classification_rvk | ST 270 |
| ctrlnum | (OCoLC)968135764 (DE-599)GBV870101412 |
| discipline | Informatik |
| format | Book |
| fullrecord | <?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim"><record><leader>02145nam a2200457 cb4500</leader><controlfield tag="001">BV043838272</controlfield><controlfield tag="003">DE-604</controlfield><controlfield tag="005">20161221 </controlfield><controlfield tag="007">t</controlfield><controlfield tag="008">161021s2016 |||| 00||| eng d</controlfield><datafield tag="020" ind1=" " ind2=" "><subfield code="a">9781680831634</subfield><subfield code="9">978-1-68083-163-4</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(OCoLC)968135764</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-599)GBV870101412</subfield></datafield><datafield tag="040" ind1=" " ind2=" "><subfield code="a">DE-604</subfield><subfield code="b">ger</subfield><subfield code="e">rda</subfield></datafield><datafield tag="041" ind1="0" ind2=" "><subfield code="a">eng</subfield></datafield><datafield tag="049" ind1=" " ind2=" "><subfield code="a">DE-355</subfield></datafield><datafield tag="084" ind1=" " ind2=" "><subfield code="a">ST 270</subfield><subfield code="0">(DE-625)143638:</subfield><subfield code="2">rvk</subfield></datafield><datafield tag="100" ind1="1" ind2=" "><subfield code="a">Hofmann, Katja</subfield><subfield code="e">Verfasser</subfield><subfield code="4">aut</subfield></datafield><datafield tag="245" ind1="1" ind2="0"><subfield code="a">Online evaluation for information retrieval</subfield><subfield code="c">Katja Hofmann, Lihong Li, Filip Radlinski</subfield></datafield><datafield tag="264" ind1=" " ind2="1"><subfield code="a">Boston</subfield><subfield code="b">Now</subfield><subfield code="c">[2016]</subfield></datafield><datafield tag="264" ind1=" " ind2="4"><subfield code="c">© 2016</subfield></datafield><datafield tag="300" ind1=" " ind2=" "><subfield code="a">x, 123 Seiten</subfield></datafield><datafield tag="336" ind1=" " ind2=" "><subfield code="b">txt</subfield><subfield code="2">rdacontent</subfield></datafield><datafield tag="337" ind1=" " ind2=" "><subfield code="b">n</subfield><subfield code="2">rdamedia</subfield></datafield><datafield tag="338" ind1=" " ind2=" "><subfield code="b">nc</subfield><subfield code="2">rdacarrier</subfield></datafield><datafield tag="490" ind1="1" ind2=" "><subfield code="a">Foundations and trends in information retrieval</subfield><subfield code="v">volume 10, issue 1</subfield></datafield><datafield tag="650" ind1="0" ind2="7"><subfield code="a">Benutzerverhalten</subfield><subfield code="0">(DE-588)4122898-4</subfield><subfield code="2">gnd</subfield><subfield code="9">rswk-swf</subfield></datafield><datafield tag="650" ind1="0" ind2="7"><subfield code="a">Information Retrieval</subfield><subfield code="0">(DE-588)4072803-1</subfield><subfield code="2">gnd</subfield><subfield code="9">rswk-swf</subfield></datafield><datafield tag="650" ind1="0" ind2="7"><subfield code="a">Erfolgskontrolle</subfield><subfield code="0">(DE-588)4015228-5</subfield><subfield code="2">gnd</subfield><subfield code="9">rswk-swf</subfield></datafield><datafield tag="650" ind1="0" ind2="7"><subfield code="a">Internet</subfield><subfield code="0">(DE-588)4308416-3</subfield><subfield code="2">gnd</subfield><subfield code="9">rswk-swf</subfield></datafield><datafield tag="689" ind1="0" ind2="0"><subfield code="a">Information Retrieval</subfield><subfield code="0">(DE-588)4072803-1</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="0" ind2="1"><subfield code="a">Erfolgskontrolle</subfield><subfield code="0">(DE-588)4015228-5</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="0" ind2="2"><subfield code="a">Internet</subfield><subfield code="0">(DE-588)4308416-3</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="0" ind2="3"><subfield code="a">Benutzerverhalten</subfield><subfield code="0">(DE-588)4122898-4</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="0" ind2=" "><subfield code="5">DE-604</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Li, Lihong</subfield><subfield code="e">Verfasser</subfield><subfield code="4">aut</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Radlinski, Filip</subfield><subfield code="e">Verfasser</subfield><subfield code="4">aut</subfield></datafield><datafield tag="830" ind1=" " ind2="0"><subfield code="a">Foundations and trends in information retrieval</subfield><subfield code="v">volume 10, issue 1</subfield><subfield code="w">(DE-604)BV035495746</subfield><subfield code="9">10,1</subfield></datafield><datafield tag="856" ind1="4" ind2="2"><subfield code="m">Digitalisierung UB Regensburg - ADAM Catalogue Enrichment</subfield><subfield code="q">application/pdf</subfield><subfield code="u">http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=029248866&sequence=000002&line_number=0001&func_code=DB_RECORDS&service_type=MEDIA</subfield><subfield code="3">Klappentext</subfield></datafield><datafield tag="856" ind1="4" ind2="2"><subfield code="m">Digitalisierung UB Regensburg - ADAM Catalogue Enrichment</subfield><subfield code="q">application/pdf</subfield><subfield code="u">http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=029248866&sequence=000004&line_number=0002&func_code=DB_RECORDS&service_type=MEDIA</subfield><subfield code="3">Inhaltsverzeichnis</subfield></datafield><datafield tag="999" ind1=" " ind2=" "><subfield code="a">oai:aleph.bib-bvb.de:BVB01-029248866</subfield></datafield></record></collection> |
| id | DE-604.BV043838272 |
| illustrated | Not Illustrated |
| indexdate | 2024-07-10T07:36:26Z |
| institution | BVB |
| isbn | 9781680831634 |
| language | English |
| oai_aleph_id | oai:aleph.bib-bvb.de:BVB01-029248866 |
| oclc_num | 968135764 |
| open_access_boolean | |
| owner | DE-355 DE-BY-UBR |
| owner_facet | DE-355 DE-BY-UBR |
| physical | x, 123 Seiten |
| publishDate | 2016 |
| publishDateSearch | 2016 |
| publishDateSort | 2016 |
| publisher | Now |
| record_format | marc |
| series | Foundations and trends in information retrieval |
| series2 | Foundations and trends in information retrieval |
| spelling | Hofmann, Katja Verfasser aut Online evaluation for information retrieval Katja Hofmann, Lihong Li, Filip Radlinski Boston Now [2016] © 2016 x, 123 Seiten txt rdacontent n rdamedia nc rdacarrier Foundations and trends in information retrieval volume 10, issue 1 Benutzerverhalten (DE-588)4122898-4 gnd rswk-swf Information Retrieval (DE-588)4072803-1 gnd rswk-swf Erfolgskontrolle (DE-588)4015228-5 gnd rswk-swf Internet (DE-588)4308416-3 gnd rswk-swf Information Retrieval (DE-588)4072803-1 s Erfolgskontrolle (DE-588)4015228-5 s Internet (DE-588)4308416-3 s Benutzerverhalten (DE-588)4122898-4 s DE-604 Li, Lihong Verfasser aut Radlinski, Filip Verfasser aut Foundations and trends in information retrieval volume 10, issue 1 (DE-604)BV035495746 10,1 Digitalisierung UB Regensburg - ADAM Catalogue Enrichment application/pdf http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=029248866&sequence=000002&line_number=0001&func_code=DB_RECORDS&service_type=MEDIA Klappentext Digitalisierung UB Regensburg - ADAM Catalogue Enrichment application/pdf http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=029248866&sequence=000004&line_number=0002&func_code=DB_RECORDS&service_type=MEDIA Inhaltsverzeichnis |
| spellingShingle | Hofmann, Katja Li, Lihong Radlinski, Filip Online evaluation for information retrieval Foundations and trends in information retrieval Benutzerverhalten (DE-588)4122898-4 gnd Information Retrieval (DE-588)4072803-1 gnd Erfolgskontrolle (DE-588)4015228-5 gnd Internet (DE-588)4308416-3 gnd |
| subject_GND | (DE-588)4122898-4 (DE-588)4072803-1 (DE-588)4015228-5 (DE-588)4308416-3 |
| title | Online evaluation for information retrieval |
| title_auth | Online evaluation for information retrieval |
| title_exact_search | Online evaluation for information retrieval |
| title_full | Online evaluation for information retrieval Katja Hofmann, Lihong Li, Filip Radlinski |
| title_fullStr | Online evaluation for information retrieval Katja Hofmann, Lihong Li, Filip Radlinski |
| title_full_unstemmed | Online evaluation for information retrieval Katja Hofmann, Lihong Li, Filip Radlinski |
| title_short | Online evaluation for information retrieval |
| title_sort | online evaluation for information retrieval |
| topic | Benutzerverhalten (DE-588)4122898-4 gnd Information Retrieval (DE-588)4072803-1 gnd Erfolgskontrolle (DE-588)4015228-5 gnd Internet (DE-588)4308416-3 gnd |
| topic_facet | Benutzerverhalten Information Retrieval Erfolgskontrolle Internet |
| url | http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=029248866&sequence=000002&line_number=0001&func_code=DB_RECORDS&service_type=MEDIA http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=029248866&sequence=000004&line_number=0002&func_code=DB_RECORDS&service_type=MEDIA |
| volume_link | (DE-604)BV035495746 |
| work_keys_str_mv | AT hofmannkatja onlineevaluationforinformationretrieval AT lilihong onlineevaluationforinformationretrieval AT radlinskifilip onlineevaluationforinformationretrieval |