Interactive IR user study design, evaluation, and reporting:
Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and va...
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Buch |
| Sprache: | English |
| Veröffentlicht: |
[San Rafael, California]
Morgan & Claypool Publishers
[2019]
|
| Schriftenreihe: | Synthesis Lectures on Information Concepts, Retrieval, and Services
67 |
| Schlagworte: | |
| Zusammenfassung: | Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and "collaborations" among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies |
| Beschreibung: | 95 Seiten |
| ISBN: | 9781681735795 |
Internformat
MARC
| LEADER | 00000nam a2200000 cb4500 | ||
|---|---|---|---|
| 001 | BV046054391 | ||
| 003 | DE-604 | ||
| 005 | 20190821 | ||
| 007 | t | ||
| 008 | 190718s2019 b||| 00||| eng d | ||
| 020 | |a 9781681735795 |9 978-1-68173-579-5 | ||
| 035 | |a (OCoLC)1119023725 | ||
| 035 | |a (DE-599)BVBBV046054391 | ||
| 040 | |a DE-604 |b ger |e rda | ||
| 041 | 0 | |a eng | |
| 049 | |a DE-12 | ||
| 100 | 1 | |a Liu, Jiqun |e Verfasser |4 aut | |
| 245 | 1 | 0 | |a Interactive IR user study design, evaluation, and reporting |c Jiqun Liu and Chirag Shah, Rutgers University |
| 264 | 1 | |a [San Rafael, California] |b Morgan & Claypool Publishers |c [2019] | |
| 264 | 4 | |c © 2019 | |
| 300 | |a 95 Seiten | ||
| 336 | |b txt |2 rdacontent | ||
| 337 | |b n |2 rdamedia | ||
| 338 | |b nc |2 rdacarrier | ||
| 490 | 0 | |a Synthesis Lectures on Information Concepts, Retrieval, and Services |v 67 | |
| 505 | 8 | |a 1. Introduction -- 1.1. Background : user study in IIR research -- 1.2. Aim of the work -- 1.3. Systematic review and user study evaluation -- 1.4. Implications of the faceted framework | |
| 505 | 8 | |a 2. Interactive information retrieval -- 2.1. Background -- 2.2. Understanding user behavior and experience -- 2.3. System/interface features evaluation -- 2.4. Meta-evaluation of evaluation metrics -- 2.5. Summary | |
| 505 | 8 | |a 3. Methodology : paper selection and coding scheme -- 3.1. Paper selection : SIGIR, CHIIR, CHI, JASIST, IP&M, and TOIS -- 3.2. Coding scheme | |
| 505 | 8 | |a 4. Faceted framework of IIR user studies -- 4.1. Structure of the faceted framework -- 4.2. Under-reported facets and factors -- 4.3. Summary | |
| 505 | 8 | |a 5. Evaluating IIR user studies of different types -- 5.1. Understanding user behavior and experience -- 5.2. System/interface features evaluation -- 5.3. Meta-evaluation of evaluation measures -- 5.4. Summary | |
| 505 | 8 | |a 6. Implications and limitations of the faceted framework -- 6.1. Implications for interaction IR user study design and evaluation -- 6.2. Implications for IIR user study reporting practices -- 6.3. Limitations and challenges -- 7. Conclusion and future directions | |
| 520 | 3 | |a Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and "collaborations" among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies | |
| 653 | 0 | |a Information storage and retrieval systems / Use studies | |
| 653 | 0 | |a Information services / Use studies | |
| 653 | 0 | |a Research / Evaluation | |
| 653 | 0 | |a LANGUAGE ARTS & DISCIPLINES / Library & Information Science / General | |
| 653 | 0 | |a Information services | |
| 653 | 0 | |a Information storage and retrieval systems | |
| 653 | 0 | |a Research / Evaluation | |
| 653 | 6 | |a Electronic books | |
| 653 | 6 | |a Electronic books | |
| 653 | 6 | |a Use studies | |
| 700 | 1 | |a Shah, Chirag |d 1978- |e Sonstige |0 (DE-588)1032412291 |4 oth | |
| 776 | 0 | 8 | |i Erscheint auch als |n Online-Ausgabe, ebk |z 978-1-68173-580-1 |
| 999 | |a oai:aleph.bib-bvb.de:BVB01-031435850 | ||
Datensatz im Suchindex
| _version_ | 1804180324257628160 |
|---|---|
| any_adam_object | |
| author | Liu, Jiqun |
| author_GND | (DE-588)1032412291 |
| author_facet | Liu, Jiqun |
| author_role | aut |
| author_sort | Liu, Jiqun |
| author_variant | j l jl |
| building | Verbundindex |
| bvnumber | BV046054391 |
| contents | 1. Introduction -- 1.1. Background : user study in IIR research -- 1.2. Aim of the work -- 1.3. Systematic review and user study evaluation -- 1.4. Implications of the faceted framework 2. Interactive information retrieval -- 2.1. Background -- 2.2. Understanding user behavior and experience -- 2.3. System/interface features evaluation -- 2.4. Meta-evaluation of evaluation metrics -- 2.5. Summary 3. Methodology : paper selection and coding scheme -- 3.1. Paper selection : SIGIR, CHIIR, CHI, JASIST, IP&M, and TOIS -- 3.2. Coding scheme 4. Faceted framework of IIR user studies -- 4.1. Structure of the faceted framework -- 4.2. Under-reported facets and factors -- 4.3. Summary 5. Evaluating IIR user studies of different types -- 5.1. Understanding user behavior and experience -- 5.2. System/interface features evaluation -- 5.3. Meta-evaluation of evaluation measures -- 5.4. Summary 6. Implications and limitations of the faceted framework -- 6.1. Implications for interaction IR user study design and evaluation -- 6.2. Implications for IIR user study reporting practices -- 6.3. Limitations and challenges -- 7. Conclusion and future directions |
| ctrlnum | (OCoLC)1119023725 (DE-599)BVBBV046054391 |
| format | Book |
| fullrecord | <?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim"><record><leader>04145nam a2200505 cb4500</leader><controlfield tag="001">BV046054391</controlfield><controlfield tag="003">DE-604</controlfield><controlfield tag="005">20190821 </controlfield><controlfield tag="007">t</controlfield><controlfield tag="008">190718s2019 b||| 00||| eng d</controlfield><datafield tag="020" ind1=" " ind2=" "><subfield code="a">9781681735795</subfield><subfield code="9">978-1-68173-579-5</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(OCoLC)1119023725</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-599)BVBBV046054391</subfield></datafield><datafield tag="040" ind1=" " ind2=" "><subfield code="a">DE-604</subfield><subfield code="b">ger</subfield><subfield code="e">rda</subfield></datafield><datafield tag="041" ind1="0" ind2=" "><subfield code="a">eng</subfield></datafield><datafield tag="049" ind1=" " ind2=" "><subfield code="a">DE-12</subfield></datafield><datafield tag="100" ind1="1" ind2=" "><subfield code="a">Liu, Jiqun</subfield><subfield code="e">Verfasser</subfield><subfield code="4">aut</subfield></datafield><datafield tag="245" ind1="1" ind2="0"><subfield code="a">Interactive IR user study design, evaluation, and reporting</subfield><subfield code="c">Jiqun Liu and Chirag Shah, Rutgers University</subfield></datafield><datafield tag="264" ind1=" " ind2="1"><subfield code="a">[San Rafael, California]</subfield><subfield code="b">Morgan & Claypool Publishers</subfield><subfield code="c">[2019]</subfield></datafield><datafield tag="264" ind1=" " ind2="4"><subfield code="c">© 2019</subfield></datafield><datafield tag="300" ind1=" " ind2=" "><subfield code="a">95 Seiten</subfield></datafield><datafield tag="336" ind1=" " ind2=" "><subfield code="b">txt</subfield><subfield code="2">rdacontent</subfield></datafield><datafield tag="337" ind1=" " ind2=" "><subfield code="b">n</subfield><subfield code="2">rdamedia</subfield></datafield><datafield tag="338" ind1=" " ind2=" "><subfield code="b">nc</subfield><subfield code="2">rdacarrier</subfield></datafield><datafield tag="490" ind1="0" ind2=" "><subfield code="a">Synthesis Lectures on Information Concepts, Retrieval, and Services</subfield><subfield code="v">67</subfield></datafield><datafield tag="505" ind1="8" ind2=" "><subfield code="a">1. Introduction -- 1.1. Background : user study in IIR research -- 1.2. Aim of the work -- 1.3. Systematic review and user study evaluation -- 1.4. Implications of the faceted framework</subfield></datafield><datafield tag="505" ind1="8" ind2=" "><subfield code="a">2. Interactive information retrieval -- 2.1. Background -- 2.2. Understanding user behavior and experience -- 2.3. System/interface features evaluation -- 2.4. Meta-evaluation of evaluation metrics -- 2.5. Summary</subfield></datafield><datafield tag="505" ind1="8" ind2=" "><subfield code="a">3. Methodology : paper selection and coding scheme -- 3.1. Paper selection : SIGIR, CHIIR, CHI, JASIST, IP&M, and TOIS -- 3.2. Coding scheme</subfield></datafield><datafield tag="505" ind1="8" ind2=" "><subfield code="a">4. Faceted framework of IIR user studies -- 4.1. Structure of the faceted framework -- 4.2. Under-reported facets and factors -- 4.3. Summary</subfield></datafield><datafield tag="505" ind1="8" ind2=" "><subfield code="a">5. Evaluating IIR user studies of different types -- 5.1. Understanding user behavior and experience -- 5.2. System/interface features evaluation -- 5.3. Meta-evaluation of evaluation measures -- 5.4. Summary</subfield></datafield><datafield tag="505" ind1="8" ind2=" "><subfield code="a">6. Implications and limitations of the faceted framework -- 6.1. Implications for interaction IR user study design and evaluation -- 6.2. Implications for IIR user study reporting practices -- 6.3. Limitations and challenges -- 7. Conclusion and future directions</subfield></datafield><datafield tag="520" ind1="3" ind2=" "><subfield code="a">Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and "collaborations" among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies</subfield></datafield><datafield tag="653" ind1=" " ind2="0"><subfield code="a">Information storage and retrieval systems / Use studies</subfield></datafield><datafield tag="653" ind1=" " ind2="0"><subfield code="a">Information services / Use studies</subfield></datafield><datafield tag="653" ind1=" " ind2="0"><subfield code="a">Research / Evaluation</subfield></datafield><datafield tag="653" ind1=" " ind2="0"><subfield code="a">LANGUAGE ARTS & DISCIPLINES / Library & Information Science / General</subfield></datafield><datafield tag="653" ind1=" " ind2="0"><subfield code="a">Information services</subfield></datafield><datafield tag="653" ind1=" " ind2="0"><subfield code="a">Information storage and retrieval systems</subfield></datafield><datafield tag="653" ind1=" " ind2="0"><subfield code="a">Research / Evaluation</subfield></datafield><datafield tag="653" ind1=" " ind2="6"><subfield code="a">Electronic books</subfield></datafield><datafield tag="653" ind1=" " ind2="6"><subfield code="a">Electronic books</subfield></datafield><datafield tag="653" ind1=" " ind2="6"><subfield code="a">Use studies</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Shah, Chirag</subfield><subfield code="d">1978-</subfield><subfield code="e">Sonstige</subfield><subfield code="0">(DE-588)1032412291</subfield><subfield code="4">oth</subfield></datafield><datafield tag="776" ind1="0" ind2="8"><subfield code="i">Erscheint auch als</subfield><subfield code="n">Online-Ausgabe, ebk</subfield><subfield code="z">978-1-68173-580-1</subfield></datafield><datafield tag="999" ind1=" " ind2=" "><subfield code="a">oai:aleph.bib-bvb.de:BVB01-031435850</subfield></datafield></record></collection> |
| id | DE-604.BV046054391 |
| illustrated | Not Illustrated |
| indexdate | 2024-07-10T08:33:57Z |
| institution | BVB |
| isbn | 9781681735795 |
| language | English |
| oai_aleph_id | oai:aleph.bib-bvb.de:BVB01-031435850 |
| oclc_num | 1119023725 |
| open_access_boolean | |
| owner | DE-12 |
| owner_facet | DE-12 |
| physical | 95 Seiten |
| publishDate | 2019 |
| publishDateSearch | 2019 |
| publishDateSort | 2019 |
| publisher | Morgan & Claypool Publishers |
| record_format | marc |
| series2 | Synthesis Lectures on Information Concepts, Retrieval, and Services |
| spelling | Liu, Jiqun Verfasser aut Interactive IR user study design, evaluation, and reporting Jiqun Liu and Chirag Shah, Rutgers University [San Rafael, California] Morgan & Claypool Publishers [2019] © 2019 95 Seiten txt rdacontent n rdamedia nc rdacarrier Synthesis Lectures on Information Concepts, Retrieval, and Services 67 1. Introduction -- 1.1. Background : user study in IIR research -- 1.2. Aim of the work -- 1.3. Systematic review and user study evaluation -- 1.4. Implications of the faceted framework 2. Interactive information retrieval -- 2.1. Background -- 2.2. Understanding user behavior and experience -- 2.3. System/interface features evaluation -- 2.4. Meta-evaluation of evaluation metrics -- 2.5. Summary 3. Methodology : paper selection and coding scheme -- 3.1. Paper selection : SIGIR, CHIIR, CHI, JASIST, IP&M, and TOIS -- 3.2. Coding scheme 4. Faceted framework of IIR user studies -- 4.1. Structure of the faceted framework -- 4.2. Under-reported facets and factors -- 4.3. Summary 5. Evaluating IIR user studies of different types -- 5.1. Understanding user behavior and experience -- 5.2. System/interface features evaluation -- 5.3. Meta-evaluation of evaluation measures -- 5.4. Summary 6. Implications and limitations of the faceted framework -- 6.1. Implications for interaction IR user study design and evaluation -- 6.2. Implications for IIR user study reporting practices -- 6.3. Limitations and challenges -- 7. Conclusion and future directions Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and "collaborations" among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies Information storage and retrieval systems / Use studies Information services / Use studies Research / Evaluation LANGUAGE ARTS & DISCIPLINES / Library & Information Science / General Information services Information storage and retrieval systems Electronic books Use studies Shah, Chirag 1978- Sonstige (DE-588)1032412291 oth Erscheint auch als Online-Ausgabe, ebk 978-1-68173-580-1 |
| spellingShingle | Liu, Jiqun Interactive IR user study design, evaluation, and reporting 1. Introduction -- 1.1. Background : user study in IIR research -- 1.2. Aim of the work -- 1.3. Systematic review and user study evaluation -- 1.4. Implications of the faceted framework 2. Interactive information retrieval -- 2.1. Background -- 2.2. Understanding user behavior and experience -- 2.3. System/interface features evaluation -- 2.4. Meta-evaluation of evaluation metrics -- 2.5. Summary 3. Methodology : paper selection and coding scheme -- 3.1. Paper selection : SIGIR, CHIIR, CHI, JASIST, IP&M, and TOIS -- 3.2. Coding scheme 4. Faceted framework of IIR user studies -- 4.1. Structure of the faceted framework -- 4.2. Under-reported facets and factors -- 4.3. Summary 5. Evaluating IIR user studies of different types -- 5.1. Understanding user behavior and experience -- 5.2. System/interface features evaluation -- 5.3. Meta-evaluation of evaluation measures -- 5.4. Summary 6. Implications and limitations of the faceted framework -- 6.1. Implications for interaction IR user study design and evaluation -- 6.2. Implications for IIR user study reporting practices -- 6.3. Limitations and challenges -- 7. Conclusion and future directions |
| title | Interactive IR user study design, evaluation, and reporting |
| title_auth | Interactive IR user study design, evaluation, and reporting |
| title_exact_search | Interactive IR user study design, evaluation, and reporting |
| title_full | Interactive IR user study design, evaluation, and reporting Jiqun Liu and Chirag Shah, Rutgers University |
| title_fullStr | Interactive IR user study design, evaluation, and reporting Jiqun Liu and Chirag Shah, Rutgers University |
| title_full_unstemmed | Interactive IR user study design, evaluation, and reporting Jiqun Liu and Chirag Shah, Rutgers University |
| title_short | Interactive IR user study design, evaluation, and reporting |
| title_sort | interactive ir user study design evaluation and reporting |
| work_keys_str_mv | AT liujiqun interactiveiruserstudydesignevaluationandreporting AT shahchirag interactiveiruserstudydesignevaluationandreporting |