AI scoring for international large-scale assessments using a deep learning model and multilingual data:
Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to...
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Weitere Verfasser: | , , , , |
| Format: | Elektronisch E-Book |
| Sprache: | English |
| Veröffentlicht: |
Paris
OECD Publishing
2023
|
| Schriftenreihe: | OECD Education Working Papers
no.287 |
| Schlagworte: | |
| Online-Zugang: | UBA01 TUM01 UBG01 UEI01 UER01 UPA01 UBR01 UBW01 FFW01 FNU01 EUV01 FRO01 FHR01 FHN01 FHI01 Volltext |
| Zusammenfassung: | Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment |
| Beschreibung: | 1 Online-Ressource (34 Seiten) |
| DOI: | 10.1787/9918e1fb-en |
Internformat
MARC
| LEADER | 00000nmm a22000001cb4500 | ||
|---|---|---|---|
| 001 | BV049038321 | ||
| 003 | DE-604 | ||
| 005 | 00000000000000.0 | ||
| 007 | cr|uuu---uuuuu | ||
| 008 | 230706s2023 |||| o||u| ||||||eng d | ||
| 024 | 7 | |a 10.1787/9918e1fb-en |2 doi | |
| 035 | |a (ZDB-13-SOC)092691269 | ||
| 035 | |a (OCoLC)1390808280 | ||
| 035 | |a (DE-599)KEP092691269 | ||
| 040 | |a DE-604 |b ger |e rda | ||
| 041 | 0 | |a eng | |
| 049 | |a DE-521 |a DE-1028 |a DE-573 |a DE-92 |a DE-898 |a DE-1049 |a DE-861 |a DE-91 |a DE-384 |a DE-473 |a DE-355 |a DE-20 |a DE-824 |a DE-29 |a DE-739 |a DE-188 | ||
| 100 | 1 | |a Okubo, Tomoya |e Verfasser |4 aut | |
| 245 | 1 | 0 | |a AI scoring for international large-scale assessments using a deep learning model and multilingual data |c Tomoya, Okubo ... [et al] |
| 264 | 1 | |a Paris |b OECD Publishing |c 2023 | |
| 300 | |a 1 Online-Ressource (34 Seiten) | ||
| 336 | |b txt |2 rdacontent | ||
| 337 | |b c |2 rdamedia | ||
| 338 | |b cr |2 rdacarrier | ||
| 490 | 0 | |a OECD Education Working Papers |v no.287 | |
| 520 | 3 | |a Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment | |
| 650 | 4 | |a Education | |
| 650 | 4 | |a Science and Technology | |
| 700 | 1 | |a Houlden, Wayne |4 ctb | |
| 700 | 1 | |a Montuoro, Paul |4 ctb | |
| 700 | 1 | |a Reinertsen, Nate |4 ctb | |
| 700 | 1 | |a Tse, Chi Sum |4 ctb | |
| 700 | 1 | |a Bastianic, Tanja |4 ctb | |
| 856 | 4 | 0 | |u https://doi.org/10.1787/9918e1fb-en |x Verlag |z URL des Erstveröffentlichers |3 Volltext |
| 912 | |a ZDB-13-SOC | ||
| 999 | |a oai:aleph.bib-bvb.de:BVB01-034300923 | ||
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l UBA01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l TUM01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l UBG01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l UEI01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l UER01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l UPA01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l UBR01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l UBW01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l FFW01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l FNU01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l EUV01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l FRO01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l FHR01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l FHN01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
| 966 | e | |u https://doi.org/10.1787/9918e1fb-en |l FHI01 |p ZDB-13-SOC |x Verlag |3 Volltext | |
Datensatz im Suchindex
| _version_ | 1804185329389797376 |
|---|---|
| adam_txt | |
| any_adam_object | |
| any_adam_object_boolean | |
| author | Okubo, Tomoya |
| author2 | Houlden, Wayne Montuoro, Paul Reinertsen, Nate Tse, Chi Sum Bastianic, Tanja |
| author2_role | ctb ctb ctb ctb ctb |
| author2_variant | w h wh p m pm n r nr c s t cs cst t b tb |
| author_facet | Okubo, Tomoya Houlden, Wayne Montuoro, Paul Reinertsen, Nate Tse, Chi Sum Bastianic, Tanja |
| author_role | aut |
| author_sort | Okubo, Tomoya |
| author_variant | t o to |
| building | Verbundindex |
| bvnumber | BV049038321 |
| collection | ZDB-13-SOC |
| ctrlnum | (ZDB-13-SOC)092691269 (OCoLC)1390808280 (DE-599)KEP092691269 |
| discipline | Wirtschaftswissenschaften |
| discipline_str_mv | Wirtschaftswissenschaften |
| doi_str_mv | 10.1787/9918e1fb-en |
| format | Electronic eBook |
| fullrecord | <?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim"><record><leader>03821nmm a22005771cb4500</leader><controlfield tag="001">BV049038321</controlfield><controlfield tag="003">DE-604</controlfield><controlfield tag="005">00000000000000.0</controlfield><controlfield tag="007">cr|uuu---uuuuu</controlfield><controlfield tag="008">230706s2023 |||| o||u| ||||||eng d</controlfield><datafield tag="024" ind1="7" ind2=" "><subfield code="a">10.1787/9918e1fb-en</subfield><subfield code="2">doi</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(ZDB-13-SOC)092691269</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(OCoLC)1390808280</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-599)KEP092691269</subfield></datafield><datafield tag="040" ind1=" " ind2=" "><subfield code="a">DE-604</subfield><subfield code="b">ger</subfield><subfield code="e">rda</subfield></datafield><datafield tag="041" ind1="0" ind2=" "><subfield code="a">eng</subfield></datafield><datafield tag="049" ind1=" " ind2=" "><subfield code="a">DE-521</subfield><subfield code="a">DE-1028</subfield><subfield code="a">DE-573</subfield><subfield code="a">DE-92</subfield><subfield code="a">DE-898</subfield><subfield code="a">DE-1049</subfield><subfield code="a">DE-861</subfield><subfield code="a">DE-91</subfield><subfield code="a">DE-384</subfield><subfield code="a">DE-473</subfield><subfield code="a">DE-355</subfield><subfield code="a">DE-20</subfield><subfield code="a">DE-824</subfield><subfield code="a">DE-29</subfield><subfield code="a">DE-739</subfield><subfield code="a">DE-188</subfield></datafield><datafield tag="100" ind1="1" ind2=" "><subfield code="a">Okubo, Tomoya</subfield><subfield code="e">Verfasser</subfield><subfield code="4">aut</subfield></datafield><datafield tag="245" ind1="1" ind2="0"><subfield code="a">AI scoring for international large-scale assessments using a deep learning model and multilingual data</subfield><subfield code="c">Tomoya, Okubo ... [et al]</subfield></datafield><datafield tag="264" ind1=" " ind2="1"><subfield code="a">Paris</subfield><subfield code="b">OECD Publishing</subfield><subfield code="c">2023</subfield></datafield><datafield tag="300" ind1=" " ind2=" "><subfield code="a">1 Online-Ressource (34 Seiten)</subfield></datafield><datafield tag="336" ind1=" " ind2=" "><subfield code="b">txt</subfield><subfield code="2">rdacontent</subfield></datafield><datafield tag="337" ind1=" " ind2=" "><subfield code="b">c</subfield><subfield code="2">rdamedia</subfield></datafield><datafield tag="338" ind1=" " ind2=" "><subfield code="b">cr</subfield><subfield code="2">rdacarrier</subfield></datafield><datafield tag="490" ind1="0" ind2=" "><subfield code="a">OECD Education Working Papers</subfield><subfield code="v">no.287</subfield></datafield><datafield tag="520" ind1="3" ind2=" "><subfield code="a">Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Education</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Science and Technology</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Houlden, Wayne</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Montuoro, Paul</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Reinertsen, Nate</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Tse, Chi Sum</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Bastianic, Tanja</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="856" ind1="4" ind2="0"><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="x">Verlag</subfield><subfield code="z">URL des Erstveröffentlichers</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">ZDB-13-SOC</subfield></datafield><datafield tag="999" ind1=" " ind2=" "><subfield code="a">oai:aleph.bib-bvb.de:BVB01-034300923</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">UBA01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">TUM01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">UBG01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">UEI01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">UER01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">UPA01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">UBR01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">UBW01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">FFW01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">FNU01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">EUV01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">FRO01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">FHR01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">FHN01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="e" ind2=" "><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="l">FHI01</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="x">Verlag</subfield><subfield code="3">Volltext</subfield></datafield></record></collection> |
| id | DE-604.BV049038321 |
| illustrated | Not Illustrated |
| index_date | 2024-07-03T22:18:26Z |
| indexdate | 2024-07-10T09:53:30Z |
| institution | BVB |
| language | English |
| oai_aleph_id | oai:aleph.bib-bvb.de:BVB01-034300923 |
| oclc_num | 1390808280 |
| open_access_boolean | |
| owner | DE-521 DE-1028 DE-573 DE-92 DE-898 DE-BY-UBR DE-1049 DE-861 DE-91 DE-BY-TUM DE-384 DE-473 DE-BY-UBG DE-355 DE-BY-UBR DE-20 DE-824 DE-29 DE-739 DE-188 |
| owner_facet | DE-521 DE-1028 DE-573 DE-92 DE-898 DE-BY-UBR DE-1049 DE-861 DE-91 DE-BY-TUM DE-384 DE-473 DE-BY-UBG DE-355 DE-BY-UBR DE-20 DE-824 DE-29 DE-739 DE-188 |
| physical | 1 Online-Ressource (34 Seiten) |
| psigel | ZDB-13-SOC |
| publishDate | 2023 |
| publishDateSearch | 2023 |
| publishDateSort | 2023 |
| publisher | OECD Publishing |
| record_format | marc |
| series2 | OECD Education Working Papers |
| spelling | Okubo, Tomoya Verfasser aut AI scoring for international large-scale assessments using a deep learning model and multilingual data Tomoya, Okubo ... [et al] Paris OECD Publishing 2023 1 Online-Ressource (34 Seiten) txt rdacontent c rdamedia cr rdacarrier OECD Education Working Papers no.287 Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment Education Science and Technology Houlden, Wayne ctb Montuoro, Paul ctb Reinertsen, Nate ctb Tse, Chi Sum ctb Bastianic, Tanja ctb https://doi.org/10.1787/9918e1fb-en Verlag URL des Erstveröffentlichers Volltext |
| spellingShingle | Okubo, Tomoya AI scoring for international large-scale assessments using a deep learning model and multilingual data Education Science and Technology |
| title | AI scoring for international large-scale assessments using a deep learning model and multilingual data |
| title_auth | AI scoring for international large-scale assessments using a deep learning model and multilingual data |
| title_exact_search | AI scoring for international large-scale assessments using a deep learning model and multilingual data |
| title_exact_search_txtP | AI scoring for international large-scale assessments using a deep learning model and multilingual data |
| title_full | AI scoring for international large-scale assessments using a deep learning model and multilingual data Tomoya, Okubo ... [et al] |
| title_fullStr | AI scoring for international large-scale assessments using a deep learning model and multilingual data Tomoya, Okubo ... [et al] |
| title_full_unstemmed | AI scoring for international large-scale assessments using a deep learning model and multilingual data Tomoya, Okubo ... [et al] |
| title_short | AI scoring for international large-scale assessments using a deep learning model and multilingual data |
| title_sort | ai scoring for international large scale assessments using a deep learning model and multilingual data |
| topic | Education Science and Technology |
| topic_facet | Education Science and Technology |
| url | https://doi.org/10.1787/9918e1fb-en |
| work_keys_str_mv | AT okubotomoya aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT houldenwayne aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT montuoropaul aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT reinertsennate aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT tsechisum aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT bastianictanja aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata |