AI scoring for international large-scale assessments using a deep learning model and multilingual data:
Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to...
Saved in:
| Main Author: | |
|---|---|
| Other Authors: | , , , , |
| Format: | Electronic eBook |
| Language: | English |
| Published: |
Paris
OECD Publishing
2023
|
| Series: | OECD Education Working Papers
no.287 |
| Subjects: | |
| Online Access: | DE-862 DE-863 |
| Summary: | Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment. |
| Physical Description: | 1 Online-Ressource (34 Seiten) 21 x 28cm. |
Staff View
MARC
| LEADER | 00000cam a22000002 4500 | ||
|---|---|---|---|
| 001 | ZDB-13-SOC-092691269 | ||
| 003 | DE-627-1 | ||
| 005 | 20241028114245.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 230515s2023 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1787/9918e1fb-en |2 doi | |
| 035 | |a (DE-627-1)092691269 | ||
| 035 | |a (DE-599)KEP092691269 | ||
| 035 | |a (FR-PaOEC)9918e1fb-en | ||
| 035 | |a (DE-627-1)092691269 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rda | ||
| 041 | |a eng | ||
| 100 | 1 | |a Okubo, Tomoya |e VerfasserIn |4 aut | |
| 245 | 1 | 0 | |a AI scoring for international large-scale assessments using a deep learning model and multilingual data |c Tomoya, Okubo ... [et al] |
| 264 | 1 | |a Paris |b OECD Publishing |c 2023 | |
| 300 | |a 1 Online-Ressource (34 Seiten) |c 21 x 28cm. | ||
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a Computermedien |b c |2 rdamedia | ||
| 338 | |a Online-Ressource |b cr |2 rdacarrier | ||
| 490 | 0 | |a OECD Education Working Papers |v no.287 | |
| 520 | |a Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment. | ||
| 650 | 4 | |a Education | |
| 650 | 4 | |a Science and Technology | |
| 700 | 1 | |a Houlden, Wayne |e MitwirkendeR |4 ctb | |
| 700 | 1 | |a Montuoro, Paul |e MitwirkendeR |4 ctb | |
| 700 | 1 | |a Reinertsen, Nate |e MitwirkendeR |4 ctb | |
| 700 | 1 | |a Tse, Chi Sum |e MitwirkendeR |4 ctb | |
| 700 | 1 | |a Bastianic, Tanja |e MitwirkendeR |4 ctb | |
| 966 | 4 | 0 | |l DE-862 |p ZDB-13-SOC |q FWS_PDA_SOC |u https://doi.org/10.1787/9918e1fb-en |m X:OECD |x Resolving-System |z lizenzpflichtig |3 Volltext |
| 966 | 4 | 0 | |l DE-863 |p ZDB-13-SOC |q FWS_PDA_SOC |u https://doi.org/10.1787/9918e1fb-en |m X:OECD |x Resolving-System |z lizenzpflichtig |3 Volltext |
| 912 | |a ZDB-13-SOC-ebook | ||
| 912 | |a ZDB-13-SOC | ||
| 912 | |a BSZ-13-SOC-education | ||
| 951 | |a BO | ||
| 912 | |a ZDB-13-SOC | ||
| 049 | |a DE-862 | ||
| 049 | |a DE-863 | ||
Record in the Search Index
| DE-BY-FWS_katkey | ZDB-13-SOC-092691269 |
|---|---|
| _version_ | 1826942501208981504 |
| adam_text | |
| any_adam_object | |
| author | Okubo, Tomoya |
| author2 | Houlden, Wayne Montuoro, Paul Reinertsen, Nate Tse, Chi Sum Bastianic, Tanja |
| author2_role | ctb ctb ctb ctb ctb |
| author2_variant | w h wh p m pm n r nr c s t cs cst t b tb |
| author_facet | Okubo, Tomoya Houlden, Wayne Montuoro, Paul Reinertsen, Nate Tse, Chi Sum Bastianic, Tanja |
| author_role | aut |
| author_sort | Okubo, Tomoya |
| author_variant | t o to |
| building | Verbundindex |
| bvnumber | localFWS |
| collection | ZDB-13-SOC-ebook ZDB-13-SOC BSZ-13-SOC-education |
| ctrlnum | (DE-627-1)092691269 (DE-599)KEP092691269 (FR-PaOEC)9918e1fb-en |
| discipline | Wirtschaftswissenschaften |
| format | Electronic eBook |
| fullrecord | <?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim"><record><leader>02546cam a22004212 4500</leader><controlfield tag="001">ZDB-13-SOC-092691269</controlfield><controlfield tag="003">DE-627-1</controlfield><controlfield tag="005">20241028114245.0</controlfield><controlfield tag="007">cr uuu---uuuuu</controlfield><controlfield tag="008">230515s2023 xx |||||o 00| ||eng c</controlfield><datafield tag="024" ind1="7" ind2=" "><subfield code="a">10.1787/9918e1fb-en</subfield><subfield code="2">doi</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-627-1)092691269</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-599)KEP092691269</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(FR-PaOEC)9918e1fb-en</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-627-1)092691269</subfield></datafield><datafield tag="040" ind1=" " ind2=" "><subfield code="a">DE-627</subfield><subfield code="b">ger</subfield><subfield code="c">DE-627</subfield><subfield code="e">rda</subfield></datafield><datafield tag="041" ind1=" " ind2=" "><subfield code="a">eng</subfield></datafield><datafield tag="100" ind1="1" ind2=" "><subfield code="a">Okubo, Tomoya</subfield><subfield code="e">VerfasserIn</subfield><subfield code="4">aut</subfield></datafield><datafield tag="245" ind1="1" ind2="0"><subfield code="a">AI scoring for international large-scale assessments using a deep learning model and multilingual data</subfield><subfield code="c">Tomoya, Okubo ... [et al]</subfield></datafield><datafield tag="264" ind1=" " ind2="1"><subfield code="a">Paris</subfield><subfield code="b">OECD Publishing</subfield><subfield code="c">2023</subfield></datafield><datafield tag="300" ind1=" " ind2=" "><subfield code="a">1 Online-Ressource (34 Seiten)</subfield><subfield code="c">21 x 28cm.</subfield></datafield><datafield tag="336" ind1=" " ind2=" "><subfield code="a">Text</subfield><subfield code="b">txt</subfield><subfield code="2">rdacontent</subfield></datafield><datafield tag="337" ind1=" " ind2=" "><subfield code="a">Computermedien</subfield><subfield code="b">c</subfield><subfield code="2">rdamedia</subfield></datafield><datafield tag="338" ind1=" " ind2=" "><subfield code="a">Online-Ressource</subfield><subfield code="b">cr</subfield><subfield code="2">rdacarrier</subfield></datafield><datafield tag="490" ind1="0" ind2=" "><subfield code="a">OECD Education Working Papers</subfield><subfield code="v">no.287</subfield></datafield><datafield tag="520" ind1=" " ind2=" "><subfield code="a">Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment.</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Education</subfield></datafield><datafield tag="650" ind1=" " ind2="4"><subfield code="a">Science and Technology</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Houlden, Wayne</subfield><subfield code="e">MitwirkendeR</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Montuoro, Paul</subfield><subfield code="e">MitwirkendeR</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Reinertsen, Nate</subfield><subfield code="e">MitwirkendeR</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Tse, Chi Sum</subfield><subfield code="e">MitwirkendeR</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="700" ind1="1" ind2=" "><subfield code="a">Bastianic, Tanja</subfield><subfield code="e">MitwirkendeR</subfield><subfield code="4">ctb</subfield></datafield><datafield tag="966" ind1="4" ind2="0"><subfield code="l">DE-862</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="q">FWS_PDA_SOC</subfield><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="m">X:OECD</subfield><subfield code="x">Resolving-System</subfield><subfield code="z">lizenzpflichtig</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="966" ind1="4" ind2="0"><subfield code="l">DE-863</subfield><subfield code="p">ZDB-13-SOC</subfield><subfield code="q">FWS_PDA_SOC</subfield><subfield code="u">https://doi.org/10.1787/9918e1fb-en</subfield><subfield code="m">X:OECD</subfield><subfield code="x">Resolving-System</subfield><subfield code="z">lizenzpflichtig</subfield><subfield code="3">Volltext</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">ZDB-13-SOC-ebook</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">ZDB-13-SOC</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">BSZ-13-SOC-education</subfield></datafield><datafield tag="951" ind1=" " ind2=" "><subfield code="a">BO</subfield></datafield><datafield tag="912" ind1=" " ind2=" "><subfield code="a">ZDB-13-SOC</subfield></datafield><datafield tag="049" ind1=" " ind2=" "><subfield code="a">DE-862</subfield></datafield><datafield tag="049" ind1=" " ind2=" "><subfield code="a">DE-863</subfield></datafield></record></collection> |
| id | ZDB-13-SOC-092691269 |
| illustrated | Not Illustrated |
| indexdate | 2025-03-18T14:29:00Z |
| institution | BVB |
| language | English |
| open_access_boolean | |
| owner | DE-862 DE-BY-FWS DE-863 DE-BY-FWS |
| owner_facet | DE-862 DE-BY-FWS DE-863 DE-BY-FWS |
| physical | 1 Online-Ressource (34 Seiten) 21 x 28cm. |
| psigel | ZDB-13-SOC FWS_PDA_SOC ZDB-13-SOC-ebook ZDB-13-SOC BSZ-13-SOC-education |
| publishDate | 2023 |
| publishDateSearch | 2023 |
| publishDateSort | 2023 |
| publisher | OECD Publishing |
| record_format | marc |
| series2 | OECD Education Working Papers |
| spelling | Okubo, Tomoya VerfasserIn aut AI scoring for international large-scale assessments using a deep learning model and multilingual data Tomoya, Okubo ... [et al] Paris OECD Publishing 2023 1 Online-Ressource (34 Seiten) 21 x 28cm. Text txt rdacontent Computermedien c rdamedia Online-Ressource cr rdacarrier OECD Education Working Papers no.287 Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment. Education Science and Technology Houlden, Wayne MitwirkendeR ctb Montuoro, Paul MitwirkendeR ctb Reinertsen, Nate MitwirkendeR ctb Tse, Chi Sum MitwirkendeR ctb Bastianic, Tanja MitwirkendeR ctb |
| spellingShingle | Okubo, Tomoya AI scoring for international large-scale assessments using a deep learning model and multilingual data Education Science and Technology |
| title | AI scoring for international large-scale assessments using a deep learning model and multilingual data |
| title_auth | AI scoring for international large-scale assessments using a deep learning model and multilingual data |
| title_exact_search | AI scoring for international large-scale assessments using a deep learning model and multilingual data |
| title_full | AI scoring for international large-scale assessments using a deep learning model and multilingual data Tomoya, Okubo ... [et al] |
| title_fullStr | AI scoring for international large-scale assessments using a deep learning model and multilingual data Tomoya, Okubo ... [et al] |
| title_full_unstemmed | AI scoring for international large-scale assessments using a deep learning model and multilingual data Tomoya, Okubo ... [et al] |
| title_short | AI scoring for international large-scale assessments using a deep learning model and multilingual data |
| title_sort | ai scoring for international large scale assessments using a deep learning model and multilingual data |
| topic | Education Science and Technology |
| topic_facet | Education Science and Technology |
| work_keys_str_mv | AT okubotomoya aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT houldenwayne aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT montuoropaul aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT reinertsennate aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT tsechisum aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata AT bastianictanja aiscoringforinternationallargescaleassessmentsusingadeeplearningmodelandmultilingualdata |