[每日中文朗读] LLM 的“母语”是什么?

要約

英語を中心としたコーパスで訓練された多言語LLMは、内部言語として英語を使用している可能性がある。

LLMの動作メカニズムを理解するために、EPFLの研究者らはLlama 2を対象に実験を行った。

実験の結果、Llama 2内部の「共通語」は英語ではなく、概念であることが判明した。しかし、これらの謎の文字は明らかに英語に偏った概念である。
原文: 新浪 IT之家 新智元 2024/6/3 7
llama2

本文(ピンイン)

[xīn][zhì][yuán][dǎo][][zài][][yīng][][wéi][zhǔ][de][][liào][][shàng][xùn][liàn][de][duō][][yán] LLM[shì][fǒu]使[shǐ][yòng][yīng][][zuò][wéi][nèi][][][yán][duì][][lái][] EPFL [de][yán][jiū][rén][yuán][zhēn][duì] Llama 2 [jiā][][jìn][xíng][le][][][liè][shí][yàn]

[][][yán][][xíng][de][][][shì][shén][me]

[][men][de][][][fǎn][yìng][hěn][][néng][shì][yīng][]

[dàn][shì][shí][guǒ][zhēn][][][ma][yóu][][shì][duì][][néng][gòu][tīng][shuō][][xiě][duō][zhǒng][][yán][de] LLM [lái][shuō]

[duì][][lái][] EPFL[luò][sāng][lián][bāng][][gōng][xué][yuàn][de][yán][jiū][rén][yuán][][biǎo][le][xià][miàn][zhè][piān][gōng][zuò][lái][][tàn][jiū][jìng]


[lùn][wén][][zhǐ]https://arxiv.org/ pdf / 2402.10588

[xiàng][][][zhǐ]https://github.com/ epfldlab / llmlatentlanguage

[zuò][zhě][] Llama2 [wèi][duì][xiàng][xiàng][][men][zhǎn][shì][le][][yǒu][duō][][yán][néng][][de] Transformer[shì][][][][kǎo][wèn][][de]

[xiàng][yáng][tuó][zhè][zhǒng][zài][yīng][][][xià][zhǎng][][de][][][de][duō][][yán][dào][][shì][běn][zhí][shǔ][xìng][hái][shì][jǐn][jǐn][tào][le][][fān][][de][]

[zhè][duì][][rén][men][][jiě] LLM [de][yùn][xíng][][zhì][zhì][guān][zhòng][yào]


[yào][tàn][jiū][][][xíng][de][nèi][xīn][shì][jiè][suī][rán][tīng][][lái][yǒu][diǎn][][][dàn][shí][][shang][][diǎn][][][jiǎn][dān]

[yán][jiū][rén][yuán][zài][zhè][][huà][fán][wèi][jiǎn]使[shǐ][yòng][][dìng][de][][shì][lái][bǎo][zhèng][shū][chū][de][wéi][][xìng][tóng][shí][] Llama27B [de] 32 [céng][shū][chū][quán][][][][chū][lái] [][céng][][céng][][céng][][][kāi][][de][xīn]


[][shì][][men][néng][zài][shàng][][qīng][chǔ][][kàn][dào][yáng][tuó][zài][][dào][zhōng][wén][fān][][huā][shí][de][zhěng][][tuī][][guò][chéng]

Transformer [jiāng][shū][] token [jìn][xíng][zhú][céng][yìng][shè][zuì][zhōng][][][chū][xià][][ge] token[zhōng][jiān][nèi][xiē][][men][][gài][néng][][jiě][huò][zhě][][néng][][jiě][de][][][chuàn][jiù][shì] LLM 使[shǐ][yòng][de][nèi][][][yán]

[xiǎn][rán][zài][zhōng][jiān][céng][de][][kǎo][huán][jié][yáng][tuó][yòng][de][shì][piān][xiàng][][yīng][][de][mǒu][zhǒng][shén][][wén][][zhè][][][yào][qiáng][diào][][xià][zhè][shì][yáng][tuó][de][][][xíng][wéi][yīn][wèi][][shì][zhōng][][gēn][jiù][méi][yǒu][][diǎn][yīng][]


[][][shàng][][shì][][zhōng][de][][][shí][yàn][gòu][jiàn][le][][][fān][][zhōng][wén][de][][shì][qiě][xiàn][zhì][le][zhèng][què][][àn][zhǐ][] 1 [] token[huā]

[ér][xià][][de][tǒng][][xiǎn][shì][zài] Llama2 [de][][][fèn][qián][xiàng][chuán][][zhōng][zhèng][què][zhōng][wén] token[lán][][de][gài][][yuǎn][][][yīng][wén][fān][][chéng][][de][gài][][zhōng][wén][zhī][zài][zuì][hòu][liǎng][céng][zhōng][zhàn][][zhǔ][dǎo][][wèi]


[wèi][le][fāng]便[biàn][][jiā][guān][chá][zuò][zhě][hái][jiāng][qiàn][][zài][gāo][wéi][kōng][jiān][zhōng][de][][jìng][de][][shì][huà][shí][][shì] 8192 [][wéi][][zhè][]使[shǐ][yòng] 2D [zhǎn][shì]

[cóng][shū][][dào][shū][chū][guǐ][][][hóng][][kāi][shǐ][][][][jié][shù][][men][][][kàn][dào][zhè][xiē][][jìng][][běn][dōu][shì][xiān][rào][dào][yīng][][rán][hòu][cái][fǎn][huí][zhèng][què][de][zhōng][wén]


[][guò][zhè][shì][fǒu][què][shí][biǎo][míng] Llama2 [xiān][yòng][yīng][wén][jìn][xíng][tuī][][rán][hòu][jiāng][zài][][fān][][chéng][zhōng][wén]

[zuò][zhě][biǎo][shì][][zhè][gèng][wēi][miào][][diǎn][nèi][xiē][kàn][][lái][xiàng][yīng][][de][zhōng][jiān][qiàn][][shí][][shang][duì][yìng][][chōu][xiàng][gài][niàn][ér][][shì][][][de][yīng][wén] token

[suǒ][][][fāng][miàn]Llama2 [nèi][][de][tōng][yòng][][][shì][yīng][][ér][shì][gài][niàn][dàn][lìng][][fāng][miàn][zhè][xiē][shén][][][][yòu][xiǎn][rán][shì][piān][xiàng][][yīng][][de][gài][niàn]

[yīn][][zài][][][shàng][ér][fēi][chún][cuì][de][][huì][][][shàng][yīng][][què][shí][][][bèi][shì][wéi][yáng][tuó][de][][]

[wǎng][yǒu][][zǎo][jiù][][xiàn][le]

[yǒu][wǎng][yǒu][biǎo][shì][shù][][zhí][yán][][jǐn][jǐn][shì][yáng][tuó][][liè][][běn][shàng][suǒ][yǒu] LLM [dōu][shì][zhè][yàng]


[duì][][][yīng][][wèi][][][de][rén][lái][shuō][zhè][][néng][huì][lìng][rén][jīng][][dàn][duì][][][][rén][lái][shuō][zhè][zhǒng][qīng][xiàng][xìng][shì][][jiàn][de][zhǐ][][guò][yǒu][shí][duō][yǒu][shí][shǎo]

[yǒu][shí][][huì][xiǎng] LLM [wèi][shí][me][yào][zhè][yàng][huí][][rán][hòu][][][shí][dào][zhè][][][àn][zài][yīng][][zhōng][gèng][yǒu][][]

[zhè][zài][shī][][zhōng][gèng][shì][xiǎn][ér][][jiàn][de]LLM [xiě][shī][hěn][piào][liàng][dàn][tōng][cháng][méi][yǒu][][yùn] [][guǒ][][][][fān][][chéng][yīng][][jiù][][yùn][le]

[lìng][][wèi][wǎng][yǒu][biǎo][shì][zhè][shì][][][xíng][dài][lái][de][piān][jiàn][yào][xiǎo][xīn][le]


[yīng][][][zhōng][wén][zuì][zhōng][jiāng][chéng][wéi] LLM [][shì][][shū][chū][de][zuì][jiā][][yán][ér][suí][zhe] LLM [de][yìng][yòng][fàn][wéi][yuè][lái][yuè]广[guǎng][fàn][shì][jiè][][][][yán][jiāng][gèng][jiā][biān][yuán][huà]

[][xíng][biǎo][][kōng][jiān][de][tàn][suǒ]

[dāng][qiàn][][zhú][céng][zhuǎn][huàn][shí][][men][huì][jīng][] 3 [][jiē][duàn]

1. [shū][][kōng][jiān][][xíng][xiāo][chú][fēn][][][dài][lái][de][yǐng][xiǎng]

2. [gài][niàn][kōng][jiān][qiàn][][jìn][][][][chōu][xiàng][de][gài][niàn][kōng][jiān][zhōng]

3. [shū][chū][kōng][jiān][gài][niàn][bèi][yìng][shè][huí][yuán][běn][de][biǎo][][xíng][shì]


[][xíng]

[shí][yàn][zhuān][zhù][] Llama2 [][liè][][yán][][xíng]Llama2 [][liè][][xíng][zài][duō][][yán][][liào][][shàng][jìn][xíng][xùn][liàn][][liào][][zhǔ][yào][yóu][yīng][][zhǔ][dǎo][zhàn] 89.70%

[][guò][kǎo][][dào][zǒng][][xùn][liàn][shù][][de][][xiǎo]2 [wàn]亿[][] token[]...

単語(ピンイン)

[yán][jiū][rén][yuán]research worker
[][][liè]a series ofa range ofrangesequencesuccessionstringroundbatteryconstellationtailparade
[][yán][][xíng]language model
[][men][de]ours
[hěn][][néng]like as notbe likely towelleasilypresumablyvery likelyinclinedin all probability
[yóu][][shì]in especialin particularnotablyabove allthe more somost of allto crown all
[duō][zhǒng][][yán]multilingualMultilingualismMultiple languagesmultilanguage
[luò][sāng]LausanneRothamstedEHLLausaneLaussane
[][gōng][xué][yuàn]institute of technologypolytechniccollege of science and engineeringpoly
[][yán][néng][]Language abilitylinguistic competence

中国語

Posted by revolution-x