Learning algorithms for feed-forward neural networks: design, combination and analysis
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Abschlussarbeit Buch |
| Sprache: | English |
| Veröffentlicht: |
Düsseldorf
VDI Verl.
1996
|
| Ausgabe: | Als Ms. gedr. |
| Schriftenreihe: | Verein Deutscher Ingenieure: [Fortschrittberichte VDI / 10]
435 |
| Schlagworte: | |
| Online-Zugang: | Inhaltsverzeichnis |
| Beschreibung: | Zugl.: Berlin, Univ., Diss. - Zsfassung in dt. Sprache |
| Beschreibung: | IX, 206 S. graph. Darst. |
| ISBN: | 3183435101 |
Internformat
MARC
| LEADER | 00000nam a2200000 cb4500 | ||
|---|---|---|---|
| 001 | BV010925435 | ||
| 003 | DE-604 | ||
| 005 | 19961001 | ||
| 007 | t | ||
| 008 | 960903s1996 d||| mm|| 00||| eng d | ||
| 020 | |a 3183435101 |9 3-18-343510-1 | ||
| 035 | |a (OCoLC)45669192 | ||
| 035 | |a (DE-599)BVBBV010925435 | ||
| 040 | |a DE-604 |b ger |e rakddb | ||
| 041 | 0 | |a eng | |
| 049 | |a DE-91 |a DE-210 |a DE-83 |a DE-188 | ||
| 084 | |a ST 301 |0 (DE-625)143651: |2 rvk | ||
| 084 | |a DAT 717d |2 stub | ||
| 084 | |a DAT 708d |2 stub | ||
| 100 | 1 | |a Pfister, Marcus |e Verfasser |4 aut | |
| 245 | 1 | 0 | |a Learning algorithms for feed-forward neural networks |b design, combination and analysis |c Marcus Pfister |
| 250 | |a Als Ms. gedr. | ||
| 264 | 1 | |a Düsseldorf |b VDI Verl. |c 1996 | |
| 300 | |a IX, 206 S. |b graph. Darst. | ||
| 336 | |b txt |2 rdacontent | ||
| 337 | |b n |2 rdamedia | ||
| 338 | |b nc |2 rdacarrier | ||
| 490 | 1 | |a Verein Deutscher Ingenieure: [Fortschrittberichte VDI / 10] |v 435 | |
| 500 | |a Zugl.: Berlin, Univ., Diss. - Zsfassung in dt. Sprache | ||
| 502 | |a Zugl.: Berlin, Freie Univ., Diss., 1995 | ||
| 650 | 0 | 7 | |a SIMD |0 (DE-588)4329642-7 |2 gnd |9 rswk-swf |
| 650 | 0 | 7 | |a Backpropagation-Algorithmus |0 (DE-588)4354627-4 |2 gnd |9 rswk-swf |
| 650 | 0 | 7 | |a Implementierung |g Informatik |0 (DE-588)4026663-1 |2 gnd |9 rswk-swf |
| 650 | 0 | 7 | |a Neuronales Netz |0 (DE-588)4226127-2 |2 gnd |9 rswk-swf |
| 650 | 0 | 7 | |a Maschinelles Lernen |0 (DE-588)4193754-5 |2 gnd |9 rswk-swf |
| 655 | 7 | |0 (DE-588)4113937-9 |a Hochschulschrift |2 gnd-content | |
| 689 | 0 | 0 | |a Maschinelles Lernen |0 (DE-588)4193754-5 |D s |
| 689 | 0 | 1 | |a Neuronales Netz |0 (DE-588)4226127-2 |D s |
| 689 | 0 | |5 DE-604 | |
| 689 | 1 | 0 | |a Neuronales Netz |0 (DE-588)4226127-2 |D s |
| 689 | 1 | 1 | |a Maschinelles Lernen |0 (DE-588)4193754-5 |D s |
| 689 | 1 | 2 | |a Backpropagation-Algorithmus |0 (DE-588)4354627-4 |D s |
| 689 | 1 | 3 | |a SIMD |0 (DE-588)4329642-7 |D s |
| 689 | 1 | 4 | |a Implementierung |g Informatik |0 (DE-588)4026663-1 |D s |
| 689 | 1 | |5 DE-188 | |
| 810 | 2 | |a 10] |t Verein Deutscher Ingenieure: [Fortschrittberichte VDI |v 435 |w (DE-604)BV000897204 |9 435 | |
| 856 | 4 | 2 | |m HBZ Datenaustausch |q application/pdf |u http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=007308499&sequence=000002&line_number=0001&func_code=DB_RECORDS&service_type=MEDIA |3 Inhaltsverzeichnis |
| 999 | |a oai:aleph.bib-bvb.de:BVB01-007308499 | ||
Datensatz im Suchindex
| _version_ | 1804125416415297536 |
|---|---|
| adam_text | Titel: Learning algorithms for feed-forward neural networks
Autor: Pfister, Marcus
Jahr: 1996
FORTSCHRITTBERICHTE VM Dipl.-Math. Marcus Pfister, Herzogenaurach Learning Algorithms for Feed-forward Neural Networks - Design, Combination and Analysis Reihe 10 : Informatik/ Kommunikationstechnik Nr. 435 VM VERLAG
Contents 1 Introduction 1 1.1 Neural Networks and Large Scale Applications................... 1 1.2 The Problem of Learning in Neural Networks ................... 2 1.3 Objective and Structure of this Thesis ....................... 3 2 Neural Networks 5 2.1 Biological Motivation................................. 6 2.1.1 Neurons and the Brain............................ 6 2.1.2 Perceptrons and Artificial Neural Networks................. 6 2.2 Feed-forward Neural Networks............................ 9 2.2.1 Neural Networks as Function-Approximators................ 9 2.2.2 Multilayered Neural Networks........................ 12 2.2.3 Layered Neural Networks and Kolmogorov’s Theorem........... 13 2.3 Definitions and Notations .............................. 15 2.4 Learning in Neural Networks............................. 16 2.4.1 Estimating the Output Error in Feed-forward Networks.......... 17 2.4.2 The Backpropagation Algorithm....................... 18 2.4.3 Second Order Backpropagation....................... 23 3 Learning Algorithms and Special Hardware 26 3.1 Parallel Hardware for Neural Networks....................... 27 3.1.1 The SPERT Node............................... 27 3.1.2 The SYNAPSE Computer.......................... 28 V
3.1.3 The CNAPS Neurocomputer......................... 30 3.1.4 The Local Network: Also a Supercomputer................. 32 3.2 Mapping Backpropagation Networks to SIMD Arrays............... 32 3.2.1 The Feed-forward Computation....................... 33 3.2.2 The Backpropagation Computation..................... 34 3.2.3 The SIMD Computations: A small Example................ 35 3.2.4 The Second Order Backpropagation Computation............. 37 3.3 Performing Special Computations.......................... 37 3.3.1 The Evaluation of the Sigmoid Function.................. 38 3.3.2 Dividing Integers............................... 38 3.4 Implementation Details................................ 48 3.4.1 Integer Scaling................................ 48 3.4.2 Preparation of the Input Data........................ 49 3.4.3 The Sigmoid Lookup Table ......................... 51 3.4.4 Mapping Feed-forward Networks to the CNAPS.............. 51 3.4.5 Maximum Network Sizes for the CNAPS.................. 52 3.4.6 Speed-up of the CNAPS over Sequential Computers............ 53 3.5 Conclusions...................................... 54 4 EVist Learning Algorithms for Neural Networks 55 4.1 Why Backpropagation-Accelerations?........................ 56 4.2 Backpropagation-Variations: A Classification ................... 56 4.3 Variations of the Standard Algorithm........................ 59 4.3.1 Initializing the Network Weights....................... 59 4.3.2 When should the learning Process be stopped?............... 60 4.3.3 Improving and Estimating Generalization Errors.............. 61 4.3.4 More than one Pattern to learn: Batch vs. On-Line............ 64 4.3.5 Introduction of a Momentum Term..................... 65 4.3.6 Using bipolar- instead of binary Vectors .................. 66 4.3.7 Handling flat Spots of the Error Function.................. 67 VI
4.3.8 Escaping local Minima............................ 69 4.3.9 Handling Redundancy in the Training Set................. 70 4.3.10 Decorrelation of the Training Set...................... 71 4.4 Adaptive Step Algorithms.............................. 77 4.4.1 The Adaptation of one global Learning Rate................ 77 4.4.2 Local Adaptation of individual Learning Rates............... 82 4.5 Second Order Methods................................ 86 4.5.1 Quasi-Newton Methods............................ 88 4.5.2 Secant Methods................................ 89 4.6 The Method of Conjugate Gradients ........................ 94 4.6.1 Original Motivation of the Conjugate Gradient Method.......... 94 4.6.2 Solving nonquadratic Problems with the CG-Method........... 96 4.6.3 The Method of Fletcher and Reeves..................... 96 4.6.4 The Scaled Conjugate Gradient (SCG) Algorithm............. 97 4.7 Alternative Approaches................................100 4.7.1 The Cascade Correlation Algorithm.....................100 4.7.2 Relaxation Methods .............................102 4.7.3 Nonlinear Least Squares Methods......................103 4.8 Evaluation of Backpropagation Variations .....................104 4.8.1 Evaluation of Adaptive Step and Second Order Methods.........105 4.8.2 Evaluation of the Standard Variations ...................107 4.9 Conclusions......................................110 5 Dynamic Combination of Learning Strategies 112 5.1 Critique of the Common Backpropagation Accelerations..............113 5.2 Hybrid Learning Algorithms.............................116 5.2.1 QRprop....................................117 5.2.2 DERprop...................................121 5.3 Related Work.....................................124 5.3.1 Muted (Quasi-) Newton Methods......................125 VII
5.3.2 Combination of Steepest Descent and Newton’s Method..........126 5.3.3 Extended Quickprop . . » ..........................127 5.3.4 Finding the Global Minimum of the Error Function............127 5.4 Conclusions............. 128 6 Parallel Implementations 129 6.1 Comparing Learning Algorithms...........................130 6.2 Benchmarks for Learning Algorithms........................131 6.2.1 Artificial Data Sets..............................131 6.2.2 Problems Arising from Real World Applications..............132 6.3 Details of the Algorithms compared.........................136 6.4 Integer- vs. Floating-point Arithmetic.......................139 6.5 Runtime Comparison.................................143 6.5.1 Effect of the chosen Standard Variations..................144 6.5.2 Results for the Sonar Signals Discrimination Problem...........147 6.5.3 Results for the English Vowel Recognition Problem............147 6.5.4 Results for NETtalk.............................149 6.5.5 Results for the Protein Recognition Problem................150 6.5.6 Results for the Handprinted Digits Recognition Problem.........151 6.6 Discussion of the Results...............................151 6.6.1 The Computational Complexity of the Algorithms.............157 6.7 Conclusions......................................159 6.8 Results of the Runtime Comparison (Tables)....................161 7 Convergence Properties of Hybrid Algorithms 164 7.1 An idealized Model of the Error Function......................164 7.2 Convergence Properties of Rpropm.........................167 7.3 Convergence Properties of QRpropm........................171 7.4 Convergence Properties of DERpropm.......................178 7.5 An Al-Dimensional Convergence Proof.......................182 7.6 Conclusions......................................188 VIII
S Conclusions 189 8.1 Outlook........................................191 A Description of the Sequential Experiments 192 A.l The Benchmarks...................................192 A.2 Tables for the Sequential Experiments .......................194 IX
|
| any_adam_object | 1 |
| author | Pfister, Marcus |
| author_facet | Pfister, Marcus |
| author_role | aut |
| author_sort | Pfister, Marcus |
| author_variant | m p mp |
| building | Verbundindex |
| bvnumber | BV010925435 |
| classification_rvk | ST 301 |
| classification_tum | DAT 717d DAT 708d |
| ctrlnum | (OCoLC)45669192 (DE-599)BVBBV010925435 |
| discipline | Informatik |
| edition | Als Ms. gedr. |
| format | Thesis Book |
| fullrecord | <?xml version="1.0" encoding="UTF-8"?><collection xmlns="http://www.loc.gov/MARC21/slim"><record><leader>02330nam a2200541 cb4500</leader><controlfield tag="001">BV010925435</controlfield><controlfield tag="003">DE-604</controlfield><controlfield tag="005">19961001 </controlfield><controlfield tag="007">t</controlfield><controlfield tag="008">960903s1996 d||| mm|| 00||| eng d</controlfield><datafield tag="020" ind1=" " ind2=" "><subfield code="a">3183435101</subfield><subfield code="9">3-18-343510-1</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(OCoLC)45669192</subfield></datafield><datafield tag="035" ind1=" " ind2=" "><subfield code="a">(DE-599)BVBBV010925435</subfield></datafield><datafield tag="040" ind1=" " ind2=" "><subfield code="a">DE-604</subfield><subfield code="b">ger</subfield><subfield code="e">rakddb</subfield></datafield><datafield tag="041" ind1="0" ind2=" "><subfield code="a">eng</subfield></datafield><datafield tag="049" ind1=" " ind2=" "><subfield code="a">DE-91</subfield><subfield code="a">DE-210</subfield><subfield code="a">DE-83</subfield><subfield code="a">DE-188</subfield></datafield><datafield tag="084" ind1=" " ind2=" "><subfield code="a">ST 301</subfield><subfield code="0">(DE-625)143651:</subfield><subfield code="2">rvk</subfield></datafield><datafield tag="084" ind1=" " ind2=" "><subfield code="a">DAT 717d</subfield><subfield code="2">stub</subfield></datafield><datafield tag="084" ind1=" " ind2=" "><subfield code="a">DAT 708d</subfield><subfield code="2">stub</subfield></datafield><datafield tag="100" ind1="1" ind2=" "><subfield code="a">Pfister, Marcus</subfield><subfield code="e">Verfasser</subfield><subfield code="4">aut</subfield></datafield><datafield tag="245" ind1="1" ind2="0"><subfield code="a">Learning algorithms for feed-forward neural networks</subfield><subfield code="b">design, combination and analysis</subfield><subfield code="c">Marcus Pfister</subfield></datafield><datafield tag="250" ind1=" " ind2=" "><subfield code="a">Als Ms. gedr.</subfield></datafield><datafield tag="264" ind1=" " ind2="1"><subfield code="a">Düsseldorf</subfield><subfield code="b">VDI Verl.</subfield><subfield code="c">1996</subfield></datafield><datafield tag="300" ind1=" " ind2=" "><subfield code="a">IX, 206 S.</subfield><subfield code="b">graph. Darst.</subfield></datafield><datafield tag="336" ind1=" " ind2=" "><subfield code="b">txt</subfield><subfield code="2">rdacontent</subfield></datafield><datafield tag="337" ind1=" " ind2=" "><subfield code="b">n</subfield><subfield code="2">rdamedia</subfield></datafield><datafield tag="338" ind1=" " ind2=" "><subfield code="b">nc</subfield><subfield code="2">rdacarrier</subfield></datafield><datafield tag="490" ind1="1" ind2=" "><subfield code="a">Verein Deutscher Ingenieure: [Fortschrittberichte VDI / 10]</subfield><subfield code="v">435</subfield></datafield><datafield tag="500" ind1=" " ind2=" "><subfield code="a">Zugl.: Berlin, Univ., Diss. - Zsfassung in dt. Sprache</subfield></datafield><datafield tag="502" ind1=" " ind2=" "><subfield code="a">Zugl.: Berlin, Freie Univ., Diss., 1995</subfield></datafield><datafield tag="650" ind1="0" ind2="7"><subfield code="a">SIMD</subfield><subfield code="0">(DE-588)4329642-7</subfield><subfield code="2">gnd</subfield><subfield code="9">rswk-swf</subfield></datafield><datafield tag="650" ind1="0" ind2="7"><subfield code="a">Backpropagation-Algorithmus</subfield><subfield code="0">(DE-588)4354627-4</subfield><subfield code="2">gnd</subfield><subfield code="9">rswk-swf</subfield></datafield><datafield tag="650" ind1="0" ind2="7"><subfield code="a">Implementierung</subfield><subfield code="g">Informatik</subfield><subfield code="0">(DE-588)4026663-1</subfield><subfield code="2">gnd</subfield><subfield code="9">rswk-swf</subfield></datafield><datafield tag="650" ind1="0" ind2="7"><subfield code="a">Neuronales Netz</subfield><subfield code="0">(DE-588)4226127-2</subfield><subfield code="2">gnd</subfield><subfield code="9">rswk-swf</subfield></datafield><datafield tag="650" ind1="0" ind2="7"><subfield code="a">Maschinelles Lernen</subfield><subfield code="0">(DE-588)4193754-5</subfield><subfield code="2">gnd</subfield><subfield code="9">rswk-swf</subfield></datafield><datafield tag="655" ind1=" " ind2="7"><subfield code="0">(DE-588)4113937-9</subfield><subfield code="a">Hochschulschrift</subfield><subfield code="2">gnd-content</subfield></datafield><datafield tag="689" ind1="0" ind2="0"><subfield code="a">Maschinelles Lernen</subfield><subfield code="0">(DE-588)4193754-5</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="0" ind2="1"><subfield code="a">Neuronales Netz</subfield><subfield code="0">(DE-588)4226127-2</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="0" ind2=" "><subfield code="5">DE-604</subfield></datafield><datafield tag="689" ind1="1" ind2="0"><subfield code="a">Neuronales Netz</subfield><subfield code="0">(DE-588)4226127-2</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="1" ind2="1"><subfield code="a">Maschinelles Lernen</subfield><subfield code="0">(DE-588)4193754-5</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="1" ind2="2"><subfield code="a">Backpropagation-Algorithmus</subfield><subfield code="0">(DE-588)4354627-4</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="1" ind2="3"><subfield code="a">SIMD</subfield><subfield code="0">(DE-588)4329642-7</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="1" ind2="4"><subfield code="a">Implementierung</subfield><subfield code="g">Informatik</subfield><subfield code="0">(DE-588)4026663-1</subfield><subfield code="D">s</subfield></datafield><datafield tag="689" ind1="1" ind2=" "><subfield code="5">DE-188</subfield></datafield><datafield tag="810" ind1="2" ind2=" "><subfield code="a">10]</subfield><subfield code="t">Verein Deutscher Ingenieure: [Fortschrittberichte VDI</subfield><subfield code="v">435</subfield><subfield code="w">(DE-604)BV000897204</subfield><subfield code="9">435</subfield></datafield><datafield tag="856" ind1="4" ind2="2"><subfield code="m">HBZ Datenaustausch</subfield><subfield code="q">application/pdf</subfield><subfield code="u">http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=007308499&sequence=000002&line_number=0001&func_code=DB_RECORDS&service_type=MEDIA</subfield><subfield code="3">Inhaltsverzeichnis</subfield></datafield><datafield tag="999" ind1=" " ind2=" "><subfield code="a">oai:aleph.bib-bvb.de:BVB01-007308499</subfield></datafield></record></collection> |
| genre | (DE-588)4113937-9 Hochschulschrift gnd-content |
| genre_facet | Hochschulschrift |
| id | DE-604.BV010925435 |
| illustrated | Illustrated |
| indexdate | 2024-07-09T18:01:13Z |
| institution | BVB |
| isbn | 3183435101 |
| language | English |
| oai_aleph_id | oai:aleph.bib-bvb.de:BVB01-007308499 |
| oclc_num | 45669192 |
| open_access_boolean | |
| owner | DE-91 DE-BY-TUM DE-210 DE-83 DE-188 |
| owner_facet | DE-91 DE-BY-TUM DE-210 DE-83 DE-188 |
| physical | IX, 206 S. graph. Darst. |
| publishDate | 1996 |
| publishDateSearch | 1996 |
| publishDateSort | 1996 |
| publisher | VDI Verl. |
| record_format | marc |
| series2 | Verein Deutscher Ingenieure: [Fortschrittberichte VDI / 10] |
| spelling | Pfister, Marcus Verfasser aut Learning algorithms for feed-forward neural networks design, combination and analysis Marcus Pfister Als Ms. gedr. Düsseldorf VDI Verl. 1996 IX, 206 S. graph. Darst. txt rdacontent n rdamedia nc rdacarrier Verein Deutscher Ingenieure: [Fortschrittberichte VDI / 10] 435 Zugl.: Berlin, Univ., Diss. - Zsfassung in dt. Sprache Zugl.: Berlin, Freie Univ., Diss., 1995 SIMD (DE-588)4329642-7 gnd rswk-swf Backpropagation-Algorithmus (DE-588)4354627-4 gnd rswk-swf Implementierung Informatik (DE-588)4026663-1 gnd rswk-swf Neuronales Netz (DE-588)4226127-2 gnd rswk-swf Maschinelles Lernen (DE-588)4193754-5 gnd rswk-swf (DE-588)4113937-9 Hochschulschrift gnd-content Maschinelles Lernen (DE-588)4193754-5 s Neuronales Netz (DE-588)4226127-2 s DE-604 Backpropagation-Algorithmus (DE-588)4354627-4 s SIMD (DE-588)4329642-7 s Implementierung Informatik (DE-588)4026663-1 s DE-188 10] Verein Deutscher Ingenieure: [Fortschrittberichte VDI 435 (DE-604)BV000897204 435 HBZ Datenaustausch application/pdf http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=007308499&sequence=000002&line_number=0001&func_code=DB_RECORDS&service_type=MEDIA Inhaltsverzeichnis |
| spellingShingle | Pfister, Marcus Learning algorithms for feed-forward neural networks design, combination and analysis SIMD (DE-588)4329642-7 gnd Backpropagation-Algorithmus (DE-588)4354627-4 gnd Implementierung Informatik (DE-588)4026663-1 gnd Neuronales Netz (DE-588)4226127-2 gnd Maschinelles Lernen (DE-588)4193754-5 gnd |
| subject_GND | (DE-588)4329642-7 (DE-588)4354627-4 (DE-588)4026663-1 (DE-588)4226127-2 (DE-588)4193754-5 (DE-588)4113937-9 |
| title | Learning algorithms for feed-forward neural networks design, combination and analysis |
| title_auth | Learning algorithms for feed-forward neural networks design, combination and analysis |
| title_exact_search | Learning algorithms for feed-forward neural networks design, combination and analysis |
| title_full | Learning algorithms for feed-forward neural networks design, combination and analysis Marcus Pfister |
| title_fullStr | Learning algorithms for feed-forward neural networks design, combination and analysis Marcus Pfister |
| title_full_unstemmed | Learning algorithms for feed-forward neural networks design, combination and analysis Marcus Pfister |
| title_short | Learning algorithms for feed-forward neural networks |
| title_sort | learning algorithms for feed forward neural networks design combination and analysis |
| title_sub | design, combination and analysis |
| topic | SIMD (DE-588)4329642-7 gnd Backpropagation-Algorithmus (DE-588)4354627-4 gnd Implementierung Informatik (DE-588)4026663-1 gnd Neuronales Netz (DE-588)4226127-2 gnd Maschinelles Lernen (DE-588)4193754-5 gnd |
| topic_facet | SIMD Backpropagation-Algorithmus Implementierung Informatik Neuronales Netz Maschinelles Lernen Hochschulschrift |
| url | http://bvbr.bib-bvb.de:8991/F?func=service&doc_library=BVB01&local_base=BVB01&doc_number=007308499&sequence=000002&line_number=0001&func_code=DB_RECORDS&service_type=MEDIA |
| volume_link | (DE-604)BV000897204 |
| work_keys_str_mv | AT pfistermarcus learningalgorithmsforfeedforwardneuralnetworksdesigncombinationandanalysis |