Diese Seite listet die Primärliteratur, auf der die Darstellung beruht.
Kapitel 1 und 2
Warren S. McCulloch, Walter Pitts (1943)
A Logical Calculus of the Ideas Immanent in Nervous Activity
Bulletin of Mathematical Biophysics 5, 115–133
DOI 10.1007/BF02478259Frank Rosenblatt (1958)
The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain
Psychological Review 65, 386–408
DOI 10.1037/h0042519Marvin Minsky, Seymour Papert (1969)
Perceptrons: An Introduction to Computational Geometry
MIT Press
George Cybenko (1989)
Approximation by Superpositions of a Sigmoidal Function
Mathematics of Control, Signals and Systems 2, 303–314
DOI 10.1007/BF02551274Kurt Hornik (1991)
Approximation Capabilities of Multilayer Feedforward Networks
Neural Networks 4, 251–257
DOI 10.1016/0893-6080(91)90009-TMoshe Leshno, Vladimir Ya. Lin, Allan Pinkus, Shimon Schocken (1993)
Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any Function
Neural Networks 6, 861–867
DOI 10.1016/S0893-6080(05)80131-5Kapitel 3 bis 5
Herbert Robbins, Sutton Monro (1951)
A Stochastic Approximation Method
Annals of Mathematical Statistics 22, 400–407
DOI 10.1214/aoms/1177729586Boris T. Polyak (1964)
Some Methods of Speeding up the Convergence of Iteration Methods
USSR Computational Mathematics and Mathematical Physics 4, 1–17
DOI 10.1016/0041-5553(64)90137-5Diederik P. Kingma, Jimmy Ba (2015)
Adam: A Method for Stochastic Optimization
International Conference on Learning Representations
arXiv 1412.6980Kapitel 6 und 7
Seppo Linnainmaa (1970)
Algoritmin kumulatiivinen pyöristysvirhe yksittäisten pyöristysvirheiden Taylor-kehitelmänä
Pro-gradu-Arbeit, Universität Helsinki — deutsch etwa: Der kumulative Rundungsfehler eines Algorithmus als Taylor-Entwicklung der einzelnen Rundungsfehler
Paul J. Werbos (1974)
Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences
Dissertation, Harvard University
David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams (1986)
Learning Representations by Back-Propagating Errors
Nature 323, 533–536
DOI 10.1038/323533a0Sepp Hochreiter (1991)
Untersuchungen zu dynamischen neuronalen Netzen
Diplomarbeit, Technische Universität München
Yoshua Bengio, Patrice Simard, Paolo Frasconi (1994)
Learning Long-Term Dependencies with Gradient Descent Is Difficult
IEEE Transactions on Neural Networks 5, 157–166
DOI 10.1109/72.279181Xavier Glorot, Yoshua Bengio (2010)
Understanding the Difficulty of Training Deep Feedforward Neural Networks
Proceedings of AISTATS 9, 249–256
Sergey Ioffe, Christian Szegedy (2015)
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
arXiv:1502.03167
arXiv 1502.03167Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (2016)
Deep Residual Learning for Image Recognition
Proceedings of CVPR, 770–778
arXiv 1512.03385Kapitel 8
John R. Firth (1957)
A Synopsis of Linguistic Theory 1930–1955
Studies in Linguistic Analysis, Blackwell, 1–32
Tomáš Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean (2013)
Efficient Estimation of Word Representations in Vector Space
arXiv:1301.3781
arXiv 1301.3781Omer Levy, Yoav Goldberg (2014)
Neural Word Embedding as Implicit Matrix Factorization
Advances in Neural Information Processing Systems 27
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Kalai (2016)
Man Is to Computer Programmer as Woman Is to Homemaker? Debiasing Word Embeddings
arXiv:1607.06520
arXiv 1607.06520Kapitel 9
Kunihiko Fukushima (1980)
Neocognitron: A Self-Organizing Neural Network Model
Biological Cybernetics 36, 193–202
DOI 10.1007/BF00344251Yann LeCun, Bernhard Boser, John S. Denker et al. (1989)
Backpropagation Applied to Handwritten Zip Code Recognition
Neural Computation 1, 541–551
DOI 10.1162/neco.1989.1.4.541Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton (2012)
ImageNet Classification with Deep Convolutional Neural Networks
Advances in Neural Information Processing Systems 25
Kapitel 10 und 11
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio (2015)
Neural Machine Translation by Jointly Learning to Align and Translate
arXiv:1409.0473
arXiv 1409.0473Ashish Vaswani, Noam Shazeer, Niki Parmar et al. (2017)
Attention Is All You Need
Advances in Neural Information Processing Systems 30
arXiv 1706.03762Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton (2016)
Layer Normalization
arXiv:1607.06450
arXiv 1607.06450Nelson Elhage, Neel Nanda, Catherine Olsson et al. (2021)
A Mathematical Framework for Transformer Circuits
Transformer Circuits Thread, Anthropic
Kapitel 12
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky et al. (2014)
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Journal of Machine Learning Research 15, 1929–1958
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals (2017)
Understanding Deep Learning Requires Rethinking Generalization
arXiv:1611.03530
arXiv 1611.03530Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal (2019)
Reconciling Modern Machine-Learning Practice and the Classical Bias-Variance Trade-Off
Proceedings of the National Academy of Sciences 116, 15849–15854
DOI 10.1073/pnas.1903070116Kapitel 13
Claude E. Shannon (1948)
A Mathematical Theory of Communication
Bell System Technical Journal 27, 379–423 und 623–656
DOI 10.1002/j.1538-7305.1948.tb01338.xJared Kaplan, Sam McCandlish, Tom Henighan et al. (2020)
Scaling Laws for Neural Language Models
arXiv:2001.08361
arXiv 2001.08361Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch et al. (2022)
Training Compute-Optimal Large Language Models
arXiv:2203.15556
arXiv 2203.15556Rylan Schaeffer, Brando Miranda, Sanmi Koyejo (2023)
Are Emergent Abilities of Large Language Models a Mirage?
arXiv:2304.15004
arXiv 2304.15004Kapitel 14
Rico Sennrich, Barry Haddow, Alexandra Birch (2016)
Neural Machine Translation of Rare Words with Subword Units
arXiv:1508.07909
arXiv 1508.07909Long Ouyang, Jeff Wu, Xu Jiang et al. (2022)
Training Language Models to Follow Instructions with Human Feedback
arXiv:2203.02155
arXiv 2203.02155Rafael Rafailov, Archit Sharma, Eric Mitchell et al. (2023)
Direct Preference Optimization: Your Language Model Is Secretly a Reward Model
arXiv:2305.18290
arXiv 2305.18290Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi (2020)
The Curious Case of Neural Text Degeneration
arXiv:1904.09751
arXiv 1904.09751Die Darstellung fasst den Inhalt dieser Arbeiten zusammen und gibt sie nicht wörtlich wieder. Kapitel 14 ist ausdrücklich als Momentaufnahme gekennzeichnet; die dort beschriebenen Verfahren und Größenordnungen ändern sich schnell.