![]() In language, unsupervised learning algorithms that rely on word prediction (like GPT-2 and BERT) have been extremely successful, achieving top performance on a wide array of language tasks. However, our results suggest that when faced with a new domain where the correct model priors are unknown, a large GPT-2 can learn excellent features without the need for domain-specific architectural design choices. As a consequence, we require significantly more compute in order to produce features competitive with those from top unsupervised convolutional nets. ![]() To highlight the potential of generative sequence modeling as a general purpose unsupervised learning algorithm, we deliberately use the same transformer architecture as GPT-2 in language. On JFT (300M images with 18K classes), achieved a result of We only show ImageNet linear probe accuracy for iGPT-XL since otherĮxperiments did not finish before we needed to transition to different Logistic regression on learned features (linear probe) As further proof, features from the model achieve state-of-the-art performance on a number of classification datasets and near state-of-the-art unsupervised accuracy on ImageNet. This is evidenced by the diverse range of coherent image samples it generates, even without the guidance of human provided labels. When we train GPT-2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D image characteristics such as object appearance and category. Transformer models like BERT and GPT-2 are domain agnostic, meaning that they can be directly applied to 1-D sequences of any form. Our work aims to understand and bridge this gap. ![]() However, the same broad class of models has not been successful in producing strong features for image classification. ![]() Recently, it has seen incredible success in language, as transformer models like BERT, GPT-2, RoBERTa, T5, and other variants have achieved top performance on a wide array of language tasks. Unsupervised and self-supervised learning, or learning without human-labeled data, is a longstanding challenge of machine learning. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |