Multisource Subnetwork Level Transfer in Deep CNNs Using Bank of Weight Filters

Abstract

The convolutional neural networks (CNNs) have become the most successful models for many pattern recognition problems in the areas of computer vision, speech, text and others. One concern about CNNs has always been their need for large amount of training data, large computational re- sources and long training time. In this regard the transfer learning is a technique that can address this concern of inefficient CNN training through reuse of pretrained networks (CNNs). In this thesis we discuss transfer learning in CNNs where the transfer is from multiple source CNNs and done at subnetwork levels. The subnetwork multisource transfer is attempted for the fi rst time and hence we begin by showing the effectiveness of such a transfer. We consider subnetworks at various granularities for the transfer. These granularities begin at a whole network-level then pro-ceed to layer-level and further fi lter-level. In order to realize this kind of transfer we create a set called bank of weight fi lters (BWF) which is a repository of the pretrained subnetworks that are used as candidates for transfer. Through extensive simulations we show that subnetwork level transfer, implemented through random selection from a BWF, is elective and is also efficient in terms of training time. We also present experimental results to show that subnetwork level transfer learning is efficient in terms of the amount of training data needed. It is seen that fi lter-level transfer learning is as effective as the whole-network-level transfer which is the conventional transfer learning used with CNNs. We then show the usefulness of the fi lter-level multisource transfer for the cases of transfer from natural to non-natural (hand drawn sketches) image datasets and transfer across different CNN architectures (having different number of layers, fi lter dimensions etc.). We also discuss transfer from CNNs trained on high-resolution images to the CNNs needed for the low-resolution im- ages and vice-versa. In the multisource transfer of prelearnt weights discussed above, the transferred weights have to be fi netuned to achieve the same accuracy as that of a CNN trained from scratch. It is certainly more bene cfiial and efficient if the fi netuning of transferred weights can be completely avoided. For this, we conceptualize we conceptualize what we call a fi lter-tree which represents the complete feature generation entity that is learnt by a CNN and propose that the a filter-tree represents a subnetwork that can be used for transfer without finetuning. Similar to BWF we create a repository of pre-learnt fllter-trees called bank of filter-trees (BFT) to realize the transfer using fi lter-trees. Through experiments we show that transfer using BFT (where the transferred weights are held fixed and are not fi netunes) has performance that is on par with training from scratch, which is the best achievable performance. The selection of the subnetworks from BWF or BFT so far for all experiments was done uniformly randomly. For the sake of completion we introduce a method that can result in informed choice of fi lters from a BFT. We propose a learnable auxilliary layer called choice layer whose learnt weights give an idea of the importance/utility of different the subnetwork (fi lter-trees here) in the BFT for the target task. We show that when the random choice from BFT does not achieve the best possible accuracy, the choice layer based method can achieve it.