| dc.description.abstract | Coarse semantic features are abstract descriptors capturing broad semantic information in an
image, including scene labels, crude contextual relationships between objects in the scene, or
even objects described using hand-drawn sketches. Derived from external sources or
pre-trained models, these features complement fine-grained representations from deep neural
networks, enhancing overall image understanding. In this thesis, we explore applications where
we integrate coarse semantic cues with deep image representations to address novel visual
analytics tasks and propose significantly improved solutions to existing challenges. In the first
part of the thesis, we present novel query-guided object localization frameworks, where
concepts like a hand-drawn sketch of an object, a ‘gloss’ delivering a crude description of the
object of interest, or a scene-graph representing objects and their coarse relationships provide
necessary cues to localize all instances of the corresponding object(s) on a complex natural
scene. Next, we utilize the information contained in edge maps via intelligent augmentation and
shuffling to improve the robustness of computer vision models against the adverse effects of
texture bias prevalent in such models. Lastly, in the realm of self-supervised learning, we use
coarse representations derived from a compact proxy model to schedule and sequence training
data for efficient training of larger models.
Locating objects in a scene through image queries is a key problem in computer vision, with
recent work highlighting the challenge of localizing both seen and unseen object categories at
test time. A possible solution is to use object images as queries; however, practical obstacles
such as copyright, privacy constraints, and difficulties in obtaining annotated data for emerging
categories pose challenges. Instead, we propose ‘sketch-guided object localization,’ which
utilizes crude hand-drawn sketches to localize corresponding objects. We employ cross-modal
attention to integrate query information into the image feature representation, facilitating the
generation of region proposals relevant to the query sketch. The region proposals are then
scored along with the sketch query for localization. Our method outperforms baselines in both
single-query and multi-query localization tasks on object detection benchmarks (MS-COCO and
PASCAL-VOC) using abstract hand-drawn sketches as queries.
Challenges in scaling sketch-guided object localization include the abstract appearance of
hand-drawn sketches, style variations, and a significant domain gap with respect to natural
images. Existing solutions using attention-based frameworks show weak alignment and
inaccurate localization because they integrate query features after extracting image features
independently. To mitigate this, we introduce a novel sketch-guided vision transformer encoder
that uses cross-attention after each transformer-based image encoder block to integrate sketch
information into the image representation. Thus, we learn query-conditioned image features for
stronger alignment with the sketch query. At the decoder’s output, cross-attention is used to
integrate sketch information into the object-level image features, thus improving their semantic alignment. The model generalizes to unseen object categories and achieves state-of-the-art
performance across both open-set and closed-set localization tasks.
The aforementioned works pioneered and optimized the use of hand-drawn sketches for
one-shot object localization. However, relying solely on crude hand-drawn sketches may
introduce ambiguity - for instance, a rough sketch of a laptop could be confused for a sofa. One
approach towards addressing this is to use a coarse linguistic definition of the category, e.g., ‘a
small portable computer small enough to use in your lap’, to complement the sketch query. We,
therefore, propose a multimodal integration of sketches with linguistic category definitions,
called ‘gloss’, for a comprehensive representation of visual and semantic cues. We use
cross-modal attention to integrate information from the multi-modal queries into the image
representation, thus generating region proposals relevant to the queries. Further, we propose a
novel orthogonal projection-based proposal scoring technique that evaluates each proposal with
respect to the multi-modal queries. Experiments on the MS-COCO dataset using ’Quick, Draw!’
sketches and ’WordNet’ glosses as queries demonstrate superior performance over related
baselines for both seen and unseen categories.
In natural scenes, single-query object localization is uncertain due to factors like
underrepresentation, occlusion, or unavailability of suitable training data. Scenes with multiple
objects exhibit visual relationships, offering strong contextual cues for improved grounding.
Scene graphs efficiently represent objects and the coarse semantic relationships (e.g., laptop
‘on’ table) between them. In this work, we study the problem of grounding scene graphs on
natural images to improve object localization. To this end, we propose a novel graph neural
network-based approach referred to as Visio-Lingual Message PAssing Graph neural Network
(VL-MPAG Net). We first construct a directed graph with object proposals as nodes and edges
representing plausible relations. Next, we employ a three-step framework involving inter-graph
and intra-graph message passing. Through inter-graph message passing, the model integrates
scene-graph information into the proposal representations, facilitating the learning of
query-conditioned proposal representations. Subsequently, intra-graph message passing refines
them to learn context-dependent representations of these proposals as well as that of the query
objects. These refined query representations are used to score the proposals for object
localization, outperforming baselines on public benchmark datasets.
Deep vision models often exhibit overreliance on texture features, resulting in poor
generalization. To address this texture bias, we propose a lightweight adversarial augmentation
technique called ELeaS that explicitly incentivizes the network to learn holistic shapes for
accurate prediction in an object classification setting. Our augmentations superpose coarser
descriptors, namely edgemaps, from one image onto another image with shuffled patches using
a randomly determined mixing proportion. To be able to classify these augmented images with
the label of the edgemap images, the model needs to not only detect and focus on edges but
also distinguish between relevant and spurious edges. We show that our augmentations
significantly improve classification accuracy and robustness measures on a range of datasets
and neural architectures. Analysis using multiple probe datasets shows substantially increased
shape sensitivity in our trained models, explaining these observed improvements.
Self-supervised learning (SSL) is vital for acquiring high-quality representations from unlabeled
image collections, but the growing dataset sizes increase the demand for computational
resources in training SSL models. To address this, we propose ‘DYSCOF’, a Dynamic data
Selection method. DYSCOF scores and prioritizes essential samples using a Coarse-toFine
schedule for optimized data selection. It employs a small proxy model pre-trained via contrastive
learning to identify a data subset based on the score difference between the representations
learned by the larger target model and the coarse representations obtained from this proxy. The
selected subset gets iteratively updated in a coarse-to-fine schedule that initially selects a larger
data fraction and gradually reduces the proportion of selected data as training progresses. To
further enhance efficiency, we introduce a distillation loss that leverages coarse representations
obtained from the proxy model to guide the target model’s learning. Validated on public
benchmark datasets, our method achieves a large reduction in computational load on all
benchmark datasets, enhancing SSL training efficiency while maintaining classification
performance. | en_US |
