dc.description.abstract | Topological insulators host exotic metallic states on their surfaces even when the bulk is insulating. The distinctive conducting surfaces of topological insulators present a novel domain for investigating the physics of quasiparticles such as Dirac or Majorana fermions and demonstrate promising potential for spintronics. However, achieving surface-dominated properties poses a substantive challenge due to the prevalent lack of bulk-insulating properties in known topological insulator materials, impeding the exploration of surface-state transport properties. With its superior spatial resolution compared to other analytical tools, Scanning Tunneling Microscopy (STM) and Scanning Tunneling Spectroscopy (STS), supplemented by advanced techniques such as conductance mapping, can contribute to the identification of topological surface states. Here we present the first insight into the experimental observation of co-existing dual topological surface states in BiTe, BiSe, and Sb-doped BiSe through STS and electrical transport experiments. The non-trivial surface states safeguarded by timereversal symmetry and crystalline symmetry in these materials open up a unique opportunity to probe the interaction and co-existence of the two surface states. Dual topological insulators exhibit two distinct topological surface states protected by the bulk symmetries. In the context of dual topological insulators, Scanning Tunneling Spectroscopic and electrical transport measurements become notably intricate due to the coexistence of two distinct surface states on different facets of the materials. In the case of BiTe, BiSe, and its Sb-doped compound, a weak topological surface state, safeguarded by time-reversal symmetry, coexists with a topological crystalline insulating state protected by the mirror symmetry of the crystal. The residence of these two topological phases on distinct crystal surfaces, perpendicular to each other, concurrently contributes to the surface transport properties. Additionally, the inherently bulk-conducting nature of the material complicates the isolation of surface state characteristics. The third chapter in this thesis deals with scanning probe measurements on standard samples, Si(111), and HOPG. These samples are ideal for calibrating the STM measurements due to their proper crystal characterizations and well-studied features. The lattice parameters extracted from STM measurements show good agreement with theoretical values, and the measurements have yielded excellent results beyond our expectations. When imaging Si(111), we observed a 7 x 7 reconstruction resolving the honeycomb lattice structure. The high quality of the sample is indicated by achieving atomic resolution and observing atomic steps. The tunneling spectra taken on Si(111) show its metallic nature. Defect states are also observed in both image and spectroscopy modes.HOPG characterization using STM yielded excellent results, capturing various features of graphite surfaces. We achieved high-resolution imaging of triangular and hexagonal honeycomb structures and observed atomic steps with the height of inter-layer separation or its multiples. The 300K and 9K measurements were consistent, and lattice parameters were determined simultaneously from topographic images and FFT patterns, closely matching standard values. Additionally, Moire patterns and periodic super-lattices were observed on the surfaces of HOPG, along with well-characterized 1D line defects exhibiting micrometer-scale extensions. The dI/dV spectra and maps provided direct experimental evidence of flat bands associated with the line defects. BiTe, a dual topological insulator with a superlattice structure, possesses both weak and crystalline topological phases at different facets. In Chapter 4, using scanning tunneling microscopy and spectroscopy, we detected a Dirac point within the band gap at the top surface, which stems from topological crystalline states, and the protection of these states by the mirror symmetry is verified by the suppression of Local Density of States (LDOS) in the vicinity of non-Bi bilayer step edges. Moreover, we observed increased differential conductance only at the step-edges that contain Bismuth bilayers, corresponding to one-dimensional conducting channels of weak topological surface states. Our experimental confirmation of the coexistence of two different topological states at different facets makes BiTe an exceptional candidate with the possibility of tuning the topological nature by controlled symmetry breaking, which would selectively destroy certain surface states. These Bismuth bilayers with 1D channels may have exotic properties, including forming Majorana modes if one can induce superconductivity into this, which is important for fault-tolerant quantum computing. From the STM/STS investigations conducted on BiSe, we have substantiated the presence of coexisting TCI (topological crystalline insulator) and WTI (weak topological insulator) surface states. Notably, the TCI Dirac surface states are found across all terraces, independent of their composition and termination, while the WTI edge modes manifest selectively only on step edges featuring a Bi bilayer termination. Significantly, these states persist in their disentangled state owing to the separation in real and momentum space which facilitates their unique interaction and preservation. In particular, the spatial separation arises from the distinct facets accommodating the TCI and WTI surface states, while the momentum separation originates from the localization of the 2D TI Dirac node to the time-reversal invariant momentum. Our empirical findings have unveiled the intricate interplay of these protective mechanisms, asserting the presence of states on terraces and step edges where their spatial separation is minimal. Our findings suggest the plausibility of engineering tailored samples featuring specific terraces, step edges, and symmetry-breaking perturbations to govern the attributes of the topological surface states they accommodate. The final chapter explores the quantum transport properties of BiSe and Sb-BiSe.We observed a prominent weak anti-localization (WAL) effect in both BiSe and 11% Sb-doped BiSe under both perpendicular and parallel magnetic fields due to contributions from dual topological surface states. The two states contribute in different ways to the WAL effect because of the different orientations of the magnetic field they experience. We performed simultaneous fitting of parallel and perpendicular Hikami-Larkin-Nagaoka(HLN) equations to extract the parameters. The temperature dependence of the phase coherence length suggests 2D transport for topological crystalline and 1D transport for the weak topological phase. The β value associated with parallel transport suggests Raichev and Vasilopoulos (RV) regime transport. We have also observed that Sb doping can effectively reduce the bulk contribution to the surface properties. The emergence of quantum oscillations on the Sb doped sample and the extracted parameters from Lifshitz-Kosevich (LK) fitting imply the Dirac nature of surface electrons. The surface states have higher mobility compared to the bulk supporting their topological nature. With more effective ways to project the topological surface states of BiSe, we can explore more exotic topological features that are still elusive in electrical transport experiments to date. | en_US |