Shweta Mahajan

I am a Machine Learning Researcher at Qualcomm AI Research. I was a postdoctoral researcher and a Vector Postdoctoral Affiliate in the Vision Group at University of British Columbia advised by Prof. Leonid Sigal and Prof. Kwang Moo Yi. My research at UBC is focussed on diffusion models for high-level as well as low-level computer vision. I obtained my Ph.D. under the supervision of Prof. Stefan Roth, Ph.D. in the Visual Inference Group, Technische Universität Darmstadt. During my Ph.D., I researched deep generative algorithms for multimodal representation learning and the efficiency of exact inference deep generative models. I received my M.Sc. from the Saarland University where I was a part of the Machine Learning Group and the Max Planck Institute of Informatics.

Email  /  CV  /  Google Scholar  /  LinkedIn

News

• 02/2024: Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models accepted at CVPR 2024.
• 02/2024: ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models accepted at CVPR 2024.
• 02/2024: Unsupervised Keypoints from Pretrained Diffusion Models accepted at CVPR 2024.
• 11/2023: Recognized as a Top Reviewer at NeurIPS 2023!
• 09/2023: Talk on Multimodal Representation Learning with Deep Generative Models at NTU Singapore.
• 09/2023: Unsupervised Semantic Correspondence Using Stable Diffusion accepted at NeurIPS 2023.
• 08/2023: Nominated for the GI-Dissertationspreis!
• 07/2023: Awarded research grant by the Vector Institute of Artificial Intelligence!
• 06/2023: Invited to the doctoral consortium at CVPR 2023!
• 06/2023: Nominated for the Bertha Benz Best Thesis Award in Germany!
• 02/2023: Make-A-Story: Visual Memory Conditioned Consistent Story Generation accepted at CVPR 2023.
• 10/2022: I have joined UBC as a postdoctoral researcher.
• 06/2022: I have successfully defended my Ph.D. dissertation (summa cum laude)!

Research

I am interested in computer vision and machine learning, specifically in deep generative models (diffusion models, normalizing flows, variational methods, GANs) for multimodal representation learning.

Inverting the diffusion model to obtain interpretable language prompts directly based on the findings that different timesteps of the diffusion process cater to different levels of detail in an image.

We utilize a pre-trained video diffusion model to solve consistency in zero-shot view synthesis.

One can leverage this semantic knowledge within diffusion models to find key points across images of similar kind.

One can leverage this semantic knowledge within diffusion models to find semantic correspondences with prompt optimization.

Sentence-conditioned soft attention over the memories enables effective reference resolution and learns to maintain scene and actor consistency when needed.

A sequential variational framework encoding the style information grounded in images for stylized image captioning.

A block-autoregressive exact inference model employing a lossless pyramid decomposition with scale-specific representations to encode the joint distribution of image pixels.

We introduce a novel factorization of the latent space to model diversity in contextual descriptions across images and texts within the dataset.

We improve the representational power of flow-based models by introducing channel-wise dependencies in their latent space through multi-scale autoregressive priors.

Our model integrates normalizing flow-based priors for the domain-specific information, which allows us to learn diverse many-to-many mappings between the image and text domains.

We propose joint Gaussian regularization of the latent representations to ensure coherent cross-modal semantics that generalize across datasets.