Asst. Prof of Ling, and Harrington Fellow at @UTAustin. language, concepts, and generalization. also on the site where the sky is blue. Aspiring wugologist
- Tweets 3,743
- Following 682
- Followers 1,667
- Likes 23,394
ALT Picture of the UT Tower with "UT Austin Computational Linguistics" written in bigger font, and "Humans processing computers processing human processing language" in smaller font
ALT Picture of the UT Tower taken by me on my first day at UT as a postdoc in 2023!
ALT Title page of our paper: βLanguage Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNsβ - Kanishka Misra and Kyle Mahowald
ALT Dissertation title slide with title: "On Semantic Cognition, Inductive Generalization, and Language Models"
ALT Title of the paper: Generating novel experimental hypotheses from language models: A case study on cross-dative generalization; along with a figure describing our overall method
ALT mirror image of twitter's logo using the bbold latex package.
ALT Placeholders for 3 students (number arbitrarily chosen) and me - to signify my eventual group!
ALT Paper title: Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently Authors: Kanishka Misra, Allyson Ettinger, Kyle Mahowald
ALT Title: minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models. Author: Kanishka Misra (email: kmisra@purdue.edu)
ALT Title page of our paper: βLanguage Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNsβ - Kanishka Misra and Kyle Mahowald
ALT Screenshot of the title of our BUCLD abstract: "Abstraction via exemplars? A representational case study on lexical category inference in BERT"
ALT screenshot from github (dark mode) showing 101 stars for minicons
ALT photo of miami (4th result on google images)
ALT Title page of SCIL extended abstract titled: semantic-features: A User-Friendly Tool for Studying Contextual Word Embeddings in Interpretable Semantic Spaces
ALT examples from direct and prepositional object datives with short-first and long-first word orders: DO (long first): She gave the boy who signed up for class and was excited it. PO (short first): She gave it to the boy who signed up for class and was excited. DO (short first): She gave him the book that everyone was excited to read. PO (long-first): She gave the book that everyone was excited to read to him.
ALT Title: "Characterizing the Role of Similarity in the Property Inferences of Language Models" Authors: Juan Diego Rodriguez, Aaron Mueller, Kanishka Misra Left figure: "Given that dogs are daxable, is it true that corgis are daxable?" A language model could answer this either using taxonomic relations, illustrated by a taxonomy dog-corgi, dog-mutt, canine-wolf, etc., or by similarity relations (dogs are more similar to corgis than cats, wolves or shar peis). Right figure: illustration of the causal model (and an example intervention) for distributed alignment search (DAS), which we used to find a subspace in the network responsible for property inheritance behavior. The bottom nodes are "property", "premise concept (A)" and "conclusion concept (B)" , the middle nodes are "A has property P", "B is a kind of A", and the top node is "B has property P".
ALT Paper title: Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently Authors: Kanishka Misra, Allyson Ettinger, Kyle Mahowald
![code demonstrating how BLIP2-OPT-2.7B can be used queried to see if a penguin vs. a cardinal is more likely to fly, given their respective images.
import torch
from minicons import scorer
from PIL import Image
penguin = Image.open('penguin.jpg')
cardinal = Image.open('cardinal.jpg')
lm = scorer.VLMScorer(
"Salesforce/blip2-opt-2.7b",
device="cuda:0"
)
lm.sequence_score(
text_batch=["This bird can fly."] * 2,
image_batch=[penguin, cardinal]
)
#> logprobs of penguin vs cardinal -> can fly
#> [-5.644123077392578, -5.129026889801025]](/pic/orig/media%2FGJN8hVZWMAAci2Y.jpg)
ALT code demonstrating how BLIP2-OPT-2.7B can be used queried to see if a penguin vs. a cardinal is more likely to fly, given their respective images. import torch from minicons import scorer from PIL import Image penguin = Image.open('penguin.jpg') cardinal = Image.open('cardinal.jpg') lm = scorer.VLMScorer( "Salesforce/blip2-opt-2.7b", device="cuda:0" ) lm.sequence_score( text_batch=["This bird can fly."] * 2, image_batch=[penguin, cardinal] ) #> logprobs of penguin vs cardinal -> can fly #> [-5.644123077392578, -5.129026889801025]
![Code:
from minicons import scorer
lm = scorer.IncrementalLMScorer("gpt2-xl", "cuda:0")
stimuli = ["I was a matron in France", "I was a mat in France"]
# old way, no correction
# P.S. gpt2 does not automatically add a bos token at the beginning...
lm.token_score(stimuli, bos_token=True, surprisal=True, base_two=True, bow_correction=False)
'''Rounded Output
[[('<|endoftext|>', 0.0),
('I', 5.85),
('was', 4.28),
('a', 4.67),
('mat', 16.34),
('ron', 1.74),
('in', 2.12),
('France', 11.43)],
[('<|endoftext|>', 0.0),
('I', 5.85),
('was', 4.28),
('a', 4.67),
('mat', 16.34),
('in', 10.78),
('France', 10.71)]]
'''
# the new way! notice the surprisal of "mat" in both cases
lm.token_score(stimuli, bos_token=True, surprisal=True, base_two=True, bow_correction=True)
'''Rounded Output
[[('<|endoftext|>', 0.0),
('I', 6.30),
('was', 3.84),
('a', 4.68),
('mat', 16.34),
('ron', 2.11),
('in', 1.75),
('France', 11.42)],
[('<|endoftext|>', 0.0),
('I', 6.30),](/pic/orig/media%2FGdPfDmmbIAAfJNS.jpg)
ALT Code: from minicons import scorer lm = scorer.IncrementalLMScorer("gpt2-xl", "cuda:0") stimuli = ["I was a matron in France", "I was a mat in France"] # old way, no correction # P.S. gpt2 does not automatically add a bos token at the beginning... lm.token_score(stimuli, bos_token=True, surprisal=True, base_two=True, bow_correction=False) '''Rounded Output [[('<|endoftext|>', 0.0), ('I', 5.85), ('was', 4.28), ('a', 4.67), ('mat', 16.34), ('ron', 1.74), ('in', 2.12), ('France', 11.43)], [('<|endoftext|>', 0.0), ('I', 5.85), ('was', 4.28), ('a', 4.67), ('mat', 16.34), ('in', 10.78), ('France', 10.71)]] ''' # the new way! notice the surprisal of "mat" in both cases lm.token_score(stimuli, bos_token=True, surprisal=True, base_two=True, bow_correction=True) '''Rounded Output [[('<|endoftext|>', 0.0), ('I', 6.30), ('was', 3.84), ('a', 4.68), ('mat', 16.34), ('ron', 2.11), ('in', 1.75), ('France', 11.42)], [('<|endoftext|>', 0.0), ('I', 6.30),
ALT Screenshot from Oh and Schuler showing surprisal values for the partial sentences "I was a matron in" and "I was a mat in" using GPT-2 XL with leading whitespaces and trailing whitespaces.
ALT A screenshot of the cogsci 2022 paper submission policy, with "unlimited pages for references" underlined
ALT screenshot from emnlp 2024 website. Salient info: all papers accepted to the main conf have to submit a pre-recorded video, a poster pdf, as well as slides.
ALT Screenshot of the title of our BUCLD abstract: "Abstraction via exemplars? A representational case study on lexical category inference in BERT"
ALT screenshot from an email I am writing, with microsoft's grammar editor recommending I change "more light" in the phrase "I am happy to shed more light..." to "lighter"
ALT screenshot of a github issue created on github.com/kanishkamisra/minicons asking whether I named the package 'minicons' due to its relevance in the fictional transformers universe. I replied with "yes"
ALT screenshot of python code that allows you to use within word l2r method developed by Carina Kauf and Anna Ivanova.
ALT screenshot demonstrating how one can use quantization using minicons, and also run stuff on multiple gpus. Here's the code: # run `pip install bitsandbytes` first import torch from transformers import BitsAndBytesConfig from minicons import scorer # multiple device support lm = scorer.IncrementalLMScorer("meta-llama/Llama-2-13b-hf", device="auto") # multiple device + quantization support bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, ) lm = scorer.IncrementalLMScorer( "meta-llama/Llama-2-13b-hf", device="auto", quantization_config=bnb_config ) # takes ~ 8gb on a A40 GPU
![from minicons import scorer
from nltk.tokenize import TweetTokenizer
lm = scorer.IncrementalLMScorer("gpt2")
# your own tokenizer function that returns a list of words
# given some sentence input
word_tokenizer = TweetTokenizer().tokenize
# word scoring
lm.word_score_tokenized(
["I was a matron in France", "I was a mat in France"],
bos_token=True, # needed for GPT-2/Pythia and NOT needed for others
tokenize_function=word_tokenizer,
bow_correction=True, # Oh and Schuler correction
surprisal=True,
base_two=True
)
'''
First word = -log_2 P(word | <beginning of text>)
[[('I', 6.1522440910339355),
('was', 4.033324718475342),
('a', 4.879510402679443),
('matron', 17.611848831176758),
('in', 2.5804288387298584),
('France', 9.036953926086426)],
[('I', 6.1522440910339355),
('was', 4.033324718475342),
('a', 4.879510402679443),
('mat', 19.385351181030273),
('in', 6.76780366897583),
('France', 10.574726104736328)]]
'''](/pic/orig/media%2FGngBaZuWIAAQya_.jpg)
ALT from minicons import scorer from nltk.tokenize import TweetTokenizer lm = scorer.IncrementalLMScorer("gpt2") # your own tokenizer function that returns a list of words # given some sentence input word_tokenizer = TweetTokenizer().tokenize # word scoring lm.word_score_tokenized( ["I was a matron in France", "I was a mat in France"], bos_token=True, # needed for GPT-2/Pythia and NOT needed for others tokenize_function=word_tokenizer, bow_correction=True, # Oh and Schuler correction surprisal=True, base_two=True ) ''' First word = -log_2 P(word | <beginning of text>) [[('I', 6.1522440910339355), ('was', 4.033324718475342), ('a', 4.879510402679443), ('matron', 17.611848831176758), ('in', 2.5804288387298584), ('France', 9.036953926086426)], [('I', 6.1522440910339355), ('was', 4.033324718475342), ('a', 4.879510402679443), ('mat', 19.385351181030273), ('in', 6.76780366897583), ('France', 10.574726104736328)]] '''
ALT screenshot of the last 3 lines in the dev set of aochildes saying: aha . is the computer a type of pencil ? we're all pencils .
ALT very contentful poster about COMPS. Paper: https://arxiv.org/abs/2210.01963
