Research Scientist, @GoogleDeepMind Past: @AIatMeta , @OpenAI, @Google Brain PhD @UMassAmherst
San Francisco
- Tweets 113
- Following 517
- Followers 6,050
- Likes 5,525
thrilled to be back @Google in the @GoogleDeepMind team! The technical breadth and expertise across the whole stack (hardware->infra->deep learning->products) is truly mind-blowing. Great to see a lot of familiar faces and meet new friends. Look forward to learning a lot!
33
31
1,141
97,366
We explore a simple approach to task-oriented dialog. A single neural network consumes conversation history and external knowledge as input and generates the next turn text response along with the action (when necessary) as output. Paper: arxiv.org/pdf/1910.14613.pdf 1/4
3
57
227
We develop a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference. Joint work with @ashVaswani @nikiparmar09 Aurko Roy arxiv.org/abs/1805.11063
1
50
206
A thread on how we evaluate our embedding models in OpenAI’s API. We achieve state-of-the-art results in linear probe classification, text search and code search. It’s not fine-tuned, so it works great in the real world — and our customers love it. 1/7
5
30
155
Zero-shot results of OpenAI API’s embeddings on the FIQA search dataset. Evaluation script: github.com/arvind-neural/bei…
We zero-shot evaluated on 14 text search datasets, our embeddings outperform keyword search and previous dense embedding methods on 11 of them!
In text search tasks, we obtain best zero-shot results in msmarco, triviaQA, and NQ and also the best transfer results on the BEIR benchmark. 5/7
14
68
We are excited to release Taskmaster-1, a new task-oriented dialog dataset. We explore two methods for data collection, two-person and self-dialogs. Surprisingly self-dialogs are an effective way to collect dialog.
Paper accepted to @emnlp2019 : arxiv.org/pdf/1909.05358v1.p…
1
9
59
look forward to working with @manohar_paluri, @Ahmad_Al_Dahle, @edunov and many others in the excellent @AIatMeta team! 2/2
4
52
20,204
Thanks for a balanced take! Couple of comments that are also added to the video description now: 1/4
🔥New Video🔥
OpenAI now offers embeddings for text similarity and search, but are they holding up? We look at the release, the paper, the criticism, and most important: the price! Are the embeddings worth it? Watch here to find out:
piped.video/5skIqoO3ku0
ALT OpenAI Embeddings
4
9
53
Small models specifically fine-tuned on a dataset can do well on a narrow benchmark, but they far underperform in real-world settings, as many of our customers are discovering. This study from @FineTuneLearn shows our API performance. 7/7
2
14
48
OpenAI embeddings work on a very broad set of use cases. Here, Viable gets a 7.7% absolute improvement in clustering quality using OpenAI embeddings when compared to previous methods!
We tested different embedding models and show the data behind why GPT-3 was the clear winner for our clustering needs
askviable.com/blog/why-we-ch…
3
43
The cost to run this experiment with text-search-ada, embedding both documents and queries, is ~$80. text-search-ada achieves a 62% relative improvement over keyword search here!
Zero-shot results of OpenAI API’s embeddings on the FIQA search dataset. Evaluation script: github.com/arvind-neural/bei…
We zero-shot evaluated on 14 text search datasets, our embeddings outperform keyword search and previous dense embedding methods on 11 of them!
1
7
38
We describe a simple technique to parallelize Scheduled Sampling across time that allows us to apply Scheduled Sampling for problems that involve generating very long sequences. We get better sample quality and train almost as fast as teacher-forcing.
arxiv.org/abs/1906.04331
2
6
35
For the same reason a kind of unsupervised learning that people were always doing was branded as self-supervised learning 😉
1
30
We've trained embedding models to produce high quality text and code embeddings. Our general purpose embeddings achieve top results in classification, text search, and code search. The models are now available in the @OpenAI API: openai.com/blog/introducing-…
We're introducing embeddings, a new feature of our API that distills relationships between concepts, sentences, and even code in a simple numerical representation — for more powerful search, classification, and recommendations. openai.com/blog/introducing-…
2
3
23
@OpenAI embeddings achieve better retrieval performance and are also lot cheaper!
Results taken from: arxiv.org/pdf/2305.06300.pdf
4
2
23
2,612
My team and I trained the model. We look at 33 datasets across four different categories: linear probe classification, sentence similarity, text search, and code search. All these results and figures were in our paper, released this week. arxiv.org/pdf/2201.10005.pdf 2/7
2
23
In text search tasks, we obtain best zero-shot results in msmarco, triviaQA, and NQ and also the best transfer results on the BEIR benchmark. 5/7
3
20
TPU -> XLA -> JAX -> Transformer, MoE, Chinchilla, AlphaGo, .... -> Gemini, Veo, .... -> Search, YouTube, Waymo, ... -> Chrome, Android, .... 🤯🤯🤯
20
2,948
OpenAI Embeddings helps you go beyond keyword search!
The code is actually extremely simple for a cool app like this - open sourced here: github.com/lilianweng/emoji-…
19
A nice blog post from @mavenoid on how to get reliable generations from GPT-3: blog.mavenoid.com/gpt3-and-c…
Unsupervised retrieval+generation FTW!
4
16
Check out our spotlight talk and poster describing the Neural Assistant work in the ConvAI workshop tomorrow @NeurIPSConf #neurips19
alborz-geramifard.com/worksh…
We explore a simple approach to task-oriented dialog. A single neural network consumes conversation history and external knowledge as input and generates the next turn text response along with the action (when necessary) as output. Paper: arxiv.org/pdf/1910.14613.pdf 1/4
1
14
We do a large-scale human study to compare different decoding methods for language generation and develop a globally normalized decoding method that optimally traverses the quality-diversity curve.
How does one trade-off sample quality and diversity in a language model? Which decoding method is best? We introduce a multi-objective framework maximizing human judgement score subject to a constraint on diversity (entropy). arxiv.org/abs/2004.10450 (1/7)
1
12
In linear probe classification, we obtain best results wrt average accuracy on seven classification tasks. 3/7
2
12
In sentence similarity tasks, we perform worse than previous work. This was explained in our paper as well. 4/7
2
12
Our paper (arxiv.org/abs/1611.08945 ) on neural program induction accepted to #ICLR2017 !
Code: github.com/tensorflow/models… #DeepLearning #NLProc
2
8
in case people are counting, I forgot to share the results for text search from 3 more datasets (apart from the 11 text search results already reported) 🙂
My team and I trained the model. We look at 33 datasets across four different categories: linear probe classification, sentence similarity, text search, and code search. All these results and figures were in our paper, released this week. arxiv.org/pdf/2201.10005.pdf 2/7
7
We get good results on real-world question answering with neural semantic parsing/program induction. Code is here: github.com/tensorflow/models…
Learning a Natural Language Interface with Neural Programmer. (arXiv:1611.08945v1 [cs.CL]) ift.tt/2fvmppE
2
6
In our experiments we find that: 1) our model was able to incorporate external knowledge and generate factual text response with weak supervision signal. 2) our model can incorporate medium-size knowledge bases with only 8K training examples over multiple verticals.
1
1
4
Implementation of Neural Assistant: Joint Action Prediction, Response Generation, and Latent Knowledge Reasoning: github.com/tensorflow/tensor…
1
5
Things are changing : arxiv.org/abs/1810.04805 and multiple other recent work in nlp
5
our method actually zero-shot transfers better than bm25 to 11 search tasks on average as shown in the entire table. even our smallest models are better than bm25. while it is not the only way to exploit training data with bm25, we perform better than one such method docT5 query
1
4
The code for FIQA experiments to reproduce the results in the paper using the API: nitter.app/arvind_io/status/14882… . There's no discrepancy AFAIK. 2/4
Zero-shot results of OpenAI API’s embeddings on the FIQA search dataset. Evaluation script: github.com/arvind-neural/bei…
We zero-shot evaluated on 14 text search datasets, our embeddings outperform keyword search and previous dense embedding methods on 11 of them!
1
4
We leave out 6 not 7 BEIR datasets.Results on MSMARCO, NQ, TriviaQA are in a separate table (Table 5 in the paper).NQ is part of BEIR too and we didn't want to repeat it.The 6 datasets we leave out are not readily available and it is common to leave them out in prior work too.3/4
1
3
For example, see SPLADE v2 (arxiv.org/pdf/2109.10086.pdf) also evaluates on the same 12 BEIR datasets. Discussion from their paper: 4/4
1
4
Agree! But, I think once widely used brown clusters (e.g., : wing.comp.nus.edu.sg/~antho/…) should also be given credit. They use language model pre-training objective on unlabeled data and transfer the word clusters to supervised tasks. They are not "contextual" though.
4
Data: ai.google/tools/datasets/tas…
Work done with many awesome colleagues at Google Assistant team and
@GoogleAI along with student researcher Chinnadhurai Shankar
3
Also, Inductive bias of Transformer makes it easier to skip words and learn long-range dependencies compared to RNNs . This paper arxiv.org/abs/1801.10198 has some supporting experiments
3
I think it's a little harsh to call that work flag-planting. They performed experiments on 4 real-world datasets that AFAIK were widely used by the NLP community. In comparison there were many novel methods during that period only evaluated on toy-data.
1
3
I've noticed some of these similarities as well and @paulg explains it well "A startup founder is in effect an economic research scientist." (paulgraham.com/growth.html)
3
I think dropout: arxiv.org/abs/1207.0580
It got into JMLR after two years but people were already using it and building upon the arxiv version
1
Paper updated with experiments on image generation.
We describe a simple technique to parallelize Scheduled Sampling across time that allows us to apply Scheduled Sampling for problems that involve generating very long sequences. We get better sample quality and train almost as fast as teacher-forcing.
arxiv.org/abs/1906.04331
2
11 beir datasets used in the embeddings v1 paper: arxiv.org/abs/2201.10005
1
2
228
Zero-shot results of OpenAI API’s embeddings on the FIQA search dataset. Evaluation script: github.com/arvind-neural/bei…
We zero-shot evaluated on 14 text search datasets, our embeddings outperform keyword search and previous dense embedding methods on 11 of them!
1
The conversation is annotated with accept/reject. At test time we would want the third-party business to implement a boolean function that returns whether transaction can be completed.Neural Assistant will learn to work with the response as it has been annotated at training time.
1
1
You can find the full table of results below. Even the smallest model outperforms bm-25 and its extension, docT5query
our method actually zero-shot transfers better than bm25 to 11 search tasks on average as shown in the entire table. even our smallest models are better than bm25. while it is not the only way to exploit training data with bm25, we perform better than one such method docT5 query
1
Joint work with Daniel Duckworth, Ben Goodrich, @lukaszkaiser and Samy Bengio
1
Improvement in decoding speed, as shown by some recent work in non-autoregressive machine translation
1
The model is trained at turn-level where the dialog history fed into model as input has previous ground-truth turns of the dialog. In the conversations here the actual text responses generated by model itself are used as the assistant’s side of dialog history to be fed as input.
2
Thanks for the interest. I think Neural Assistant + Taskmaster (ai.google/tools/datasets/tas…) + Google search results as source for external knowledge can work really well for task-oriented dialog!
1
1