(cache)Pengfei Liu (@stefan

Pinned

Seedance 2.0 is impressive. But it's closed-source! Introducing our daVinci-MagiHuman — a single-stream 15B Transformer trained from scratch that jointly generates video + audio. No cross-attention. No multi-stream branches. Just self-attention.

5s 1080p video in 38s on a

2:00

The Alpaca moment of Large Multimodal Models! Can we build native LMMs just like Llama for simple multimodal generation? Introducing Anole: the first open-source, autoregressive native LMM for multimodal generation. Building on Chameleon by

@AIatMeta

: github.com/GAIR-NLP/anole

What is prompt-based learning, and what challenges are there? Will it be a new paradigm or a way for human-PLMs communication? How does it connect with other research and how to position it in the evolution of the NLP research paradigm? We released a systematic survey and beyond

Pengfei Liu

@stefan_fee

Jun 9, 2023

I just released the dataset of LIMA:

GAIR/lima · Datasets at Hugging Face

We released an elaborate and exhaustive paper list for the #NER task, covering papers from #seven top-conferences and #eight-years. Each paper has been annotated with rich information. github.com/pfliu-nlp/Name Thanks for @928673031Fu 's contribution and

's concept idea.

The potential of SFT is still not fully unlocked!!!!! Without using tools, without continue pre-training on math corpus, without RLHF, ONLY SFT, we achieve SOTA across opensource LLMs (no use external tool) on the GSM8k (83.62) and MATH (28.26) datasets: github.com/GAIR-NLP/abel

What's your system good/bad at? Where can your model outperform others? What are the mistakes that the top-10 systems make？ We are always struggling with these questions. A new academic tool can help us answer them in a one-click fashion and many more:explainaboard.nlpedia.ai

GIF

Pengfei Liu

@stefan_fee

Apr 22, 2024

Crazy finding!!!!! -> ” Without introducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B’s mathematical reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy"

Quote

Run-Ze Fan

@Vfrz525_

Apr 20, 2024

Been diving into some papers on data synthesis lately, especially those about enhancing math reasoning. Most of them seem to miss our work on 'Reformatted Alignment' (arxiv.org/abs/2402.12219)—another approach to boosting data for math reasoning.

We present a new paradigm for evaluating generated text, BARTScore: conceptualize evaluation as a text generation problem. We empirically demonstrated benefits using such framework: multi-perspective evaluation, prompt, and winning in 16 of 22 settings. Demo/Code/Paper (1/n)

Pengfei Liu

@stefan_fee

Sep 18, 2020

We have one paper. ACL20 reviewer: this should be a long paper, reject. After extending it to a long version, EMNLP20 reviewer: this should be a short paper, reject...

Quote

Yonatan Belinkov

@boknilev

Sep 18, 2020

Short paper acceptance rate is very low, which seems to continue a trend in recent #NLProc conferences. Are reviewers expecting too much from short papers? x.com/emnlp2020/stat…

Pengfei Liu

@stefan_fee

Nov 18, 2020

Interpretable Evaluation for NLP + ExplainBoard !!

@JinlanFu

@gneubig

1. Performance reached a plateau? What's left & next? 2. Can metrics tell the pros & cons of systems? 3. ExplainBoard: Next Generation Leaderboard?

@emnlp2020

arxiv.org/pdf/2011.06854 arxiv.org/pdf/2011.06858

GIF

Pengfei Liu

@stefan_fee

May 3, 2024

How much data *leakage* do popular LLMs have on public benchmarks? Our recent work will tell you: Benchmarking Benchmark Leakage in Large Language Models arxiv.org/pdf/2404.18824

Quote

Alexandr Wang

@alexandr_wang

May 2, 2024

How overfit are popular LLMs on public benchmarks? New research out of @scale_ai SEAL to answer this: - produced a new eval GSM1k - evaluated public LLMs for overfitting on GSM8k VERDICT: Mistral & Phi are overfitting benchmarks, while GPT, Claude, Gemini, and Llama are not.

The last step of the #ACL2021 submission: manual text compression! We should keep our data before and after compression so that we can automate this process next year

Pengfei Liu

@stefan_fee

Apr 22, 2020

#ACL2020 A new framework for Extractive Summarization: don't label sentences individually, directly M-A-T-C-H! Even instantiating the framework with a simple model, the SOTA result has been driven to a new level. But how does it work? [1/n] Paper&code: arxiv.org/pdf/2004.08795

Pengfei Liu

@stefan_fee

Sep 14, 2024

OpenAI o1-preview achieved an average improvement of 20+ points on OlympicArena (e.g., math, physics, chemistry, bio, geo, astro) !!! However, there's still significant room for improvement (average score below 60 points)

Quote

Zhen Huang

@Z_Huang_02

Sep 14, 2024

o1-preview has shown incredible improvements in reasoning ability across complex disciplines on our OlympicArena (val + text-only) subset! We’re also eagerly looking forward to the performance of the multimodal version of o1 in the future! x.com/Z_Huang_02/sta…

Excited to share our #EMNLP2020 work: REALSum: Re-evaluating Evaluation in Text Summ: arxiv.org/pdf/2010.07100 (super awesome coauthors:

and

) Are existing automated metrics reliable??? All relevant resource has been released (1/n)!

GIF

Pengfei Liu

@stefan_fee

Jun 23, 2021

What are the strengths and weaknesses of your Machine Translation systems? Where does your model outperform others? ExplainaBoard currently supports the diagnostic and nice case analysis of 200+ machine translation systems，18 language pairs from WMT20: explainaboard.nlpedia.ai/leaderboard/ta

Pengfei Liu

@stefan_fee

Jul 21, 2021

I'm super honored that our paper was awarded as "Best Demo Paper" at ACL 2021!! Special thanks to all awesome collaborators, reviewers' insightful suggestions, and the committee's recognition. We will continue optimizing ExplainaBoard to make it a truly useful tool!

Quote

Pengfei Liu

@stefan_fee

Apr 15, 2021

GIF

What's your system good/bad at? Where can your model outperform others? What are the mistakes that the top-10 systems make？ We are always struggling with these questions. A new academic tool can help us answer them in a one-click fashion and many more:explainaboard.nlpedia.ai

Pengfei Liu

@stefan_fee

Mar 1, 2022

Getting good data for ML training is a science. How well do you know your data? DataLab: a Platform for Data Analysis and Intervention. This is your lab! Web: datalab.nlpedia.ai Paper: arxiv.org/pdf/2202.12875 At

@DataLab007

with dataset names, we will do analysis for u.

GIF

Pengfei Liu

@stefan_fee

Nov 11, 2021

Three months on, let's look back and see where the field of prompt-based learning is at? 74 -> 120+ papers in the past two months

Quote

Pengfei Liu

@stefan_fee

Jul 30, 2021

What is prompt-based learning, and what challenges are there? Will it be a new paradigm or a way for human-PLMs communication? How does it connect with other research and how to position it in the evolution of the NLP research paradigm? We released a systematic survey and beyond

Pengfei Liu

@stefan_fee

Mar 6, 2020

-What are the hot topics for Text Summarization in 2019? (BERT? GNNs? Not exactly!) -What are the most-cited papers for text summarization since 2014? Check out our #Summarization_Paper_List_and_Beyond github.com/neulab/Text-Su

Exciting news! Our

demo for Anole is now live on

@huggingface

!

huggingface.co/spaces/ethanch

Quote

Pengfei Liu

@stefan_fee

Jul 10, 2024

The Alpaca moment of Large Multimodal Models! Can we build native LMMs just like Llama for simple multimodal generation? Introducing Anole: the first open-source, autoregressive native LMM for multimodal generation. Building on Chameleon by @AIatMeta: github.com/GAIR-NLP/anole

How can we reliably estimate a system's performance without performing experiments? Check out our work arxiv.org/pdf/2102.05486 (EACL) 1. Formulate performance prediction as Tensor Completion problem 2. Establish a set of reliability analysis mechanisms (confidence, calibration)

Pengfei Liu

@stefan_fee

Sep 13, 2022

Just as prompts can connect human and large language models, Inspired Cognition aims to close human with AI by standardizing chaos, integrating fragmentation, and democratizing domain expertise:)

Quote

Graham Neubig

@gneubig

Sep 13, 2022

Happy to announce that I've formed a company, Inspired Cognition (inspiredco.ai) together with @stefan_fee and @odashi_en! Our goal is to make it easier and more efficient to build AI systems (particularly NLP) through our tools and expertise. 1/2

Pengfei Liu

Pengfei Liu’s posts