research area Speech and Natural Language Processing, research area Tools, Platforms, Frameworksconference NAACL
content type paperpublished August 2024

APE: Active Prompt Engineering - Identifying Informative Few-Shot Examples for LLMs

AuthorsKun Qian, Farima Fatahi Bayat, Anton Belyi, Yash Govind, Rahul Khot, Katherine Luna, Azadeh Nikfarjam, Xiaoguang Qi, Yisi Sang, Fei Wu, Victor (AIML) Zhang, Yunyao Li, Samira Khorshidi, Xianqi Chu

View publication

Prompt engineering is an iterative procedure that often requires extensive manual efforts to formulate suitable instructions for effectively directing large language models (LLMs) in specific tasks. Incorporating few-shot examples is a vital and efficacious approach to provide LLMs with precise and tangible instructions, leading to improved LLM performance. Nonetheless, identifying the most informative demonstrations for LLMs is labor-intensive, frequently entailing sifting through an extensive search space. In this demonstration, we showcase an interactive tool called APE (Active Prompt Tuning) designed for refining prompts through human feedback. Drawing inspiration from active learning, APE iteratively selects the most ambiguous examples for human feedback, which will be transformed into few-shot examples within the prompt.

Related readings and updates.

Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models

*Equal Contributors Large language models (LLMs) are increasingly being adapted to achieve task-specificity for deployment in real-world decision systems. Several previous works have investigated the bias transfer hypothesis (BTH) by studying the effect of the fine-tuning adaptation strategy on model fairness to find that fairness in pre-trained masked language models have limited effect on the fairness of models when adapted using fine-tuning…
See paper details

SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Tracking

In-context learning with Large Language Models (LLMs) has emerged as a promising avenue of research in Dialog State Tracking (DST). However, the best-performing in-context learning methods involve retrieving and adding similar examples to the prompt, requiring access to labeled training data. Procuring such training data for a wide range of domains and applications is time-consuming, expensive, and, at times, infeasible. While zero-shot learning…
See paper details