VISPROG Paper Weak Reject

Anonymous

write a review for this paper

Reviewer 2 logo
Reviewer 2

Summary

The paper "Visual Programming: Compositional Visual Reasoning Without Training" by Gupta and Kembhavi presents VISPROG, a neuro-symbolic approach for solving complex visual tasks using natural language instructions. VISPROG avoids task-specific training, instead leveraging large language models like GPT-3 to generate modular Python-like programs that are executed to provide solutions and interpretable rationales. The system demonstrates flexibility across various tasks: compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing.

Strengths

  1. Innovative Approach: VISPROG's use of neuro-symbolic methods to generate programs for vision tasks is novel and shows potential in tackling a variety of complex visual tasks.

  2. Interpretability: The system enhances interpretability by breaking down predictions into simpler steps, allowing users to diagnose errors and potentially intervene in the reasoning process.

  3. No Task-specific Training: VISPROG's ability to function without task-specific training, leveraging in-context learning of large language models, is a significant advantage, making it adaptable to a wide range of tasks.

Weaknesses

  1. Dependence on Large Language Models: The system's reliance on models like GPT-3 raises concerns about its adaptability and performance in scenarios where such models may not be as effective or available.

  2. Generalization and Robustness: The paper does not sufficiently address the generalization and robustness of VISPROG across diverse real-world scenarios, which is crucial for practical applications.

  3. Error Analysis: While the paper discusses error analysis, it lacks a deeper exploration of the limitations and failure modes of VISPROG, particularly in complex or ambiguous visual scenes.

Questions

  1. How does VISPROG handle scenarios where GPT-3's in-context learning might not provide accurate or relevant program generation?

  2. Can VISPROG be adapted to work with other language models or is it specifically tailored for GPT-3?

  3. What are the computational requirements for running VISPROG, especially when generating and executing complex programs?

Recommendation

Weak Reject. While VISPROG presents an innovative approach in compositional visual reasoning without training and enhances the interpretability of AI solutions in vision tasks, the paper lacks a comprehensive exploration of the system's robustness and generalization across diverse real-world scenarios. Additionally, the dependence on large language models like GPT-3 might limit its applicability in varied contexts. Further research and development are needed to address these concerns and fully realize the potential of this approach.