Challenges in LLMs: Single-Shot Q&A vs. Back-and-Forth Tasks

·

turns out LLMs are great at single-shot q&a but get lost when it becomes a back-and-forth. researchers shard a coding task into multiple hints; models answered too early, kept clinging to bad assumptions, and produced longer yet worse code on each turn.

4 comments

· Sorted by Oldest

Anirudh M.
·
·
something they also suggest to mitigate this summarize requirements each turn (“snowball”) or at least once before final answer (“recap”)
Anirudh M.
·
·
i think this is more with vibe coding. not sure. did anyone else face this?
Ariel R.
·
·
true, specially true while coding on large. codebases
Ariel R.
·
·
that being said, in tools like cursor (if we focus on coding) that problem is pretty much mitigated because you can choose what context to provide and iterate on small parts of code

Anirudh M.
·
·
something they also suggest to mitigate this summarize requirements each turn (“snowball”) or at least once before final answer (“recap”)
Anirudh M.
·
·
i think this is more with vibe coding. not sure. did anyone else face this?
Ariel R.
·
·
true, specially true while coding on large. codebases
Ariel R.
·
·
that being said, in tools like cursor (if we focus on coding) that problem is pretty much mitigated because you can choose what context to provide and iterate on small parts of code