@zswang24@OpenAI also "100% said it helps them take on more work than they otherwise could" is kind of a depressing stat when stated like that. probably nicer to say that tasks (they don't like doing) take less time, giving them more time other things (they do like).
@zswang24@OpenAI just to hedge: there is always room for improvement, and it's plain to anyone who works in data that this exciting progress, but there is still a long way to go...
Just once, I wish AI would ask a clarifying question. Like "How do you define revenue?"
Data is often ambiguous, and AIs LOVE making assumptions. These demos always built on top of data that is neat in a way that isn't reflective of the real world. Ironic because AI is actually great for cleaning/reasoning. Demo that!
@nikillinit i've had this problem in nyc of course, but do most people have this issue? I feel like most people fill at the same pharmacy by their home
I have done a ton of prompting and evals trying to get it to ask questions. I did have to build my own harness that gave it no other options, but that isn't really helpful in practice. Claude et al need to respect a prompt such as "when encountering ambiguity, seek clarification from the user" or something. They don't. They're far too bench-maxxed
Just once, I wish AI would ask a clarifying question. Like "How do you define revenue?"
Data is often ambiguous, and AIs LOVE making assumptions. These demos always built on top of data that is neat in a way that isn't reflective of the real world. Ironic because AI is actually great for cleaning/reasoning. Demo that!