Great writeup, thanks for sharing your process! I'm glad to hear other folks are focusing on latency and waiting time. I've found it helps my patience to use OpenAI's async streaming API. The chat and chat completion endpoints accept a "stream": True argument, and with some work you can incrementally lay out pages for impatient readers. hth, good luck!
Thanks again for this nudge! I was intimidated by it, but I think I might have this working now. It’s comments like this that really help push this product to the next level ❤️
Thanks so much for taking the time to share your thought here. :) I've looked at the stream stuff, and I initially dismissed it because I have a couple of checks on content to make sure it has 3 options when the output is returned and I often don't know that until the end; BUT!
Your post reminded me that a) I've been improving the reliability on responses and b) maybe I'm making the wrong choice. Is it better to wait a potentially long time, but only ever get correct output? Or is it better to give the user some updates and things to read as they render, and give the user the option to roll again if the LLM doesn't return the expected output? I'd love your thoughts on that. And thanks for the nudge :)
Great writeup, thanks for sharing your process! I'm glad to hear other folks are focusing on latency and waiting time. I've found it helps my patience to use OpenAI's async streaming API. The chat and chat completion endpoints accept a "stream": True argument, and with some work you can incrementally lay out pages for impatient readers. hth, good luck!
Thanks again for this nudge! I was intimidated by it, but I think I might have this working now. It’s comments like this that really help push this product to the next level ❤️
Thanks so much for taking the time to share your thought here. :) I've looked at the stream stuff, and I initially dismissed it because I have a couple of checks on content to make sure it has 3 options when the output is returned and I often don't know that until the end; BUT!
Your post reminded me that a) I've been improving the reliability on responses and b) maybe I'm making the wrong choice. Is it better to wait a potentially long time, but only ever get correct output? Or is it better to give the user some updates and things to read as they render, and give the user the option to roll again if the LLM doesn't return the expected output? I'd love your thoughts on that. And thanks for the nudge :)