I am working with an API that gives data back to me that is paginated. IE, I request a page size of 50, get some data back, and then use the ID of the last bit of data I received to make another request. I want to write a function that returns data as it receives it, is this creating a stream and sending data through it? What I’m trying to avoid is waiting for one function to finish running through ALL of the pages, using up lots of memory, and then returning a massive array so that the main function can then pass all of that to another function to start processing the data. Ideally as data is retrieved from my API function it will return it to main and main can start processing it right away. The purpose of this is to limit how much memory is used at once and limit execution time. Is this a reasonable way to structure this and how would I go about implementing it? Are there better ways to structure this that accomplish the same goals without shoving all of the code into one giant/messy function?
TIA
>Solution :
This sounds like a good use-case for an AsyncGenerator. You can loop over an asyncgenerator and get multiple results, instead of 1 final return.