How to send http requests in ForEach loop with async / await

December 8, 2022

The following code is a linqpad program.

The payloadList contains json-objects like {"id": 1, "foo": "one" }.
Each object of payloadList should be sent to a server with httpClient.SendAsync()
The response for each request should be stored in responseList

The code below does partly work. But i do not understand why some parts are not working. I assume that the responses are not completed when responseList.Add(foo) is executed.

This request shall be send for each json-object {"id": 1, "foo": "one" }

public static async Task<string> DoRequest(HttpClient client, string payload)
{   
    var request = new HttpRequestMessage(HttpMethod.Post, 
                           "http://httpbin.org/anything");
    request.Content = new StringContent(payload
                           , Encoding.UTF8, "application/json");        
    var response = await client.SendAsync(request); 
    string responseContent = await response.Content.ReadAsStringAsync(); 
    return responseContent;
}

The DoRequest()-method wraps the request and can be used inside main like this

static async Task Main()
{
    var responseList = new List<string>();  
    var payloadList = new List<string>{"{ 'id': 1, 'bar': 'One'}",
                                       "{ 'id': 2, 'bar': 'Two'}",
                                       "{ 'id': 3, 'bar': 'Three'}"};
        
    var client = new HttpClient();
    
    payloadList.ForEach(async (payload) => {
            var foo = await DoRequest(client, payload);
            foo.Dump("foo"); // this works
            responseList.Add(foo);      // this not 
    });                     
    responseList.Dump();    // is empty
}

The responseList is empty. Expected responseList.Dump() contains all responses foo. Actual responseList is empty.

Questions

How can each response for await client.SendAsync(request) be added to a responseList?
Why is responseList empty despite that foo.Dump() works?
How to confirm or check if every client.SendAsync is finished?
Would you write the code above different – why?

>Solution :

List.ForEach is not Task-aware and will execute everything in parallel without waiting for the results (i.e. it will create tasks for all items in payloadList and will continue to the next statement responseList.Dump(); without awaiting them).

In newer versions of .NET you can use Parallel.ForEachAsync(for example as in this answer) combined with use of appropriate collection from System.Collections.Concurrent, for example ConcurrentBag<T>. List is not thread-safe and modifying it concurrently can lead to a lot of problems.

Code can look like the following:

var responseList = new ConcurrentBag<string>();

await Parallel.ForEachAsync(payloadList, async (payload, ct) =>
{
    var foo = await DoRequest(client, payload, ct);
    responseList.Add(foo);     
});

static async Task<string> DoRequest(HttpClient client, string payload, CancellationToken ct)
{   
    var request = new HttpRequestMessage(HttpMethod.Post, "http://httpbin.org/anything");
    request.Content = new StringContent(payload, Encoding.UTF8, "application/json");        
    var response = await client.SendAsync(request, ct); 
    string responseContent = await response.Content.ReadAsStringAsync(ct); 
    return responseContent;
}

If you are fine with all requests running in parallel – just create a enumerable of tasks and use Task.WhenAll<T>():

var tsks = payloadList
    .Select(payload => DoRequest(client, payload));

string[] result = await Task.WhenAll(tsks);

If you want to execute requests one after another – just switch to ordinary foreach:

var responseList = new List<string>(); // no need for concurrent collection in this case 

foreach (var payload in payloadList)
{
    var foo = await DoRequest(client, payload);
    responseList.Add(foo);  
}

If you want to dive deeper here some links: