WebRequest slow?

I recently tried to do a web request in .NET C#. So I looked it up in the MSDN and wrote down some lines of code and was happy that it worked  straightaway (ok, it’s not very complex anyhow).

It looked something like this:

    string responseContent;
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = "GET";

    HttpWebResponse response = (HttpWebResponse)request.GetResponse();

    Stream responseStream = response.GetResponseStream();
    using (StreamReader sr = new StreamReader(responseStream))
        responseContent = sr.ReadToEnd();

    return responseContent;

But this code was very slow and it took several seconds to get the response.
One reason for that is that if you don’t use a proxy or just don’t set the proxy property of the WebRequest, it tries to auto-detect and configure a proxy what causes a significant delay.
So just adding

    request.Proxy = null;

before the GetResponse() call will fix that problem.

Remark/Update: To be more correct, you should set “request.Proxy = GlobalProxySelection.GetEmptyWebProxy();” instead of just null. (MSDN: HttpWebRequest.Proxy property)

Another point is that you should always close the WebResponse after processing its contents. That might also speed up your requests. I prefer to use “using” statements to do that. It will get a bit nested sometimes but this way you just can’t forget it.
So here’s the final code for a simple “GET” request:

public string MakeWebRequest(string url)
{
    try
    {
        string responseContent;
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
        request.Method = "GET";
        request.Proxy = GlobalProxySelection.GetEmptyWebProxy(); // null;

        using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
        {
            using (Stream responseStream = response.GetResponseStream())
            {
                using (StreamReader sr = new StreamReader(responseStream))
                    responseContent = sr.ReadToEnd();
            }
        }

        return responseContent;
    }
    catch (Exception ex)
    {
        // Error handling
        return ex.Message;
    }
}

Extending the code for a “POST” request:

To make a “POST” request we have to add the following:

public string MakeWebRequest(string url, string postData)
{
    try
    {
        string responseContent;
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
        // Set Method to "POST"
        request.Method = "POST";
        // Convert "POST" data to a byte array
        byte[] byteArray = Encoding.UTF8.GetBytes(postData);
        // Set the content type of the WebRequest
        request.ContentType = "application/x-www-form-urlencoded";
        // Set the content length
        request.ContentLength = byteArray.Length;
        // Get the request stream
        using (Stream requestStream = request.GetRequestStream())
        {
            // Write the "POST" data to the stream
            requestStream.Write(byteArray, 0, byteArray.Length);
        }
        // Don't forget to set the proxy
        request.Proxy = GlobalProxySelection.GetEmptyWebProxy(); // null;

        using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
        {
            using (Stream responseStream = response.GetResponseStream())
            {
                using (StreamReader sr = new StreamReader(responseStream))
                    responseContent = sr.ReadToEnd();
            }
        }

        return responseContent;
    }
    catch (Exception ex)
    {
        // Error handling
        return ex.Message;
    }
}
public string MakeWebRequest(string url)
{
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = “GET”;
request.Proxy = null;

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream responseStream = response.GetResponseStream();
using (StreamReader sr = new StreamReader(responseStream))
{
string responseContent = sr.ReadToEnd();
return responseContent;
}
}
}
catch (Exception ex)
{
return ex.Message;
}
}

About these ads

13 Responses to “WebRequest slow?”

  1. Vincent Says:

    Hi Steffen,

    Where does it gets the proxy list from? ie settings?
    I actually tried to set proxy=null but there wasnt much difference. Does setting proxy=null means avoid lookup for proxy?

    Thanks
    Vincent

    • Steffen Höhle Says:

      Hi Vincent,
      if you don’t specify a proxy, the app/web/machine.config will be searched for the “system.net/defaultProxy” property. If this property is not defined it uses the IE proxy settings, that’s right. Though I’m not really sure if these operations are the main reason for the large delay. On my machine it takes up to 10-15sec longer than with a nulled out proxy property but I’ve also head of people who didn’t have that much problems with that.
      But if you set the proxy to a value different than the default one, the lookup will be skipped.
      As a side note, if you don’t use a proxy you should set the “request.Proxy = GlobalProxySelection.GetEmptyWebProxy();” instead of just null. I will correct that in the Blog post.

      Here are some links to the MSDN pages that may be interesting. You should have a look at the remarks sections:
      HttpWebRequest.Proxy Property
      GlobalProxySelection
      system.net/defaultProxy

      Regards,
      Steffen

  2. Michael Says:

    You Sir, are an absolute lifesaver.

    I was fiddling with WebRequest myself and was puzzled for at least an hour. Scanned network activity to see if it might be some odd connection problem.

    What I don’t understand though, is why Microsoft did this. Wouldn’t it be more logical to pick the proxy config from a config file or the IE settings? This should literally only take seconds in my opinion. I guess MS’s ambition to make things smart bit them in the back this time…

    Thanks a lot anyway!

    • Steffen Höhle Says:

      Hi Michael,
      it actually does search the .config files and finally falls back to the IE settings, as I stated above. And the IE seems to be the bottleneck.

      A few days ago I wanted to set up a WebDAV drive and it was incredibly slow on Windows 7 while on XP everything worked just fine. And the solution I found was to open IE and head to Tools->Options->Connections->LAN Settings and uncheck “Automatically detect settings”.
      And this little checkbox also affects the WebRequest. So if you uncheck it you will also notice that the delay is gone.

  3. Portland Oregon seo Says:

    Kudos

  4. Grant Says:

    “It will get a bit nested sometimes…”

    You can combine using statements like this:
    using (foo)
    using (bar) {
    // do stuff
    }

    • Steffen Höhle Says:

      Hi Grant,

      Thanks for the tip. I didn’t know that. This definately makes it more readable.

  5. Rotem BLoom Says:

    Does it happned on all external URL or just internal URL contain localhost?
    We also got a delay when use localhost in URL when we change the URL to the external URL the delay gone.

    Any how you can change the setting in the web.config in order to remove the proxy:

  6. erdpghjrop Says:

    request.Proxy=null ; worked for me.

    also you can add the following to youw config file:

  7. Felipe Ranieri Says:

    Hi,
    I’ve saw this workaround somewhere else and I don’t recall where.
    I don’t actually really know and didn’t have time to research yet and tell why or how it works.
    But it worked for me, made my connections quite very faster when I’ve add this parameter at my class’ constructor before instantiating my HTTPWebRequest object:

    System.Net.ServicePointManager.Expect100Continue = false;

    Please, could you try it and tell us?

    Best wishes, mate!
    Felipe Ranieri

    • Steffen Höhle Says:

      Hi Felipe,
      the Expect100Continue property should only affect POST requests. I think normally it should be left at the default value (true), so if the server requires authentication the client just sends the request headers and after it is authenticated it continues to send the POST body. Otherwise it sends the POST data twice, which would increase the request time.

      I tried it out anyway and inserted the line but it had no effect on the delay on my machine.

  8. poolhall Says:

    I think you should set the proxy before you send the request, not after like you have in the POST request example.

  9. മിന്നാമിന്നി Says:

    I tried this but got below error
    The remote server returned an error: (401) Unauthorized
    I commented below code and set proxy to nothing
    ‘Dim proxy As IWebProxy = Net.WebRequest.GetSystemWebProxy()
    Commented Code begins
    ‘proxy.Credentials = CredentialCache.DefaultCredentials
    ‘request.Proxy = proxy
    ‘Dim netcredit As NetworkCredential = New NetworkCredential(“admin”, “meinsm”)
    ‘request.Credentials = netcredit
    Commented code ends

    request.Proxy = Nothing


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: