[C#] HtmlAgilityPack doesn't work with Windows Forms but works in Console Application

07/10/2019 10:40 Saisama#1
I've got a HTML document with HtmlWebRequest. Then I've tried to parse it and getting a value out of it using HtmlAgilityPack. I've found out that if I do this in a console application there is not any issue and I can get the value. But if I use a Windows Forms application with this same code to get this value and assign it to another variable, it doesn't work and I get a NullReferenceException. And what I've found out is that my Windows Forms application works while Fiddler is running. And this console application works without Fiddler.

What I've tried so far:
  • Using a proxy
  • Compared GET request's what my application sends to Internet Explorer's request. They looked similar.
  • Using RegEx instead of HtmlAgilityPack

Console Application:

Code:
using HtmlAgilityPack;
using System;
using System.IO;
using System.Net;
using System.Threading;

namespace HttpTest
{
    class Program
    {
        static void Main(string[] args)
        {
            Thread worker;
            var token = "";
            worker = new Thread(() =>
            {
                token = getToken();
                Console.WriteLine("Token: " + token);
            });
            worker.Start();
        }

        public static string GetRequest(string url, string host, bool keepAlive)
        {
            CookieContainer cookie = new CookieContainer();
            string html = string.Empty;
            var req = (HttpWebRequest)WebRequest.Create(url);
            req.Method = "GET";
            req.Host = host;
            req.KeepAlive = keepAlive;
            req.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko";
            req.Timeout = 15000;
            req.Accept = "text/html, application/xhtml+xml, image/jxr, */*";
            req.CookieContainer = cookie;

            var resp = (HttpWebResponse)req.GetResponse();

            var stream = new StreamReader(resp.GetResponseStream());
            html = stream.ReadToEnd();

            stream.Close();
            resp.Close();
            return html;
        }

        public static string getToken()
        {
            var content = GetRequest("url", "host", true);
            string signupFormId;

            var doc = new HtmlAgilityPack.HtmlDocument();
            doc.LoadHtml(content);
            HtmlNode signupFormIdElement = doc.DocumentNode.SelectSingleNode("//input[@name='login']");
            signupFormId = signupFormIdElement.GetAttributeValue("value", "");
            return signupFormId;
        }
    }
}
Error which I get in my Forms app:
Quote:
`System.NullReferenceException: 'Object reference not set to an instance of an object.'

signupFormIdElement was null.`
My WindowsForms application starts the main thread with a button click.

When I debugged the application I've find out that

var resp = (HttpWebResponse)req.GetResponse();

this expression returns null.

Edit:
After hours of debugging I finally found there is a socket error, and I forgot that 2 days ago I changed machine.config file to configure Fiddler as .NET proxy.