How does search engine spider crawl web pages ?
On this article, I will try to show How we can achieve similar task using a C# .NET.
The following code will read a web page using a virtual path and get the response then store it locally.
WebRequest request = WebRequest.Create(“https://ujjwaladatta.wordpress.com/2011/05/31/membership/”);
using (WebResponse response = request.GetResponse())
using (StreamReader responseReader = new StreamReader(response.GetResponseStream()))
string responseData = responseReader.ReadToEnd();
using (StreamWriter writer = new StreamWriter(@”C:\sample.html”))