c#: crawler project here is the Answer:

Hi,
Could I get very easy to follow code examples on the following:

1. Use browser control to launch request to a target website.
2. Capture the response from the target website.
3. convert response into DOM object.
4. Iterate through DOM object and capture things like "FirstName" "LastName" etc if they are part of response.

thanks

Answer is:


Here is code that uses a WebRequest object to retrieve data and captures the response as a stream.

    public static Stream GetExternalData( string url, string postData, int timeout )
    {
        ServicePointManager.ServerCertificateValidationCallback += delegate( object sender,
                                                                                X509Certificate certificate,
                                                                                X509Chain chain,
                                                                                SslPolicyErrors sslPolicyErrors )
        {
            // if we trust the callee implicitly, return true...otherwise, perform validation logic
            return [bool];
        };

        WebRequest request = null;
        HttpWebResponse response = null;

        try
        {
            request = WebRequest.Create( url );
            request.Timeout = timeout; // force a quick timeout

            if( postData != null )
            {
                request.Method = "POST";
                request.ContentType = "application/x-www-form-urlencoded";
                request.ContentLength = postData.Length;

                using( StreamWriter requestStream = new StreamWriter( request.GetRequestStream(), System.Text.Encoding.ASCII ) )
                {
                    requestStream.Write( postData );
                    requestStream.Close();
                }
            }

            response = (HttpWebResponse)request.GetResponse();
        }
        catch( WebException ex )
        {
            Log.LogException( ex );
        }
        finally
        {
            request = null;
        }

        if( response == null || response.StatusCode != HttpStatusCode.OK )
        {
            if( response != null )
            {
                response.Close();
                response = null;
            }

            return null;
        }

        return response.GetResponseStream();
    }

For managing the response, I have a custom Xhtml parser that I use, but it is thousands of lines of code. There are several publicly available parsers (see Darin's comment).
EDIT: per the OP's question, headers can be added to the request to emulate a user agent. For example:
request = (HttpWebRequest)WebRequest.Create( url );
                request.Accept = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, */*";
                request.Timeout = timeout;
                request.Headers.Add( "Cookie", cookies );

                //
                // manifest as a standard user agent
                request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US)";


Force local user to change password at next login with C#

I'm writing a function for a web app in ASP.NET where the client logs into the server machine, which is Windows authenticated against the local users on the server. The function I am writing resets the users password and emails them the new one. I do this like so:

String userPath = "WinNT://"  + Environment.MachineName + "/" + username.Text;
DirectoryEntry de = new DirectoryEntry(userPath);
de.Invoke("SetPassword", new object[] { password });

How can I also check the flag to force the user to change their password the next time they log in with the password emailed to them? I tried using pwdLastSet like so:

de.Properties["pwdLastSet"].Value = 0;

But this apparently only works with LDAP, not WinNT, and I am doing this locally.
Any experts know any better than me? I have even tried looking for a way to do this through the command line so that I can just create a Process, but I haven't been able to find a way to do it that way, either.


Answer is:

For WinNT, you must set the value to 1 rather than 0, and the property name is "PasswordExpired" rather than "pwdLastSet"; see http://msdn.microsoft.com/en-us/library/aa746542(VS.85).aspx
In other words, do this for WinNT:

de.Properties["PasswordExpired"].Value = 1;

(It is confusing, I know, but for LDAP you need to set the property "pwdLastSet" to 0. How's that for inconsistency!)

Get DisplayName Attribute without using LabelFor Helper in asp.net MVC

What is the best way to retrieve the display name attribute for an item in your model? I see a lot of people using the LabelFor helper for everything, but a label isn't appropriate if I just want to list the data out. Is there an easy way just get the Name Attribute if I just want to print it out in, say a paragraph?

Answer is"

<p>
    <%= Html.Encode(
        ModelMetadata.FromLambdaExpression<YourViewModel, string>(
            x => x.SomeProperty, ViewData).DisplayName
    ) %><p>

Obviously in order to avoid the spaghetti code it is always a good idea to write a helper:

public static class HtmlExtensions
{
    public static MvcHtmlString GetDisplayName<TModel, TProperty>(
        this HtmlHelper<TModel> htmlHelper, 
        Expression<Func<TModel, TProperty>> expression
    )
    {
        var metaData = ModelMetadata.FromLambdaExpression<TModel, TProperty>(expression, htmlHelper.ViewData);
        string value = metaData.DisplayName ?? (metaData.PropertyName ?? ExpressionHelper.GetExpressionText(expression));
        return MvcHtmlString.Create(value);
    }
}

And then:

<p>
    <%: Html.GetDisplayName(x => x.SomeProperty) %></p>