;

Creating a Web Hook listener using ASP.NET

Posted : Wednesday, 20 January 2010 22:34:08

Below is my account of the technique I used to first analyze a set of request data containing a Web Hook and then process and act upon that data. If you're only interested in the processing of the data skip ahead to about halfway down where I detail the ProcessWebHook ActionResult method or click here.

I recently came across an excellent online source-code hosting product called Beanstalk - this is based on the open source subversion file tracking system that has been around for a while. There is a rich integration API made available by the BeanStalk guys and a few weeks ago I had cause to make use of it. I am currently building an MVC.NET application working remotely with a Mac using designer (aren't they all?). When I perform site upgrades I use a websetup project which copies all the files included in the solution to the development machine. If the designer adds any files to the subversion repository, they will get copied on to my machine when I update but crucially they will not be deployed to the webserver unless I include them in the solution.

Beanstalk provides a number of integration points that enable users to add certain functionality to various points in the project lifecycle. One such integration point is a Web Hook which, once correctly configured, will post a json formatted string to a user-supplied url every time a changeset is committed to the repository. The data sent contains details about the changeset including details of any added or modified files. The purpose of this post is to outline the process I used to receive this Web Hook, parse the data, and then act on it accordingly - in this case email me with a list of any added files so I could make sure to manually add the files to the Visual Studio solution.

On the Beanstalk site the Web Hook instructions state that a single post parameter called "commit" containing the json string. In order to set up the Web Hook, you need to provide a Url to which this information will be posted. Unfortunately I dont have a publically accessible webserver so I couldn't just point the Web Hook to my local machine, stick a breakpoint in the code and then commit a file change. I therefore decided on the following approach. I would configure the Web Hook to post to one of my hosting accounts and then record all of the data included in the request such that I could replay exactly the same request data during development.

In visual studio I created a new project from the MVC Application template and then deleted most of the files created by default. Given that there is no real publically viewable interface for this project I decided that getting rid of the master page would speed up the initial project construction. During development a simple interface was required to give me feedback but because the ultimate aim of this project was to create a web method to receive posted data, these pages required only minimal attention to page layout. The resultant project looked as follows:

default MVC projct strucuture with unnessary files removed

I built this project with testability in mind and although I'm not including any tests here, I may post in the future detailing how to write an accompanying unit test project. So with testing in mind, as well as to follow good practice, I have implemented component interaction using interfaces and will use the open source StructureMap project to resolve the dependencies (more on this later). The first interface I called IRequestLogger and it consisted of a single method:

    9     public interface IRequestLogger

   10     {

   11         void CaptureRequestData(HttpRequestBase request);

   12     }

Incidentally I created a subfolder of the Models directory called "Interfaces" and placed the file containing the interface definition into this folder. With the interface definition in place I added an instance of IRequestLogger as a private class-level variable to the HomeController class and then altered the default Index method of this controller as follows:

   36         public ActionResult Index()

   37         {

   38             //capture request data

   39             _RequestLogger.CaptureRequestData(Request);

   40 

   41             //message will be displayed to user

   42             ViewData["Message"] = "default page - request logged";

   43 

   44             //return basic view template

   45             return View();

   46         }

This is a pretty straighforward method. First it passes the current Request property (an instance of HttpRequestBase provided by the controller base class) to the current IRequestLogger.CaptureRequestData method. Next a small feedback message is added to to the ViewData collection and finally an instance of ViewResult is returned from the method. By default MVC will look for a page template with the same name as the current method, as mentioned above I deleted all the view templates that Visual Studio creates by default (including all shared views) so had to (re)create the Index.aspx view template to accompany this method. I did this by right-clicking within the method above and selecting "Add View" from the context menu. NB Given that I deleted the master page I needed to make sure I unchecked the "select a master page" option when creating any new Views. The new Index was very simple so doesn't warrant much explanation suffice to say it writes out whatever the current contents of ViewData["Message"] to the page, like I said above - a very simple UI.

So now to start capturing some request data. I decided to go down the route of storing the captured data in an XML file, both for speed of implementation and more importantly I could quickly open the file in any test editor and see exactly what had been captured. Most of the important information for the purposes of this task is stored a number of NameValueCollection objects of the HttpRequestBase class, the three I was interested in here are Request.Headers, Request.Post and Request.Querystring. I added a class to the Models directory, in it I implemeted the IRequest interface and then wrote the following code:

   14     public class SerializationUtility : IRequestLogger

   15     {

   16         private const string REQUEST_DATA_FILEPATH = @"/App_Data/requestdata.xml";

   17 

   18         #region IRequestLogger Members

   19 

   20         public void CaptureRequestData(HttpRequestBase request)

   21         {

   22             //load any previuous data from file

   23             var xDoc = LoadRequestData(HttpContext.Current.Server.MapPath(REQUEST_DATA_FILEPATH));

   24 

   25             //add <Request> element containg data for all the request collections we're interested in

   26             xDoc.Element("Requests").Add(

   27                 new XElement("Request",

   28                     CreateXmlFromNameValueCollection("Headers", request.Headers),

   29                     CreateXmlFromNameValueCollection("QueryString", request.QueryString),

   30                     CreateXmlFromNameValueCollection("Form", request.Form)

   31                 )

   32             );

   33 

   34             // finally save the document

   35             xDoc.Save(HttpContext.Current.Server.MapPath(REQUEST_DATA_FILEPATH));

   36         }

   37 

   38         #endregion

The LoadRequestData method is fairly simple and just looks for the specified file, if it doesn't exist it just returns a new instance of the XDocument class, I haven't listed the method here but it is available in the accompanying download. The new request data is appended as a new <Request> child node of the root <Requests> node. Linq to XML makes this a trivial task. Each NameValueCollection of interest is passed to the CreateXmlFromNameValueCollection method shown below:

   70         private static XElement CreateXmlFromNameValueCollection(string rootElementName, NameValueCollection nvc)

   71         {

   72             XElement element = new XElement(rootElementName);

   73             for (int i = 0; i < nvc.Count; i++)

   74             {

   75                 var key = nvc.Keys[i];

   76                 element.Add(

   77                     new XElement(key, nvc[key])

   78                 );

   79             }

   80             return element;

   81         }

As you can see it creates a new parent element and then loops through the collection, adding a subelement for each item. The new XElement is then returned to the calling function and finally the document is martinesaved (overwriting any previous file).

The last thing I needed to do in order to actually capture some data was to configure StructureMap to use the correct implementation of IRequestLogger. This is fairly straightforward and there are plenty of examples online of how to do this so I won't cover it except to say I added a constructor to the HomeController class to accept an Instance of IRequestLogger which was then used to set the class-level instance variable I added earlier:

   21         private IRequestLogger _RequestLogger;

   22 

   23         public HomeController(IRequestLogger requestLogger)

   24         {

   25             _RequestLogger = requestLogger;

   26         }

If you're following along using this post as a guide for your own implementation then provided you have configured StructureMap correctly you should now start capturing data as soon as you run the project and browse the site. If you have any problems please consult the available download or failing that post your issue using the link at the end of this article.

So now the request data was being captured I compiled the project, uploaded it to a virtual directory of my hosting account and pointed the Beanstalk Web Hook at the corresponding URL. I made a few changes to the subversion project, eg adding and deleting files as well as changing some and then committed these changes.

On inspection of the requestdata.xml file sure enough I found entries such as that below:

<Request>

  <Headers>

    <Connection>close</Connection>

    <Content-Length>466</Content-Length>

    <Content-Type>application/x-www-form-urlencoded</Content-Type>

    <Accept>*/*</Accept>

    <Host>thecodeface.co.uk</Host>

  </Headers>

  <QueryString />

  <Form>

    <commit>{"author_email":"dave@thecodeface.co.uk","author":"Dave","time":"2010/01/18 20:32:02 +0000","author_full_name":"David Lowe","message":"","revision":38,"changeset_url":"http://thecodeface.beanstalkapp.com/thecodeface/changesets/38","changed_dirs":[],"changed_files":[["trunk/posts/Document.txt","delete"]]}</commit>

  </Form>

</Request>

Once I had the exact format of the Web Hook data I added an ActionResult method and corresponding View template called PostedWebHooks to the controller to enable me to view any request data sent from Beanstalk. Next I created a method to retrieve all the saved Beanstalk data (the posted json string values from any submitted forms containing a "commit" element):

   83         public static IEnumerable<string> GetSavedBeanstalkWebHooks()

   84         {

   85             //load all request data

   86             var xDoc = LoadRequestData(HttpContext.Current.Server.MapPath(REQUEST_DATA_FILEPATH));

   87 

   88             //then find all saved requests that contained an element called "commit" within the posted form

   89             return from xel in xDoc.Descendants("Form").Descendants("commit")

   90                      select xel.Value;

   91         }

I then passed this data to the PostedWebHooks view where it is rendered as a series of postable forms. Doing this allowed me to repost this data as well as modify the payload and examine the effect this had. I haven't shown this page as it was only really useful during development, it is however available in the download.

Armed with the correctly formatted data I created an ActionResult method called ProcessWebHook on the HomeController class. As the name suggests this method comprises the main functionality of this project, ie to receive a Web Hook from Beanstalk, process the posted data and act on it accordingly. I decorated the method with the AcceptVerbs attribute to accept only post requests, I also took advantage of the default MVC model binding by making the method accept a string parameter named "commit":

   53         [AcceptVerbs(HttpVerbs.Post)]

   54         public ActionResult ProcessWebHook(string commit)

   55         {

Next I turned my attention to how to process the changeset data received in the json format. I created a class named ChangeSet to represent this data:

    7 namespace WebHookListener.Models

    8 {

    9     [DataContract]

   10     public class ChangeSet

   11     {

   12         [DataMember(Name="author")]

   13         public string Author { get; set; }

   14         [DataMember(Name="author_email")]

   15         public string AuthorEmail { get; set; }

   16         [DataMember(Name="author_full_name")]

   17         public string AuthorFullName { get; set; }

   18         [DataMember(Name = "time")]

   19         public string CommitTime { get; set; }

   20         [DataMember(Name = "changeset_url")]

   21         public string ChangeSetUrl { get; set; }

   22         [DataMember(Name = "changed_dirs")]

   23         public List<List<string>> ChangedDirs { get; set; }

   24         [DataMember(Name = "changed_files")]

   25         public List<List<string>> ChangedFiles { get; set; }

   26         [DataMember(Name = "member")]

   27         public string Message { get; set; }

   28         [DataMember(Name = "revision")]

   29         public int Revision { get; set; }

   30     }

   31 }

Decorating the class and its members with the DataContract and DataMember attributes respectively meant it became a relatively simple matter of using the DataContractJsonSerializer class to create instances of this class from the corresponding data. Incidentally the DataContractJsonSerializer is located within the System.Runtime.Serialization.Json namespace of the System.Runtime.Web assembly so I added a project reference to that file. Next I added a method to the SerializationUtility class:

   94         public static ChangeSet GetChangelistFromJsonString(string json)

   95         {

   96             //wrap in a try/catch block

   97             try

   98             {

   99                 //create DataContractJsonSerializer for deserializing

  100                 var serializer = new DataContractJsonSerializer(typeof(ChangeSet));

  101                 //load json data into memorystream

  102                 MemoryStream ms = new MemoryStream(Encoding.Unicode.GetBytes(json));

  103                 //deserialize

  104                 ChangeSet changeList = serializer.ReadObject(ms) as ChangeSet;

  105                 //close the stream

  106                 ms.Close();

  107                 //return deserialized instance

  108                 return changeList;

  109             }

  110             catch (Exception e)

  111             {

  112                 return null;

  113             }

  114         }

It was then a simple matter of calling this deserialization method from the ProcessWebHook method, passing the commit variable containing the json string as the target of deserialization:

   56             //use serialization class to parse changeList

   57             ChangeSet changeList = SerializationUtility.GetChangelistFromJsonString(commit);

At this point I had an in-memory representation of the changeset data so the next step was to decide what to do with it. I chose to send an email so wanted someway to represent the changeset as a HTML string. I added a method called ToHtml() to the ChangeSet class and in order to test this I created a very simple View called "ChangeSetDetails.aspx" which simply called the ToHtml() method for the changeset, this View template is returned from the ProcessWebHook method. Once I was happy with the HTML I began work on the final piece of functionality, namely processing the ChangeSet instance and sending an email. It occurred to me early on that I might want to change the action taken on receiving a Web Hook, for example I might want to post the data to twitter or maybe log to a database etc. For this reason I added another interface to the project called IChangeSetProcessor shown below:

    8     public interface IChangeSetProcessor

    9     {

   10         void ProcessChangeList(ChangeSet changeList);

   11     }

Included in the download is a very simple implementation of IChangeSetProcessor which creates a new smtp mail message object, writes the output of the ChangeSet.ToHtml() method to the message body and then sends it. I followed the same procedure as for IRequestLogger and altered the HomeController constructor to accept an instance of this interface and store it as class level variable. The complete ProcessWebHook method is shown below:

   53         [AcceptVerbs(HttpVerbs.Post)]

   54         public ActionResult ProcessWebHook(string commit)

   55         {

   56             //use serialization class to parse changeList

   57             ChangeSet changeList = SerializationUtility.GetChangelistFromJsonString(commit);

   58 

   59             //then hand off to IChangeListProcessor for processing

   60             _ChangeSetProcessor.ProcessChangeList(changeList);

   61 

   62             //return basic view template

   63             return View("ChangeSetDetails", changeList);

   64         }

So that's essentially it, the final steps necessary to 'switch it on' were to configure StructureMap to use the correct implementation of IChangeSetProcessor and point the Beanstalk Web Hook to post to the URL of the ProcessWebHook method (by default this will be ~/home/ProcessWebHook). I deployed this code a week or two ago and it has been working excellently since.

Subscribing to a Web Hook enables any user defined action to be performed everytime a Web Hook enabled event occurs. I configured StructureMap using the web.config file which is generally considered to be rather antiquated (the current vogue is to fluently configure via code) but I took this choice as I consider it a distinct possibility that I may wish to change the implementation of IChangeSetProcessor. Using the config file means I could easily create the new implementation in a seperate assembly, update the web.config and then Bin-deploy both files without requiring recompilation of the main site dll. I hope this post proves helpful, let me know if so or you think of any improvements/refinements.

  • (This will not appear on the site)