Custom WCF REST service in SharePoint

Recently I have been working with Windows Communication Foundation (WCF) Service in SharePoint 2010. We built a jQuery AJAX heavy web application on top of the SharePoint platform, with business data exposed from a custom Microsoft SQL database. The solution should be accessed from within a SharePoint intranet site and within its security boundaries and SharePoint groups. Based on those requirements we chose WCF Service as communication architecture to expose the object model and the business objects.

WCF configuration

Windows Communication Foundation (WCF) is a unifying programming model for creating service oriented applications, that supports many hosting options and communications protocols. The technology may seem complex in terms of all the configuration possibilities. For this solution we have used a WCF JSON enabled REST service, which is a service configured to use the HTTP protocol. The requests and responses are text based (not SOAP) and the formatting is JSON. The serializing and deserializing of objects on both sides of the service is taken care of by the service, simply passing JavaScript objects on frontend and .NET objects on backend.

To set up a SharePoint project with WCF service some dll references needs to be added manually: System.ServiceModel, System ServiceModel.Web and System.Runtime.Serialization. To expose the service to run inside the SharePoint project an ISAPI mapped folder is added to the solution with a service file (.svc). This is a text file that contains information about the service that can be run using Microsoft Internet Information Services.

The svc file point to the implementation of the service:

<%@ ServiceHost Language="C#" Debug="true" Service="Myservice.Service,Myservice,
Version=1.0.0.0,Culture=neutral, PublicKeyToken=a0b806fa45919e70" %>

The service is defined by a service contract which is an interface class defining data and operations, and a service class which implements the contract. The service is a wrapper for business logic class which processes the requests and responses, and the data layer which accesses the database.

The configuration of the service is set up in web.config file:

<system.serviceModel>
  <serviceHostingEnvironment aspNetCompatibilityEnabled="true"/>
    <!-- Behavior setting for wcf  service -->
    <behaviors>
      <endpointBehaviors>
      <!-- configuration for rest relies on web http -->
        <behavior name="RestBehavior">
          <webHttp />
        </behavior>
      </endpointBehaviors>
    </behaviors>
  <services>
    <!-- register wcf service -->
    <service name="Myservice.Service">
      <endpoint address=""
        binding="webHttpBinding"
        behaviorConfiguration="RestBehavior"
        contract="Myservice.IService"
        bindingConfiguration="WindowsAuthenticationBasicHttpBinding">
     </endpoint>
    </service>
  </services>
  <bindings>
    <!-- webhttp binding for service-->
    <webHttpBinding>
      <binding name="WindowsAuthenticationBasicHttpBinding">
        <security mode="TransportCredentialOnly">
          <transport clientCredentialType="Windows" />
        </security>
      </binding>
    </webHttpBinding>
  </bindings>
</system.serviceModel>

<serviceHostingEnvironment aspNetCompatibilityEnabled=”true” />
Setting this value to true indicates that all WCF services running in the application run in ASP.NET Compatibility Mode. In WCF, a binding determines how WCF is going to communicate. For a RESTful endpoint, the binding is set to <WebHttpBinding>.  An endpoint behavior for the service is set to to enable the web programming model for WCF.

The communication channel for WCF is HTTP instead of SOAP, configured by <webHttpBinding> as binding configuration. This channel is configured with security on the transport channel, using windows credentials (which would be integrated SharePoint authentication). Note that clientCredentialType is set to “Windows”. Another option is “Ntlm”, but when Kerberos is used as authentication provider, only “Windows” would work.

The service itself is configured in the configuration with the service interface contract and the mapping to the binding endpoints and behaviors.

WCF implementation

The WCF service is defined with a service contact which specifies the operations the client can perform on the service.

[ServiceContract]
public interface IService
{
[OperationContract]

[WebInvoke(Method = "POST", BodyStyle = WebMessageBodyStyle.WrappedRequest,
ResponseFormat = WebMessageFormat.Json)]
String HelloWorld(string Message);
}

To pass a custom object as request or response specify a DataContract that agree on the exchange format. The datamembers must be serializable:

[DataContract]
public class Tag
{
[DataMember]
public Int32 TagID{ get; set; }

[DataMember]
public string KeyT { get; set; }
}

The service class implementation:

[AspNetCompatibilityRequirements(RequirementsMode =
AspNetCompatibilityRequirementsMode.Allowed)]
public class Service : IService {
...
}

Security and permissions

The application is hosted in a SharePoint site and uses SharePoint authentication. The WCF service is also configured to use the same security provider. To expose different functionality to different user groups, a set of SharePoint groups was created.  The access level is implemented both on fronted and backend, authenticating user against the specified groups. The service call itself is not doing the access check, except from the authentication. The business layer checks what access level the current user has and returns appropriate data. This is possible beacuse the service is running with the current user credentials.  It is also possible to configure permissions levels on the WCF itself by using role based security. This would be easy to setup for AD user groups,  but I’m not sure how it would work for SharePoint groups.

Testing WCF services

The code in the service itself is very simple; it does almost nothing because it is an interface and external point to the application. A service could do data validation and the main action is to transfer objects to call methods on the business model. But since the service contains code it is still prone to have defects and bugs like other parts of the application. Therefor it must be a good idea to do some validation and verification to ensure proper quality. And the idea was to do unit testing and to use Test Driven Development (TDD). But to make a WCF service testable and take advantage of TDD, the code has to be refactored to accept dependencies.

What made unit testing so difficult to start with is that the service has dependencies on the business and data access layers. When testing relies on resources like database it becomes unstable and hard to debug as there are dependencies through different layers.  In addition there were dependencies to SharePoint framework as well. This would result in an another type of test called integration test. So I ended up implementing an integration test even that was not the intention when I started. They are still a good value as you could test the overall system. But I experienced it to be very fragile and hard to detect errors because it would go through several layers and paths on system. There were also problems running the tests when the system was  depending on sharepoint authentication.

A unit test should test only one single specified requirement of the system.  I started looking at Pex and Moles framework for unit testing. There are other testing and mock frameworks like NUnit and Moq, but I needed a framework that is able to also mock SharePoint libraries. So I could do testing for the buisness and database layers. But I did not have time to do it properly because of the dependency injection learning curve. I have studied an example of using Ninject for WCF dependency injection, so maybe I will do a post on it later when I have tested it out.

The frontend

The frontend was implemented with Knockout,  a JavaScript MVVM inspired framework for separating view from model. Maybe another post on that later..  As a developer tool it is simply great. It uses JQuery data binding and templates to build rich and responsive user interfaces. We also did some unit testing on the frontend using QUnit framework. This part of the application was not my main responsibility, but was developed by one of my great colleagues

Sharepoint deployment with powershell

My first powershell script. The name says it all: it’s powerfull, you can do a lot with it. But what is Powershell:

Windows PowerShell is a new Windows command-line shell designed especially for system administrators. The Windows PowerShell includes an interactive prompt and a scripting environment that can be used independently or in combination.

Unlike most shells, which accept and return text, Windows PowerShell is built on top of the .NET Framework common language runtime (CLR) and the .NET Framework, and accepts and returns .NET Framework objects. This fundamental change in the environment brings entirely new tools and methods to the management and configuration of Windows.

Windows PowerShell introduces the concept of a cmdlet (pronounced “command-let”), a simple, single-function command-line tool built into the shell. You can use each cmdlet separately, but their power is realized when you use these simple tools in combination to perform complex tasks. Windows PowerShell includes more than one hundred basic core cmdlets, and you can write your own cmdlets and share them with other users.

Like many shells, Windows PowerShell gives you access to the file system on the computer. In addition, Windows PowerShell providers enable you to access other data stores, such as the registry and the digital signature certificate stores, as easily as you access the file system.

Working with Sharepoint, you can do most of the things STSADM does, and much more. A good starting point is the Powershell reference for Sharepoint by technet. Also a nice overview and collection of resources in this article by “sharepoint Joel”, from the very beginner to more advanced.
My first script simply installs all wsp file located in a dropfolder  (remove and uninstall first).
(more…)

Sharepoint 2010 – filter content on managed metadata term

We want to filter content on managed terms. Create a new column with metadata field type and bind it to a termset from the termstore. Create new content types with the metadata columns. Add some content based on it (lists, calender, pages) with different metadata. To retrieve data you for example use contentbyquerwebpart or Spquery. We want to set the query to a caml query filtered on a metadata field. A metadata field is a lookup type, and you can not write a query as it was a text field (althoug the term itself is text).

<Eq><FieldRef ID='" + metacolumnid + "' LookupId='TRUE' Nullable='TRUE'/>
<Value Type='Lookup'>" +id + "</Value></Eq>

The Fieldref ID is the GUID of the metadata column. The lookup value is the GUID of the term we want to filter on. Or it is not the GUID itself, but a lookup property: WssId.  This is an identifier (ID) of the list item that the lookup field points to. And how to find it? Well it took me some time to find out.. There is some hidden magic behind the scenes! All terms are actually stored in a list (everything in Sharepoint is lists, so not a big suprise . This list is hidden, but can be found: [site url]/Lists/TaxonomyHiddenList/AllItems.aspx.

Based on the IdForTerm we can retreive the wssid by TaxonomyField.GetWssIdsOfTerm

TaxonomyField.GetWssIdsOfTerm(site, _
	termStoreId, termSetId, termId, includeDescendants, _
	limit)

But we don’t want to hardcode the id values. To get the data more dynamically, we can create term store objects based on the default termstore and we probably know which group and termset we filter on (because a metadata column is bound to a termset).

   TaxonomySession session = new TaxonomySession(curSite);
   TermStore termStore = session.DefaultSiteCollectionTermStore;
   Group termGroup = termStore.Groups["MyGroup"];
   TermSet termSet = termGroup.TermSets["MyTermset"];
   String metacolumnid = "{603E67C1-99C9-4EB7-B4D8-A299DF4AF468}"; //metadata column id

 ............

    private int GetTermWssId(SPWeb rootWeb, string termtitle, TermStore termStore,
       TermSet termSet)
       {
           Guid guidid = new Guid();
           int result = 0;

           if (rootWeb.Properties.ContainsKey("TaxonomyHiddenList"))
           {
               Guid taxonomyHiddenListId =
                   new Guid(rootWeb.Properties["TaxonomyHiddenList"]);
               SPList taxonomyHiddenList = rootWeb.Lists[taxonomyHiddenListId];
               SPQuery query = new SPQuery();
               // we might have included the IdForTermSet in the query but we assume
               // that Guid is really unique
               // so there should not be temrs in other terms sets having the same ID
               string guid = termSet.Id.ToString();
               guid = guid.Replace("{", "");
               guid = guid.Replace("}", "");
               query.Query = String.Format(@"{0}{1}", termtitle, guid);
               // query.Query = String.Format(@"{0}", guid);
               SPListItemCollection items = taxonomyHiddenList.GetItems(query);
               if (items.Count == 1)
               {
                   guidid = new Guid(items[0]["IdForTerm"].ToString());
                   int[] wssIds = TaxonomyField.GetWssIdsOfTerm(rootWeb.Site,
                   termStore.Id,  termSet.Id, guidid, false, 1);
                   result = wssIds[0];
               }

           }
           return result;
       }

This is the solution I came up with, maybe it could be solved more elgant than this?

Another thing to remember.. To be able to query the metadata store and to get the above code to work, make sure the conent type syndication is activated on your site.

Sharepoint webpart error causing page to fail

It happens.. You have a webpart with error, causing the whole page to fail. What to do? You can quickly access the webpart and delete it through webpart maintance page [sitecoll]/?contents=1

Sharepoint 2010 – term store management

Managed metadata with term store management is an improvement of taxonomy management in Sharepoint 2010. This is a hierarchical collection of centrally managed terms that you can define, and then use as attributes for items in.
.A term set is a collection of related terms
Management of taxonomy takes place within the Term Store Management Tool, which is accessible through either Central Administration or Site Administration.

At top we have the term store, which can contain several term groups. A  term group defines a security boundary, and sets permissions for administration of terms. A group can contain several term sets (up to 1,000). A term set can contain terms (up to 30,000).  Managed metadata terms can be dfined in multiple languages. This is done by distinguishing the term itself from the words that represent the term. A label is a word or a phrase that represents a term. A term can have multiple labels

The management of the taxonomy service takes place in a Term Store management tool, available trough Central administration or site administration. First you need to create a managed metadata service, which sets up a database to be used as the term store, and a connection that  provides access to the service.

Go to central administration and manage service applications. Create a new managed service.

When the service is created select in the list and select the properties option from the ribbon bar.

In the properties edit window set the content hub by entering the url of the sitecollection you want to be the main term store service for yout site collections. Other site collections can subscribe to the content types and use the term store published from the hub site collection. It means you can share content types and terms across site collections. But first the content hub has to be published. Select the manage metadata service connection from the service list and then select properties from the ribbonbar. Check the publish settings below. Then publish the selected  service connection by publish from the ribbonbar.

The hub synchronization runs as a timer job every 15 minutes. If you want to speed up the publishing, you can run the Content Type Hub and Content Type Subscriber jobs manually from the Monitoring page of Central Administration.

To check which service a site collection is subscribing to, go to Site Collection Administration >Content Type Publishing.

New job

I started in a new job at Bouvet 1.oct, joining a Microsoft team with allmost 40 talented and really nice people. I’m very happy and proud to be a part of it. And I have started looking into Sharepoint 2010. Lots of fun and new challenges!

Summer

Summer and  vacation time! There has been few updates here lately.. and it will not be updated berfore after a long summer vacation

Playing with Surface

This post has been in a draft version for a couple of months now, so it´s time to publish..
So this year I have worked with some exciting technologies, including iPhone development, Microsoft Surface Table and now I’m investigating the IPad (which I owe . They have in common: multitouch screens, taking the user experience to the next level, it is easy and intuitive to use, making it possible to create visualizations of data with added value for the users. And this is the main reasons for why I think this is such an exciting area to work with, together with investigating new technology .

Surface table

Earlier this year we got or own Surface Table in our office at work, and I had the pleasure of playing with it for a while. This technology is not completly new and it has exsisted for some years now.
From Microsofts own site: Microsoft Surface is a revolutionary multi-touch computer that responds to natural hand gestures and real-world objects, helping people interact with digital content in a simple and intuitive way. With a large, horizontal user interface, Surface offers a unique gathering place where multiple users can collaboratively and simultaneously interact with data and each other. Surface table has some differences compared to a normal touch screens, it can read “tagged” objects and have multiple (52) touch spots which make a true Multi user experience and users can work together simultaneously. Another difference is of course that it is not only a screen, but a table standing on the floor where people can gather around. It has a 360-degree user interfaces having the content appear correctly for everybody.
The screen is a 30-inch XGA DLP® projector and the table is heavy ( almost 100kg!). The resolution is 1024 x 768px, which is the only thing that didn’t impress me..
The surface is really just a computer which runs Windows vista, so to develop applications you use .NET (WPF or XNA). This makes it easy for .NET developers to get started to play with it. To develop you can download a Surface Client, which is a simulator, but it can only be installed on Win Vista. I developed directly on the Surface, just connect a monitor and you use is as it was a computer and do your developement in Visual Studio as normal.

This how a standard byte tag looks like. Surface can recognize this identity tag. It is possible to make your own tags according to certain spesifications

You need to setup and get to  know the Surface SDK which includes the APIs, documentation, and tools to help you develop Surface touch-enabled applications. The core functionality of the sdk is the ScatterView and serves as a useful starting point for understanding how it works. To make some object move, rotate or resize you use the ScatterView control and add scatterview items containing your objects. It’s so simple, and you have already made a cool demo in no time!

I created a small simple demo app, which is a whiteboard with virtual sticker notes, which can be added and moved around and written to by the using the virtual keyboard. It can be used for project planning and so on. I also made a simple log in where employees can unlock the notes and edit information by entering his employe card upon the table (with a tag sticked to it for identifaction). I have a video showing this demo app and the source code, but I can´t find  it at the moment, because it has been some months ago since I worked with it. And end of story is that the table just a while ago somehow managed to get broken, someone slipped it onto the floor? Well it was not me, so it seems like there will not be any more development soon..

Surface demo app. Wish I had better photo or video

iPhone development

Since I have been working with iPhone development the lasts months, I will try to give a quick summary of my experiences so far, while still fresh in memory. I will not get into what you need to do before getting started, that´s another long story But you need an iPhone Developer license and a mac and the iPhone SDK setup and installed with XCode. And you need to learn Objective C which is the coding language. It may be a though start, but dont give up! A good starting point for learning is the Standford University iPhone Application Development podcasts on iTunes, or just check out this guide to all resources you need.

Some challenges

Storing photos
The iPhone sdk doesnt always provide us with the same functionality as found in the built in apps from Apple. One issue was that we wanted to use the camera from our application to take photos and store it in a photo album on the iphone with geotags. First problem: there is no way to create a new photo album by from an applicaton, so we had to store to the camera roll. Next: geotags EXIF data are stripped off photos when saved by UIImagePickerController into the photo library. Even though the geotags are provided by the camera by default. There may be workarounds, but I didn’t find a good and easy one. We could have stored the photo in the application document folder, but then we have to make our own gallery displaying the photos and methods to send it outside the iPhone. And there is no way to reference a photo in the camera roll from the application. So we couldn’t find a way to match a photo from our application with the iphone library, nor a way to delete it from the library after storing. In this first version of the app we ended up storing photos in the photoroll so the photos is available through iTunes. Next version needs improvements. One way could be to use a 3party photoview component, for example checkout the three20 and photoviewer component, also used by facebook.

Understanding gps and location accuracy
Determining the location is done by the Core Location framework. The location manager is the main responsible for polling location data and is pretty straightforward to work with. But the technologies used to deliver locations is hidden from your application. There are 3 possible ways to determine the position: GPS, cell tower triangulation and Wi-Fi positioning service (WPS). GPS is most accurate and WPS may be least accurate, depending on how much location data is registered in  your area. WPS is using the IP address from the iphones wi-fi connection to look up in a large database of known service providers (mathing MAC addresses). Which method the iPhone uses to determine the location you as a developer have no control of, but it will use the best method available. You can specify a desired accuracy setting on the location manager. But be aware of that a greater degree of accuracy will use more resources (battery). In this application we needed high accuracy, so the accuracy was set to 10m. But still we had problems getting accuracy, sometimes it was far off! When you start the location manager with startUpdatingLocation it will poll for new locations until you call stopUpdatingLocation. You should stop polling as soon as you got the accuracy information the application needs. The first response from the location manager is usally a cached location, the next is more accurate, the third is usually very accurate. But you have no control of how many responses you will get and the responses may be received non cronlogical in time.  So check the timestamp on the received location. Also check the properties horizontalAccuracy and verticalAccuracy on the location object to see if the last location has a better accuracy than the last one, and then store it. If you have received the accuracy you needed than stop updating, if not keep looking and set a timeout. This method made our location pretty accurate! But still, testing with different iphones, made different results so you can not guarantee it. While testing it is always a good idea to check the position on the buildt in map application, if that one is more accurate than yours, than probably you can do better too.

Interface Builder
IB is a visual editor for developers to create user interfaces and is a part of Xcode. The interface file is saved as a .xib file. Its pretty simple to work with as you just can drag and drop elements onto the stage. To connect events and map the elements to your code everything is done visual by dragging connectors. A bit tricky in the start if you never used this tool before. I started developing using IB but was told by other experienced developers that they would not use it! But as I often experience, some hard core developers often has this attitude and like to hate everything visual so I had to take a closer look at it. There are lots of debate on the internet about why or why not you should use it. I still believe it is a good practice to seperate design from code and I like to have that visual control of the design. IB seperates but also hides implementation behind the scenes and the connectors makes it easy to break something. And if you are using localized texts there is a bit more work to get xibs localized. There is no problem avoiding IB as you quickly will learn how to do everything by code. So I did not use IB, but I have not given it completly up yet.

Localization
You should consider to make your application localized. To do this you create localization folders in your project with extension .lproj. If you support norwegian and english, create two folders named no.lproj and en.lproj where you put your localized resource files for texts and so on. If the application is non localized it uses the development base language (set in property of the project). The base language also determines which language to use if none of the provided languages matches the user language setting. The user language is controlled by the language setting on the iPhone. In the distribution process to App store you are also localizing the descriptions and meta data of your application to be viewed on App store. Here I was a bit confused because Norwegian is not a supported option.

Data roaming
Not a real problem, but just be aware of it if you are developing applications that are supposed to by used in foreign countries and you need to use network resources. Users may have set data roaming setting to off because use of foreign network providers is usually very expensive. We wanted to show a message to the user that data roaming is off and because of this the application cannot be used properly. Is it possible to detect data roaming state with the iPhone SDK? I did find a way to check it, but the application is not allowed to set the state.

- (void)checkNetworkConnection
{
	NSURL *theURL = [ NSURL  URLWithString: @"http://www.google.com" ];
	NSURLRequest *request = [ NSURLRequest requestWithURL:  theURL
				cachePoliy: NSURLRequestReloadIgnoringCacheData
					timeoutInterval: 65536.0
							 ];
	[ NSURLConnection connectionWithRequest: request delegate: self ];
}

- (void)connection:(NSURLConnection *)connection didFailWithError:
(NSError *)error
{
	if ([error code] == -1018){ //NSURLErrorInternationalRoamingOff
	   // do something
        } else { // other network errors
	}
}

Launching Other Apps within an iPhone Application
Launch other apps with NSURL. Here is an overview of codes to use. We wanted to call a number from our application which will close our application and bring up the phone. When the call has ended we want to return to our application. This is possible when opening the url with [UIApplication sharedApplication].

NSString *telephoneURLString = [NSString stringWithFormat:@"telprompt:%@",
[self getPhoneNumber]]; // tel:
NSURL *telephoneURL = [[NSURL alloc] initWithString:telephoneURLString];
[[UIApplication sharedApplication] openURL:telephoneURL];
[telephoneURL release];

Distribution
Before distributing to App store we needed to share the application with testers and the customer. To do that we used Ad Hd distribution which allows you to share your application with up to 100 iPhones, and to distribute your application through email or by posting it to a web site or server. To prepare the application it needs to be compiled with a distribution certificate and provision file and each iPhone needs to be registered with the UDID identifier. These steps is well documented by Apple.

To distribute to app store you first create a app store certificate and distribution profile and then build your app with it. In your build settings specify which iPhone OS version you are supporting.  Set Base SDK to the version you are building the app with and deployment target to the lowest version number you are supporting. To find your app file go to products folder in xcode and reveal in finder. Package the app file/folder into a zip file. Be aware that no spaces are allowed in the zip file name.  To upload the binary log into Itunes connect or use the application loader.

If your app requires specific hardware to run, or you only want to target iphone and not ipod touch, you can set this in the info.plist file of the application. This is important, if you don’t specify any requirements your app will be tested on both iPhone and ipod touch by Apple.

Conclusion

The first weeks I was completely lost and almost wanted to give up, but now after 2 months it feels almost natural to write Objective C. First time you are doing an iPhone project it is a lot to learn: how to set up you project and certificates, learning a new programming language and SDK and then in the end understand the distribution process. Apple has documented well but there are lot of material to go through. So in your first project: calculate extra time!

I really got my eyes up for mobile development, it’s an exciting platform to work with where changes happens quickly. I would love to do more mobile development for sure. But still I want to be a .NET developer

2010

I have been thinking about this for a while now: I really need to update my blog!
2010 – a new year, lots of new possibilities and exciting technology to learn and play with.
I have been really busy lately at work, the last month I have been learning iPhone development while developing an iPhone application. It means working in a complete new environment and platform, learning Objective C, working on a Mac. Lots of fun, and some challenges So this is a jump in a new direction. Last year I have worked most of the time as part of a distributed scrum team across different countries, developing a completely new intranet solution on the Sharepoint platform – for 17 000 users! I have also done some prototyping projects, working with asp.net, Silverlight and FAST search. And some work related to Flash and Flex as well. We have recently got a Microsoft Surface table in our office, unfortunately this year has been to busy due to the iPhone project that I have not had time to play with it yet..

But 2010 is also going to be a good year, because of the next versions of SharePoint and Visual Studio! I would love to work with those products. I’m going to a seminar on monday and looking forward to hear Sahil Malik talk about SP 2010!

2010 also means.. I will update my blog more often? More to come..