Customizing Content Processing Pipeline in SharePoint 2013

Often there is a need to customize the content processing pipeline to meet certain business requirements. There are various approaches to this that are discussed in this blog.

The figure below is showing the logical overview of how crawling and content processing works for SharePoint 2013 Enterprise search.

Now if requirement is such that the managed properties of crawled items need to be modified before being indexed then customized business logic needs to be implemented somewhere in the content processing pipeline. The only place where SharePoint 2013 allows us to call external SOAP services (wcf and web services) is during “Content Enrichment”.

In this article, 2 cases will be discussed:

Case 1:

“Calcutta” is a city in India, recently renamed as “Kolkata”. Now some people may search with “Kolkata” and others with “Calcutta”. Here the “Location” property can be modified to “Kolkata” whenever “Calcutta” entry is found against “Location”. For this the following steps are needed:

1. Create a wcf application in Visual Studio 2012 and add a reference to “Microsoft.office.server.search.contentprocessingenrichment.dll” which you can find in “c:program filesMicrosoft office servers15.SearchApplicationExternal”.
2. Delete default interface (e.g. IService1)
3. Add following references to Service1.svc.cs file
   • Microsoft.office.server.search.contentprocessingenrichment
   • Microsoft.office.server.search.contentprocessingenrichment.PropertyTypes
4. Inherit “Icontentporcessingenrichmentservice” in Service1.svc.cs file
5. Implement the method “ProcessItem”. This is the method where you get required properties for each items.
6. Add following in <system.servicemodel> in web.config file<bindings><basicHttpBinding><!– The service will accept a maximum blob of 8 MB. –><binding maxReceivedMessageSize = “8388608”><readerQuotas maxDepth=”32″maxStringContentLength=”2147483647″maxArrayLength=”2147483647″maxBytesPerRead=”2147483647″maxNameTableCharCount=”2147483647″ /><security mode=”None” /></binding>

   </basicHttpBinding>

   </bindings>

7. Host this wcf to IIS (Create a virtual directory). Map this to the physical path of wcf application. Right click on the Virtual Directory and click on “Convert to Application”
8. Browse and get the url for hosted .svc file
9. Execute following PowerShell script to map “Content Enrichment” to hosted custom wcf$ssa = Get-SPEnterpriseSearchServiceApplication$config = New-SPEnterpriseSearchContentEnrichmentConfiguration$config.Endpoint = http://Site_URL/<service name>.svc$config.InputProperties = “Location”$config.OutputProperties = “Location”$config.SendRawData = $True$config.MaxRawDataSize = 8192Set-SPEnterpriseSearchContentEnrichmentConfiguration –SearchApplication$ssa –ContentEnrichmentConfiguration $config
10. Run a full crawl on the content source.

Case 2:

In case of more than one content source, one can be configured for “Advanced” and the other for “Intermediate” resources. Here “Author” property of one can be modified to “Advanced” and the other to “Intermediate” so that when users search using “Advanced” key word they can see files from first content source and vice versa.

Here the two different content sources are segregated first and then processed differently. That is why there is a need of a “WCF Router” to identify the content source and map to wcfs accordingly.

• Create two wcfs as directed in Case 1
• Create a WCF Application and open web.config and configure as described below :
• Here “basicHttpBinding” is being used<basicHttpBinding><bindingname=”basicHttpBinding_IContentProcessingEnrichmentService”maxReceivedMessageSize = “8388608”><readerQuotasmaxDepth=”32″maxStringContentLength=”2147483647″maxArrayLength=”2147483647″maxBytesPerRead=”2147483647″maxNameTableCharCount=”2147483647″ /><security mode=”None” />

  </binding>

  </basicHttpBinding>

• Then configure Services section (base address not required if service is hosted in IIS)<services><service behaviorConfiguration=”RoutingServiceBehavior” name=”System.ServiceModel.Routing.RoutingService”><endpoint name=”RoutingServiceEndpoint” address=”” binding=”basicHttpBinding” bindingConfiguration=”basicHttpBinding_IContentProcessingEnrichmentService” contract=”System.ServiceModel.Routing.IRequestReplyRouter”/></service>
• A service behavior needs to be created where the name of the filter table is referenced. This will be defined in the next step. To enable full inspection of the SOAP envelopes in the XPath filters, the attribute “routeOnHeadersOnly” is set to false.<behavior name=”RoutingServiceBehavior”><routingfilterTableName=”ContentSourceFilters”routeOnHeadersOnly=”False”/></behavior>
• Now configure client wcfs<client><endpoint name=”ContentProcessingEnrichmentService” address=”http://localhost:300/ContentProcessingEnrichmentService/Service1.svc” binding=”basicHttpBinding” bindingConfiguration=”basicHttpBinding_IContentProcessingEnrichmentService” contract=”*”/><endpoint name=”ContentProcessingEnrichmentServiceDB” address=”http://localhost:300/ContentProcessingEnrichmentServiceDB/Service1.svc” binding=”basicHttpBinding” bindingConfiguration=”basicHttpBinding_IContentProcessingEnrichmentService” contract=”*”/></client>
• Now “Routing” will be configured i.e. the mapping section between wcf and contentsource. Here the filters and the filter table are defined to map the filters to normal endpoints (and optionally to backup endpoints). Here “Xpath” filtertype is used as e XPath expressions look for all Property nodes in the SOAP envelope.<routing><namespaceTable><add prefix=”cc” namespace=”http://schemas.microsoft.com/office/server/search/contentprocessing/2012/01/ContentProcessingEnrichment”/></namespaceTable><filters><filter name=”Sharepoint” filterType=”XPath” filterData=”//cc:Property[cc:Name[. = ‘ContentSource’] and cc:Value[. = ‘Local SharePoint sites’]]”/><filter name=”WCMContent” filterType=”XPath” filterData=”//cc:Property[cc:Name[. = ‘ContentSource’] and cc:Value[. = ‘WCM’]]”/></filters><filterTables><filterTable name=”ContentSourceFilters”>

  <add filterName=”Sharepoint” endpointName=”ContentProcessingEnrichmentService”/>

  <add filterName=”WCMContent” endpointName=”ContentProcessingEnrichmentServiceDB”/>

  </filterTable>

  </filterTables>

  </routing>

• Build solution and host to IIS as case 1
• Execute following PS commands to integrate routing.svc (router service) to content enrichment$ssa = Get-SPEnterpriseSearchServiceApplication$config = New-SPEnterpriseSearchContentEnrichmentConfiguration$config.Endpoint = “http://localhost:300/Router/Router.svc”$config.InputProperties = “Author”, “Filename”, “ContentSource”$config.OutputProperties = “Author”$config.SendRawData = $True$config.MaxRawDataSize = 8192Set-SPEnterpriseSearchContentEnrichmentConfiguration –SearchApplication $ssa –ContentEnrichmentConfiguration $config
• Run a full crawl on both content sources.

Hope you find this article interesting and helpful. Let us know in the comments below if you have any questions.