Updating Search Index Manually/Search Index Fields - Front- & Back-End Development - Front- & Back-End Development - Progress Community
 Front- & Back-End Development

Updating Search Index Manually/Search Index Fields

  • Updating Search Index Manually/Search Index Fields
  • Hi,

    We are working through custom content types and publishing pipes with 4.2. I have two questions:

    1) How do I go about triggering an incremental update to a search index? I have an inbound pipe hooked up to the "SearchItemTemplate" pipe template and it works fine when reindexing through the UI. I can't figure out how to hook into the pipe, however, to add or delete an item individually. I've tried grabbing a handle to the pipe using PublishingSystemFactory.GetPipe(s) when executing the ContentManagerBase.Publish() method of the custom content but everything in the pipe seems to be null — pipesettings, PublishingPoint, etc.

    2) In Sitefinity 3.7 I knew how to add custom fields to the Lucene index so that I could do a targeted search. In 4.2 I'm not sure where to do that or if it's even possible. Somehow the inbound pipe is hooked up to Lucene using the SearchItemTemplate but I can't find a way to manipulate the fields that Lucene generates in the index.

    Thank you.


    Michael
  • Hi Michael,

    1) There is no way to trigger an incremental update to a search index at the moment.

    In order to trigger a pipe you need to call method

    public virtual void Initialize(PipeSettings pipeSettings), pipeSettings can be retrieved from publishing point or via Publishing manager like this:

    pipeSettings = PublishingManager.GetManager(provider.Name).GetPipeSettings<SearchIndexPipeSettings>().Where(ps => ps.PublishingPoint.Id == pointGuid).FirstOrDefault();

    After the pipe is initialized you can call method

     public virtual void PushData(IList<PublishingSystemEventInfo> items) 

    I need to make proof of concept for extending search fields at Lucene and I need more time. I will provide you with a sample code when I am ready.

    Let me know if you still have issues with first point.

    Greetings,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  • Hello Michael,

     About your second question:

    1) I created a custom field for news module via user interface with name "TestSearch".

    2) Inherited  ContentInboundPipe and overrided property PipeSettings.

    public class ContentInboundPipeNew : ContentInboundPipe
       
           public override Sitefinity.Publishing.Model.PipeSettings PipeSettings
           
               get
               
                   var settings = base.PipeSettings;
                              var mappings = settings.Mappings.Mappings;
                   var customField = "TestSearch";
                   var customFieldMapping = PublishingSystemFactory.CreateMapping(customField, ConcatenationTranslator.TranslatorName, true, customField);
                   if(!mappings.Contains(customFieldMapping))
                     mappings.Add(customFieldMapping);
                  
                   return settings;
               
               set
               
                   base.PipeSettings = value;
               
           
       

    3) Inherited SearchIndexOutboundPipe and overrided property PipeSettings.

    public class SearchIndexOutboundPipeNew : SearchIndexOutboundPipe
        public override Sitefinity.Publishing.Model.PipeSettings PipeSettings
        
            get
            
                var settings = base.PipeSettings;
                                                   var mappings = settings.Mappings.Mappings;
                var customField = "TestSearch";
                var customFieldMapping = PublishingSystemFactory.CreateMapping(customField, ConcatenationTranslator.TranslatorName, true, customField);
                if (!mappings.Contains(customFieldMapping))
                    mappings.Add(customFieldMapping);
     
                return settings;
            
            set
            
                base.PipeSettings = value;
            
        

    4) Replaced SearchIndexOutboundPipe and ContentInboundPipe with newly created pipes. I did this at Global.asax like this:

    protected void Application_Start(object sender, EventArgs e)
        
            Bootstrapper.Initialized -= Bootstrapper_Initialized;
            Bootstrapper.Initialized += Bootstrapper_Initialized;
        
     
        protected void Bootstrapper_Initialized(object sender, Telerik.Sitefinity.Data.ExecutedEventArgs e)
        
            if (e.CommandName == "Bootstrapped")
            
                this.RegisterCustomSearchFieldMapping();
            
        
     
        public void RegisterCustomSearchFieldMapping()
        
           PublishingSystemFactory.UnregisterPipe(SearchIndexOutboundPipe.PipeName);
            PublishingSystemFactory.RegisterPipe(SearchIndexOutboundPipe.PipeName, typeof(SearchIndexOutboundPipeNew));
     
            PublishingSystemFactory.UnregisterPipe(ContentInboundPipe.PipeName);
            PublishingSystemFactory.RegisterPipe(ContentInboundPipe.PipeName, typeof(ContentInboundPipeNew));
        

    These steps will ensure that added custom field is indexed at Lucene.

    In order to be able to search by field "TestSearch"  you need to make some further steps:

    1) You need to inherit class LuceneResultSet and override method ExecuteQuery:

    protected virtual void ExecuteQuery(bool tryToFix = true)
           
               try
               
                   var path = this.provider.GetCataloguePath(catalogueName);
                   if (Directory.Exists(path))
                   
                       var defautlOperator = QueryParser.AND_OPERATOR;
                       var analyzer = this.GetAnalyzer();
                        
                       var parser = new MultiFieldQueryParser(new string[] "Title", "Content", "TestSearch", analyzer);
                       parser.SetDefaultOperator(defautlOperator);
     
                       //query = QueryParser.Escape(query);
     
                       if (searcher != null)
                           searcher.Close();
     
                       searcher = new IndexSearcher(path);
                       Query queryObj = searcher.Rewrite(parser.Parse(query));
                       Query = queryObj;
     
                       //Search only for current language
                       if (AppSettings.CurrentSettings.Multilingual == true)
                       
                           TermQuery languageQuery = new TermQuery(new Term("Language", CultureInfo.CurrentUICulture.Name));
                           TermQuery languageNullQuery = new TermQuery(new Term("Language", SearchProvider.FieldNullValue));
     
                           BooleanQuery filterQuery = new BooleanQuery();
                           filterQuery.Add(languageQuery, BooleanClause.Occur.SHOULD);
                           filterQuery.Add(languageNullQuery, BooleanClause.Occur.SHOULD);
     
                           QueryFilter languageFilter = new QueryFilter(filterQuery);
                                                    
                           this.hits = searcher.Search(queryObj, languageFilter);
                       
                       else
                           this.hits = searcher.Search(queryObj);
                       
     
                       for (var iter = 0; iter < this.HitCount; iter++)
                       
                           var doc = this.hits.Doc(iter);
                       
                   
               
               catch (Exception ex)
               
                   if (tryToFix && ex is ParseException)
                   
                       this.query = this.TryFixQuery(this.query);
                       this.ExecuteQuery(false);
                   
                   else
                   
                       this.error = ex;
                       this.hits = null;
                   
               
           

    2) You need to inherit LuceneSearchProvider class and override methods Find:
    public class LuceneSearchProviderNew : LuceneSearchProvider
        
            public override IResultSet Find(string catalogueName, string query, int skip, int take, params string[] orderBy)
            
                return new LuceneResultSetNew(this, catalogueName, query, skip, take, orderBy);
            
     
            public override IResultSet Find(string catalogueName, string query, params string[] orderBy)
            
                return new LuceneResultSetNew(this, catalogueName, query, orderBy);
            
        

    3) You need to replace LuceneSearchProvider with newly created provider via configuration.

    Let me know if you have any issues with achieving required scenario!


    Greetings,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  • Milena,
    I'm trying to search by a custom field, "Brand", but no results are returned when i try it out. I copied the code sample that you posted (exchanging "TestSearch" with "Brand"). One part I wasn't sure of is configuring the new custom search provider. I'm using my code below (actually from another post). Should that configuration code work? Does configuring the search provider need to come before or after registering the new inbound and outbound pipes?

    <BR>
    void configureSearchProvider()
          var section = Config.Get<SearchConfig>();
          foreach (var provider in section.Providers.Values)
             if (provider.Name == "LuceneSearchProvider")
                if (provider.ProviderType == typeof(LuceneSearchProvider))
                   provider.ProviderType = typeof(LuceneSearchProviderNew);
                   ConfigManager.GetManager().SaveSection(section);
                   break;
                
             
          
       

  • 340983_SearchProviderConfig.jpg
    Hello Casey ,

    you can replace the old search provider and verify that it is replaced via Sitefinity configuration interface.

    Under Administration > Settings > Advanced > Search > Providers> replace old LuceneSearchProvider.
    Screenshot attached. 

    Let me know if you still have any issues with customizing search.
    Thanks!

    Greetings,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  • Thanks Milena,

    I registered the LuceneSearchProviderNew. But the search still isn't including the custom field, "Brand".
    I put a break point in the LuceneSearchProviderNew.Find method and can see that the LuceneResultSetNew is returned. However the breakpoint in LuceneResultSetNew.ExecuteQuery never gets reached.
    Do you have any ideas what might be wrong?

    Any help would be appreciated,
    Casey
  • Hi Casey,

    I think that problem is at LuceneResultSet,  ExecuteQuery method declaration should be: 

    protected override void ExecuteQuery(bool tryToFix = true), in order to replace the behaviour,  otherwise old code is executed.

    Let me know if this helps! Thanks!

    Regards,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  • thanks again Milena,
    That was the reason ExecuteQuery wasn't being called; I should have noticed that.

    So now my break point in ExecuteQuery is hit, but searching by my custom field "Brand" still doesn't work. If I search by Title or Content then the search works. It looks like the query is set correctly and i checked that products have the Brand term I'm searching for but still no results.

    I noticed that my breakpoint in my ContentInboundPipeNew.PipeSettings (which I copied from your post) never gets hit even though I register that pipe same as the outbound one. I don't know if that is an issue.

    Do you have any ideas why searching by Brand isn't working? Or know of anything else I should check?

    I posted the ExecuteQuery method below in case there is an issue there. It should be the same as the code you  posted here except we don't need the multilingual part yet and we're using the default "or" operator (but i tried it with "and" too and it still didn't work).

    public class LuceneResultSetNew : LuceneResultSet
       private Exception error;
       private string query;
       private IndexSearcher searcher;
     
       public LuceneResultSetNew(LuceneSearchProvider provider, string catalogueName, string query, params string[] orderBy)
          : this(provider, catalogueName, query, 0, 0, orderBy)
     
       
     
       string catalogueName = String.Empty;
       public LuceneResultSetNew(LuceneSearchProvider provider, string catalogueName, string query, int skip, int take, params string[] orderBy)
          : base(provider, catalogueName, query, 0, 0, orderBy)
          this.catalogueName = catalogueName;
          this.query = query;
       
     
       protected override void ExecuteQuery(bool tryToFix = true)
       
          try
             var provider = SearchManager.GetManager("").Provider as LuceneSearchProvider;
             var path = provider.GetCataloguePath(catalogueName);
     
             if (System.IO.Directory.Exists(path))
                var analyzer = this.GetAnalyzer();
                var parser = new MultiFieldQueryParser(new string[] "Title", "Content", "Brand" , analyzer);
      
                if (searcher != null)
                   searcher.Close();
      
                searcher = new IndexSearcher(path);
                Query queryObj = searcher.Rewrite(parser.Parse(query));
                System.Diagnostics.Debug.WriteLine(queryObj.ToString());
                Query = queryObj;
                this.hits = searcher.Search(queryObj);
      
                for (var iter = 0; iter < this.HitCount; iter++)
                   var doc = this.hits.Doc(iter);
                
             
          
          catch (Exception ex)
             if (tryToFix && ex is ParseException)
                this.query = this.TryFixQuery(this.query);
                this.ExecuteQuery(false);
             
             else
                this.error = ex;
                this.hits = null;
             
          
       
     
       public void Dispose()
          if (this.searcher != null)
             this.searcher.Close();
             this.searcher = null;
          
       

  • 341780_Search.zip
    Hi,

    I tested with version Sitefinity 4.2 and I managed to index and search by custom field "brand" which is added to News module.
    If you do not break at ContentInboundPipeNew, this means that you didn't replaced the ContentInboundPipe,  therefore the "Brand" field is not indexed.

    I am providing sources of the solution, if this do not work for you let me know!

    Regards,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  • thanks Milena,

    Will the sample you posted search different product types with a custom field "Brand"?

    Casey
  • Hi,

    the provided sample will work with Sitefinity content types - news, events, blogs, etc.

    For product types you need to replace the product pipe with new pipe(inherit the ProductInboundPipe) and at the new pipe add additional mappings to product pipe for "brand" field, the same way it is done at ContentInboundPipeNew.

    Let me know if you have any issues.

    Regards,

    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  • Thanks for your help Milena. I got searching products by custom field "brand" working.

    I registered a new product inbound pipe like you said. I also had to move the new product inbound pipe registration from the bootstrapper_initialized event handler to a later event because the default product inbound pipe wasn't registered yet when i was handling the bootstrapper intialized event.
  • The only field that I am not able to search by that I need to is "Sku". Is there anything different I need to do in order to search by "Sku"?

    I noticed that product.DoesFieldExist("Sku") is true, but product.FieldValue<object>("Sku") throws and argument exception.
    No such persistent field known.
    Parameter name: nameOfPersistentField
    Actual value was Sku.

    Sorry for all the questions but I can't find information on these specifics anywhere else.
  • Hi Casey,

    In order to be able to index and search by "Sku" product field the steps are the same:

    1) add "Sku" field to mappings at class that override ProductInboundPipe (ProductInboundPipeNew) and mappings of  SearchOutboundPipeNew

    2) LuceneResultSetNew add "Sku" field in order to be able to  search by new field:
    var parser = new MultiFieldQueryParser(new string[] "Title", "Content", "Brand", "Sku" , analyzer);

    I tested step1 and the product "Sku" field was indexed successfully.

    Regards,

    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items