Deploy Services Failed on Server Startup - Forum - Corticon - Progress Community

Deploy Services Failed on Server Startup

 Forum

Deploy Services Failed on Server Startup

  • Hi,

    We've Corticon Server instaled in Azure (programmed to restart every day) and we're facing the followin problem.


    Context:
    Some Services are using EDC to connect Oracle Database.

    At some days on Server start up something fails on Oracle Listener and causes Corticon Services that use that EDC were undeploy. Services that don't use EDC were correctly deployed.

    Is there a way to force Service Deploy without validating the EDC?

    Thanks,
    Ricardo Jorge

  • Hi Ricardo,

    The same behaviour can be seen when the datamodel of a database is altered, without importing the metadata in Studio after that database-change and re-publishing the Decision Service.

    As far as I know there is no fix yet, but I'll leave it to the dev-team to answer that.

    -Gertjan

  • Hi, Gerjan,

    Thanks for the replying.

    Is there a way of using a command line to check if services are deployed and if not deploy them. We was thinking on a Schedule task that time to time checks it and deploy if necessary.

    This enviromnent have to be available all days and we have to create a work around for this behaviour.

    Thanks,

    Ricardo Jorge

  • Hi Ricardo,

    The quick and dirty option that I see to work around this, would be to create a copy of a valid ServerState.xml with all Decision Services succesfully deployed. In case the Oracle Listener was unsuccesful, Corticon Server will clear the ServerState.xml, with no deployed Decision Services as a result. The ServerState.xml can be found in ~\CorticonWork\SER\CcServerSandbox\DoNotDelete\.

    Perhaps your scheduled task can check the ServerState.xml to see if it is invalid (no decision service present), stop Corticon Server, replace the faulty ServerState.xml with the backed-up valid ServerState.xml, and restart Corticon Server again when the Listeners are up and running.

    I hope this makes sense.

    -Gertjan

  • Ricardo,
     
    The testserveraxis.bat utility can be used to inspect a Corticon server but it’s an interactive tool and not good in a script. However the Corticon server REST and SOAP APIs can be used to do what you want. For the REST APIs have a look at the swagger documentation (change the host and port to your Corticon server)
     
     
    For example the API
     
     
    Lists deployed decision services. Using curl you can get a list such as
     
    {
      "decisionServices": [
        {
          "loadedFromCdd": true,
          "name": "AllocateTrade",
          "edsUri": "C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C2_1499171248430.619502/AllocateTrade_v1_14.eds",
          "majorVersion": 1,
          "minorVersion": 14
       },
        {
          "loadedFromCdd": true,
          "name": "Candidates",
          "edsUri": "C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C3_1499171254573.549221/Candidates_v1_14.eds",
          "majorVersion": 1,
          "minorVersion": 14
        },
        {
          "loadedFromCdd": true,
          "name": "Cargo",
          "edsUri": "C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C0_1499171236109.374753/Cargo_v0_16.eds",
          "majorVersion": 0,
          "minorVersion": 16
        },
        {
          "loadedFromCdd": true,
          "name": "ProcessOrder",
          "edsUri": "C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C1_1499171240858.284274/ProcessOrder_v1_10.eds",
          "majorVersion": 1,
          "minorVersion": 10
        }
      ]
    }
     
    That’s formatted, curl will give unformatted output:
     
    C:\tools\curl-7.48.0-win64-mingw\bin>curl localhost:8850/.../list
    {"decisionServices":[{"loadedFromCdd":true,"name":"AllocateTrade","edsUri":"C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C2_1499171248430.619502/AllocateTrade_v1_14.eds","majorVersion":1,"minorVersion":14},{"loadedFromCdd":true,"name":"Candidates","edsUri":"C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C3_1499171254573.549221/Candidates_v1_14.eds","majorVersion":1,"minorVersion":14},{"loadedFromCdd":true,"name":"Cargo","edsUri":"C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C0_1499171236109.374753/Cargo_v0_16.eds","majorVersion":0,"minorVersion":16},{"loadedFromCdd":true,"name":"ProcessOrder","edsUri":"C:/Progress/CorticonWork_5.7/SER/CcServerSandbox/DoNotDelete/DecisionServices/C1_1499171240858.284274/ProcessOrder_v1_10.eds","majorVersion":1,"minorVersion":10}]}
     
    But there are json utilities available to help you parse it.
     
    Jim
     

    James Arsenault  |  Product Development  |  Progress Software
    email: james.arsenault@progress.com |  direct: 781-280-4934

  • Hi Gertjan,

    We have chosen your "dirty" way.

    I created a powershell script that checks if service is on and if it is not restarts all necessary services replacing the faulty ServerState.xml with the backed-up valid one.

    Thanks.

    Ricardo Jorge

  • Hi Ricardo,

    Good to hear I could help you on this. You're welcome. And yes, keep in mind that this fix is very dirty ;-)

    -Gertjan

  • Sorry for the late reply, but there is a property to disable the validation of EDC enabled decision services at startup

    ###############################################################################################################

    # Determines whether the server will invoke the JPOX metadata validation service at Ruleset deployment

    # time. If this property is set to "true" the service will compare the external JDO document with the

    # database metadata and will prevent deployment if any discrepancies are detected.

    #

    # Default is true

    ###############################################################################################################

    com.corticon.ccserver.validateSchema=true

    If you set this property to false, then the validation step should be skipped at startup.

    This might be a workable solution until a more robust error handling is put in pace.

  • Thanks for this suggestion, Chris.

    As changing the datamodel, without re-importing the metadata and re-deploying *all* decision services after the change, can also cause an undeployment of decision services (even a datatype-change in a database-table can trigger this!), this is an acceptable work-around to avoid this problem.

    Yet good error handling on this is much preferred.

  • Hi Chris,

    First of all thanks for your suggestion, that seems to be a more correct way of doing it.

    As far as I know that configuration you mention is part of CcServer.properties file right?

    Well, we installed the corticon server in a websphere environment and I can't find the file there... is there a way to configure it somewhere else?

    Hi Gertjan,

    The dirty way fails when someone changed rules and din't backed up the ServerState.xml file with the new changes. That way when server restarts we loose lastest rule changes.

    I'll try to use Chris suggestion but if it won't work (i'm not finding where to change property) I'll consider the following workaround:

    1) Iterate ServerState.xml file and for each DecisionService (name and version);

    2) Search DecisionService most recent file (ex. servicev1.0.eds) in the folder "DoNotDelete\DecisionServices"

    3) then I'll update EDS path with the one of most recente file

    4) I'll do the same with database properties.

    I think that way I'll mitigate the problem.

    Ricardo Jorge

  • The properties files can either be modified inside CcConfig.jar file or you can create a brms.properties file and place it in the Server Work directory. Any setting specified in there will over write existing settings.
     
    For more details on setting properties, look at “Configuring Corticon properties and settings” from the Corticon Server:Integration & Deployment Guide
     
    Christopher S. Hogan
    Principal Systems Engineer
     
    Progress
     
    Office:  +1-346-352-0699
    Mobile:  +1-646-243-4282
     
     
    TwitterFacebook LinkedIn Google+
     
  • Hi Chris,

    I have read the section you refered from https://documentation.progress.com/output/Corticon/5.6.1/corticon_integration_deployment.pdf but didn't found anything related to the property "com.corticon.ccserver.validateSchema".

    Even so i followed your instructions: created a brms.properties file and placed it at corticon work directory (C:/IBM/WebSphere/AppServer)

    My brms file only have one uncommented line: "com.corticon.ccserver.validateSchema=false"

    Today on server startup something have failed with oracle listener.
    Corticon Decision Services didn't deploy as cause of listener failure.CcServerDatabaseValidationException

    What did i do wrong?

    Error Log from "CcServer.log"

    2017-08-07 07:50:29.382 INFO  [server.startup : 2] Cc  - Corticon Work: C:/IBM/WebSphere/AppServer

    2017-08-07 07:51:07.029 ERROR  [server.startup : 2] Cc  -  com.corticon.service.ccserver.exception.CcServerDatabaseValidationException: [progress][Oracle JDBC Driver]Error establishing socket to host and port: 0:0:0:0:0:0:0:1:1521. Reason: Connection refused: connect

    2017-08-07 07:51:07.029 ERROR  [server.startup : 2] Cc com.corticon.eclipse.server.core.impl.CcServerInitialization - CcServerInitialization().loadAndDeployCcServerInfoNonCDDs() = Failed to deploy Decision Service. com.corticon.service.ccserver.exception.CcServerDatabaseValidationException: [progress][Oracle JDBC Driver]Error establishing socket to host and port: 0:0:0:0:0:0:0:1:1521. Reason: Connection refused: connect

    at com.corticon.eclipse.server.core.impl.CcServerImpl.configureAndValidateForDecisionsServiceDatabaseAccess(CcServerImpl.java:4095)

    at com.corticon.eclipse.server.core.impl.CcServerImpl.addDecisionService(CcServerImpl.java:3611)

    at com.corticon.eclipse.server.core.impl.CcServerInitialization.loadAndDeployCcServerInfoNonCDDs(CcServerInitialization.java:494)

    at com.corticon.eclipse.server.core.impl.CcServerInitialization.loadAndDeployDecisionServices(CcServerInitialization.java:72)

    at com.corticon.eclipse.server.core.impl.CcServerImpl.initialize(CcServerImpl.java:805)

    at com.corticon.eclipse.server.core.impl.CcServerImpl.<init>(CcServerImpl.java:685)

    at com.corticon.eclipse.server.core.CcServerFactory.getCcServer(CcServerFactory.java:30)

    at com.corticon.servlets.CcServerServletInitializer.<clinit>(CcServerServletInitializer.java:21)

    at java.lang.J9VMInternals.initializeImpl(Native Method)

    at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)

    at java.lang.Class.forNameImpl(Native Method)

    at java.lang.Class.forName(Class.java:213)

    at java.beans.Beans.instantiate(Beans.java:189)

    at java.beans.Beans.instantiate(Beans.java:80)

    at com.ibm.ws.webcontainer.servlet.ServletWrapper$1.run(ServletWrapper.java:1483)

    at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)

    at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadServlet(ServletWrapper.java:1472)

    at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadOnStartupCheck(ServletWrapper.java:1351)

    at com.ibm.ws.webcontainer.webapp.WebApp.doLoadOnStartupActions(WebApp.java:606)

    at com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinally(WebApp.java:576)

    at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:425)

    at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)

    at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:169)

    at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:749)

    at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:634)

    at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:426)

    at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:718)

    at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1175)

    at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1370)

    at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)

    at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:968)

    at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:774)

    at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2182)

    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:445)

    at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)

    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:388)

    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:116)

    at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:994)

    at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)

    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)

     

     

    Thanks for all help,

    Ricardo Jorge

  • Ricardo,
     
    Are you sure you have the correct work dir location? This is the Corticon Server work dir, not the app server work dir.
     
    You can use this SOAP API to retrieve the appropriate setting for the Corticon Server:
     
    Christopher S. Hogan
    Principal Systems Engineer
     
    Progress
     
    Office:  +1-346-352-0699
    Mobile:  +1-646-243-4282
     
     
    TwitterFacebook LinkedIn Google+
     
  • I think it's the correct one.

    I called the webmethod getCcPropertyValue for the property CORTICON_WORK_DIR and it returned:

    "C:/IBM/WebSphere/AppServer"

    Thanks,

    Ricardo Jorge

  • Ricardo,

    It appears that support for this feature was removed in a previous version. I am checking to see if it can be added back in a future release. However, for the time being it looks like you will have to use the previous solution.

    Chris