Compression for large (memptr) messages sent from an ABL client to a PASOE Agent - Web Transport versus APSV and -mc startup parameter - Forum - OpenEdge General - Progress Community

Compression for large (memptr) messages sent from an ABL client to a PASOE Agent - Web Transport versus APSV and -mc startup parameter

 Forum

Compression for large (memptr) messages sent from an ABL client to a PASOE Agent - Web Transport versus APSV and -mc startup parameter

This question is answered

Dear forum,        doing with 11.7.5

In a recent thread about sending a large binary file to an AppServer, I was advised to adopt the Web transport with WebHandlers.  At a point I thought I might first compress my file before sending it.  Now that I am about doing it, I recall http has native compression capabilities...

This KBase says how to enable compression for the REST Transport :

https://knowledgebase.progress.com/articles/Article/How-to-enable-HTTP-Compression-for-an-OpenEdge-REST-web-application

Q0 : Having played with different levels of compression with 7zip, I found out the default compression rate was a good compromise between the time it takes to make it and the obtained compression.  Based on that, would you rather compress your file locally before sending a zipped version of it, or would just rely on the native compression capabilities of http to do it on the fly, hoping it would negotiate the rate adaptively?

Q1 : I suppose the KBase works for the WEB transport as well as ofr REST, correct?

Q2 : How about the APSV transport?  Does Tomcat enable compression for it as well ?

Q3 : Does the -mc startup parameter work for the PASOE, or does it apply only for the old classical AppServer?

Q4: Aside, with the APSV transport and the Classic AppServer, I do not recall if we can send a MEMPTR to an AppServer session (it may make no sense...) and cannot find it in the Doc.   If it is possible, would it better to send a large binary message with APSV transport and a MEMPTR param with -mc, or with the WEB transport with the enable compression mechanism described in the KBase?

Verified Answer
  • Hi Sébastien

    I'd suggest to use zlib (fast, free software) to compress/decompress your payloads.

    The dll exists in both 32 and 64 bits, Windows and Linux (and other unices too) and can be easily used from ABL.

    Regards

    JC

  • Q6.  -mc should  work with PASOE. Please log a bug if it doesn't.

    Q7.  The HTTP client doesn't do anything special for Content-Encoding (either for requests or responses).

    Untested approach below but ...

    The OpenEdge.Net.HTTP.Filter.Payload.DefaultRequestFilter class is where you would add support for Content-Encoding for requests and for responses, in the OpenEdge.Net.HTTP.Lib.ABLSockets.ABLSocketLibrary class. In both cases it'd be in/after the ExtractEntity method. You could write a class that extends the above, and register them as the appropriate handlers.

    For requests

    OpenEdge.Net.HTTP.Filter.Writer.RequestWriterBuilder:Registry:Put('HTTP/1.1':u,         get-class(Your.Class.That.Extends.DefaultRequestFilter)).

    For responses

    OpenEdge.Net.HTTP.Lib.ClientLibraryBuilder:Registry:Put(get-class(IHttpClientLibrary):TypeName, get-class(Your.Class.That.Extends.ABLSocketLibrary)).

    If you'd like to log a bug for this too, please do. It might end up being an enhancement but I'd think that it's worth logging. As I mentioned earlier, the PASOE server does the Content-Encoding in the Tomcat layer.

  • > On Oct 14, 2019, at 4:44 AM, Jean-Christophe Cardot wrote:

    >

    > I'd suggest to use zlib (fast, free software) to compress/decompress your payloads.

    >

    >

    the compression code used by the app server is zlib. the zlib shared library is included with openedge.

    in my own test harness, i found that the zlib compression makes little difference on fast networks but was useful in slow networks.

All Replies
  • Four properties control compression on PASOE, and they are transport independent. Below are the defaults.
     
    -Dpsc.as.http.compress=on
    -Dpsc.as.https.compress=on
    -Dpsc.as.compress.min=2048
    -Dpsc.as.compress.types=text/html,text/xml,text/javascript,text/css,application/json
     
    So, compression is enabled for payloads > 2048 and for the mime types specified. To enable compression in APSV, you need to add application/octet-stream (please double check this first) to the compress types. There is an addition setting that isn’t exposed as a property, but is what Tomcat provides as a default.
     
    noCompressionUserAgents="gozilla,traviata"
     
    Dave
     
  • In PASOE , there are properties in conf/catalina.properties to control this. The values below are defaults in 11.7.something (but don't ask which version I used to create the instance).
    # HTTP & HTTPS  Compression
    psc.as.http.compress=on
    psc.as.https.compress=on
    psc.as.compress.min=2048
    psc.as.compress.types=text/html,text/xml,text/javascript,text/css,application/json
     
    Since the config is for the HTTP and HTTPS connectors, and since all client traffic flows through one of these schemes/connectors, I'd expect that all the transports were covered.
     
    That said, I don't know how you'd get an ABL client (ie APSV transport) to compress the HTTP request.  The answer is probably your musings around Q4 . I don't know where the "-mc" compression is de-compressed, but if it's done in the MS-Agent then you may get a benefit from it since the data would be passed from the Tomcat process to the MS-Agent.
     
     
  • Just to clarify, the KBase you reference is for the Classic REST Adapter on Tomcat 5.5.
     
  • Forgot about the client. Yes, for APSV and an ABL Client, you should use -mc. The OpenClients can probably support HTTP Compression but it isn’t anything we investigated.
     
    Dave
     
  • Thank you David and Peter.  I was using the httpClient (OpenEdge.Net.*) to use the WEB transport from an ABL Client.  

    Q5 Can it support compression enablement?

    To what I understand, I'd better simply use the APSV transport with -mc and you seem to answer Q4 saying we can use a MEMPTR input param to send something.  Correct?

  • No. Not natively - you have to compress (and de-compress) the body yourself with the HTTP client. I thought there was a bug logged for this but apparently not. So feel free to log one.
     
     
     
  • Thank you Peter, I will log one but could you please tell me a out Q4?  Can a memptr be sent via APSV transport, and can compression. Occur?

  • to remove any confusion : I am interested in the new PASOE appserver, not the classic one to be deprecated. I'm interested in sending large stuff from an ABL client, with either WEB or APSV transport.

  • (post removed because duplicated for obscur reasons)

  • Just did a few tests : Yes we can simply pass a MEMPTR to an AppServer, with the APSV Transport.  But -mc seems to have no effect.  

    I can test it by sending a 10MBytes file filled with the character 'A'  (so 10 millions 'A' s) that can be highly compressed (down to 12kBytes with 7zip)

    With Wireshark, I can see a the long message sent to AppServer is cut in slices of 8402 bytes, with a content full of 'A', the same way with or without -mc.

    I've tried that with both MEMPTR and LONGCHAR parameter types with same result.

    I am surprised -mc has no effect.  According to this KBase, I have enabled it the right way on the client side (I even message SESSION:STARTUP-PARAMETERS to make sure the launcher inserts -mc) :

    knowledgebase.progress.com/.../P69169.  

    Q6 : Did I miss something?  Would -mc work only for temp-table parameters?

    Q7 : Perhaps a question for Peter, so I could keep using the WEB Transport and the native decompression capabilities mentioned above : could you give me a hint to compress my http bodies myself on the ABL side as suggested ?  (Any native .Net System method?).

  • Hi Sébastien

    I'd suggest to use zlib (fast, free software) to compress/decompress your payloads.

    The dll exists in both 32 and 64 bits, Windows and Linux (and other unices too) and can be easily used from ABL.

    Regards

    JC

  • Q6.  -mc should  work with PASOE. Please log a bug if it doesn't.

    Q7.  The HTTP client doesn't do anything special for Content-Encoding (either for requests or responses).

    Untested approach below but ...

    The OpenEdge.Net.HTTP.Filter.Payload.DefaultRequestFilter class is where you would add support for Content-Encoding for requests and for responses, in the OpenEdge.Net.HTTP.Lib.ABLSockets.ABLSocketLibrary class. In both cases it'd be in/after the ExtractEntity method. You could write a class that extends the above, and register them as the appropriate handlers.

    For requests

    OpenEdge.Net.HTTP.Filter.Writer.RequestWriterBuilder:Registry:Put('HTTP/1.1':u,         get-class(Your.Class.That.Extends.DefaultRequestFilter)).

    For responses

    OpenEdge.Net.HTTP.Lib.ClientLibraryBuilder:Registry:Put(get-class(IHttpClientLibrary):TypeName, get-class(Your.Class.That.Extends.ABLSocketLibrary)).

    If you'd like to log a bug for this too, please do. It might end up being an enhancement but I'd think that it's worth logging. As I mentioned earlier, the PASOE server does the Content-Encoding in the Tomcat layer.

  • > On Oct 14, 2019, at 4:44 AM, Jean-Christophe Cardot wrote:

    >

    > I'd suggest to use zlib (fast, free software) to compress/decompress your payloads.

    >

    >

    the compression code used by the app server is zlib. the zlib shared library is included with openedge.

    in my own test harness, i found that the zlib compression makes little difference on fast networks but was useful in slow networks.

  • Thank you Jean-Christophe, Peter and Gus.

    Peter, I'll try to soon report a bug for -mc.  Regarding Q7, I will have to do it quick.  Compression is important for my case because my calls go over WAN.  The messages are so large that I decided to send them by Chunks so it offers monitoring and stop + continue possiblities.  I was interested in native compression of my chunks at the transfer level if it was available, but since it is not and I am getting short in time, I am wondering if I'd better produce a zip file of what I want to send, then send that zip by chunks rather than taking a risk at implementing the compression of payload on the fly for now.

    Now, I agree this is an important feature.

  • We send/receive large files in chunks from/to our Java (passed to server via MEMPTR) and have been happy with the results. It's been a while, but we tried various sizes and landed on 10MB chunks being the optimal size.

    Jeff Ledbetter
    Product Architect | Roundtable Software