JsonObject Returned is "application/octet-stream"

Posted by v205 on 02-Jan-2019 22:28

Hello:

Hope you all had relaxing holidays. 

So, I am trying a POST request while using the .Net.HTTP classes inside OpenEdge 11.7.2.

The endpoint in question is a REST API endpoint that expects JSON and returns JSON. The request and response are good using Postman. 

When I use SampleX.p to call the apiPost method inside RestConnector.cls (both programs are attached), instead of getting oResp:ContentType of "application/json", I am getting "application/octet-stream". 

Please note that the host, URI and other related parameters are deconstructed from a DB record inside the connectEndpoint METHOD inside RestConnector.cls. I have checked all of these parameters and they are formed correctly.

Upon more research, I came across this KB entry that talks about the response being truncated:

https://knowledgebase.progress.com/articles/Article/Sometimes-an-HTTPS-response-contains-an-empty-or-incomplete-content-when-using-the-OpenEdge-HTTP-client

After reading this article, I decided to examine the contents of the response and sure enough, I see a partial entry in the response:

"Success (note the ending "" missing)

Where it should be "Success"

The debugging code that shows the response converted into character from the memptr is also at the end of the apiPost method inside RestConnector.cls

Below is the JSON that is POST'ed to this endpoint: 

https://webservices.i-parcel.com/api/SubmitCatalog

{
"key": "keyObscured",
"SKUs": [
{
"SKU": "Product123",
"Length": 1.0,
"Width": 1.0,
"Height": 1.0,
"Weight": 1.0,
"CurrentPrice": 12.35,
"SKN": "p123",
"CountryOfOrigin": "CN",
"ProductName": "TestProduct",
"ProductURL": "http:\/\/exampe.com\/product123"
}
]
}

I am wondering at this point if this is a bug, or is it something I am not using correctly. 

Any help is much appreciated! 

Thanks,[View:/cfs-file/__key/communityserver-discussions-components-files/19/2161.SampleX.p:320:240][View:/cfs-file/__key/communityserver-discussions-components-files/19/7776.RestConnector.cls:320:240]

All Replies

Posted by Peter Judge on 03-Jan-2019 01:22

There's nothing obviously wrong in the attached code.

I'd turn on logging to see if there's anything useful in it. If the HTTP client does return application/octet-stream when you're expecting JSON, then there'll be a message saying why (as bet it can).

Add

 log-manager:logfile-name = 'api.log'.

 log-manager:logging-level = 6.

 log-manager:clear-log().    //optional

and the HTTP client and its components will write their log data into api.log.

Posted by v205 on 04-Jan-2019 20:50

@Peter Thank you for following up. So, I did some more debugging following your advice and I am seeing the following logs (all attached). 

From the "response-data-received" file, it is clear that the response Content-Type is "application/json" as that is what is in the response header. 

Also, the response content, as expected, is just a simple "success" (9 characters).

One problem is that when I parse the response using a OpenEdge.Core.Memptr:GetString(1) method call, it only returns the first 8 characters from "success". This is also evidenced by the following set of lines from the debug logs (attached as api.log):

[19/01/04@15:37:03.498-0500] P-012148 T-010128 1 4GL LogMgrWrtr Error(s) raised: Progress.Json.JsonParserError
[19/01/04@15:37:03.498-0500] P-012148 T-010128 1 4GL LogMgrWrtr Json Parser Error at offset 9: parse error: client cancelled parse via callback return value. (16068) (16068)
[19/01/04@15:37:03.498-0500] P-012148 T-010128 1 4GL LogMgrWrtr [OE.N.HTTP.L.ABLS.ABLSocketLibrary ERROR] EXTRACT ENTITY: Response entity contains raw message body
[19/01/04@15:37:03.499-0500] P-012148 T-010128 1 4GL LogMgrWrtr [OE.N.HTTP.L.ABLS.ABLSocketLibrary ERROR] EXTRACT ENTITY: Response Content-Type changed from application/json to application/octet-stream

It appears to be that the endpoint is returning the "success" response in the expected "application/json" content type, but the ABL library has a problem parsing the response correctly. Could this be a bug? 

I am re-attaching both the calling program "SampleX.p" and the underlying Class I am using for the REST methods - "RestConnector.cls". 

TIA,

h[View:/cfs-file/__key/communityserver-discussions-components-files/19/8306.RestConnector.cls:320:240][View:/cfs-file/__key/communityserver-discussions-components-files/19/6278.SampleX.p:320:240][View:/cfs-file/__key/communityserver-discussions-components-files/19/api.log:320:240][View:/cfs-file/__key/communityserver-discussions-components-files/19/request_2D00_raw.txt:320:240][View:/cfs-file/__key/communityserver-discussions-components-files/19/1680.response_2D00_data_2D00_received.txt:320:240][View:/cfs-file/__key/communityserver-discussions-components-files/19/response_2D00_data_2D00_chunk_2D00_00001.txt:320:240]

Posted by Matt Baker on 04-Jan-2019 21:05

Poking through the response-data-chunk-00001.txt file...

The HTTP response header specifies that the response Content-Type is supposed to be "application/json", but it is returning some string data...then it isn't JSON and the content type is wrong.  The file looks like the quoted word "Success" followed by a whole bunch of null characters.  This certainly doesn't look like JSON data..  Seems like the server is returning incorrectly formatted data.

On top of that the Content-Length header is specifying that the response is supposed to be 9 bytes long.   But the response body is a lot longer than just those 9 characters.  So either the ABL HTTP client is writing more than it should to the debug log, or the server is sending lots of padding that it shouldn't be sending.

Posted by Peter Judge on 07-Jan-2019 14:11

My emailed responses don't seem to have come through (yet?).

The HTTP client is getting a complete response: we know it is complete because the Content-Length header is 9 and there  are 9 characters (bytes) in “Success”.

The JSON parser in OE does not recognise primitives as legal JSON: it only recognises JSON when its an object “{}” or an array “[]”.

The string response _is_ legal JSON but we don’t do the right thing because the JsonEntityWriter (the code that converts the response body into a ‘rea’ object) simply uses the JSON parser. See github.com/.../JsonEntityWriter.cls  for what it does.

You can certainly log a bug to have us handle the other cases. From json.org, that would be any “value” from json.org.

If you want to work around the issue, you can write an EntityWriter that knows how to deal with non-array and non-object values, and have the HTTP client use that instead of the shipped JsonEntityWriter. I can provide more info if needed.

The response-*.txt files have the entire (8k by default) chunk of data received by the socket written into them. You can see from the first few lines in the chunk files that there are 8k bytes read and 607 bytes received. You can see that in the  "chunk dump"

header.

================================

NEW CHUNK (8192)

BYTES READ (607)

================================

The HTTP client does do the right thing, and discards and extraneous nulls.

Posted by v205 on 07-Jan-2019 14:48

@Peter, thanks for the follow up. There is definitely some problem with posts coming through via e-mail but not getting updated in the forum thread here.

I will certainly report this as a bug with Technical Support. In the meantime, please send me more information for the EntityWriter workaround for handling non-object and non-arrays.

Thanks!

h

Posted by v205 on 07-Jan-2019 14:49

Thanks for your input, Matt!

Posted by Peter Judge on 08-Jan-2019 20:46

You can see an example of an extension to the JsonEntityWriter at github.com/.../entity_writer . This shows an example of a wrapper around an existing EntityWriter and also how to register it for use.

Posted by v205 on 11-Jan-2019 14:17

Thanks, Peter. I will try out what you are suggesting and update this thread.

This thread is closed