David Beavon (Software) asked a question.

Kafka producers in ABL that don't rely on strings/json

I was wondering if anyone has experience with the new kafka messaging in Progress ABL. I don't see it discussed much in this community.

 

I can see how it might be easy for a developer to send string/json data; but is there anything faster or smaller that could be used as a payload? Has anyone tried using protobuf or avro, for example?

 

I suspect that working with binary formats is a lot more challenging from a programmer standpoint. And it wouldn't necessary increase performance, when producing data from a single AVM session.

 

I started investigating, in order to create an integration with Databricks. Databricks now has a kafka-compliant API that can be used to transmit data to a lakehouse. It is part of their "zerobus-ingest" feature. I'm guessing that performance will max out before we reach 100k rows/minute on a simple table being sent from ABL, especially given the dependency on string/json data. Perhaps we can scale that up by using lots of parallel AVM sessions to produce records simultaneously?


Peter Judge and Mike Fechner like this.
    • David Beavon (Software)

      OK thanks for the update. This is one of the few places where Avro really makes sense (else it is not a format that anyone uses very often - not even data engineers).

       

      Things get complicated whenever a developer is sending a large amount of kafka data to a very distant broker, over a small pipe. In that case it makes no sense to send a massive payload of of text (xml or json or csv). Avro will be a great improvement for these cases.