uLinga for Kafka at EBITUG

Infrasoft, along with our partners comforte, have recently returned from the excellent EBITUG conference in Edinburgh.  It proved to be a great event, with good numbers of both NonStop customers and vendors.  The buzz on the tradeshow floor was pretty much back to pre-pandemic levels, and indeed the last time we were here was in 2019, before everything was put on hold for a few years!

Our focus at the conference was uLinga for Kafka. We’ve had significant interest in that product in Europe, so we were looking forward to presenting and discussing use cases and other potential requirements with our European friends.

Infrasoft’s Andrew Price co-presented with David Lock from comforte, and covered a number of ways in which Kafka can be used – data analytics, alerting, and even as a general “data lake” – once data is available in Kafka, the potential use cases are virtually limitless. The presentation also explained how uLinga for Kafka can stream data to Kafka (“Produce”) and from Kafka (“Consume”) allowing NonStop data and applications to be comprehensively integrated with Kafka.  uLinga for Kafka is unique in the NonStop space, as it provides both Change Data Capture (CDC) support, for all Enscribe file types and SQL/MP tables, but also a number of programmatic interfaces, allowing NonStop applications to explicitly send and receive data to/from Kafka.

For example, uLinga’s IPC interface allows any NonStop application to directly open uLinga and send/receive data with it via $RECEIVE.  We have previously demonstrated how this can be used to stream EMS events to Kafka using the standard NonStop EMSDIST utility.  That same IPC interface allows NonStop applications to consume data directly from Kafka, and have it presented via $RECEIVE (using a WRITEREAD for instance) in a very simple manner.

Probably the most interest in the presentation was around the ability of uLinga to combine these two features, to utilise Kafka as extremely high-performance transaction processing middleware.

Consider the following example diagram, which outlines the steps for a Guardian application or Pathsend client to simply invoke Kafka in this way:

1. Guardian application issues WRITEREAD/Pathway application issues SERVERCLASS_SEND_
2. uLinga for Kafka PRODUCEs data from WRITEREAD/SERVERCLASS_SEND_ to Kafka, including correlation data (passed either in the Key or a custom header field)
3. Linux Consumer logic retrieves data from Kafka, including correlation data, Linux application does some processing of data
4. Linux Producer logic sends response as a Kafka PRODUCE, including original correlation data
5. uLinga for Kafka CONSUMEs response, including correlation data
6. uLinga for Kafka correlates response to original request and completes the WRITEREAD/SERVERCLASS_SEND_

Not only does this approach provide a reliable, high-performance middleware capability (Infrasoft testing has shown throughput rates over 5000 TPS with single digit millisecond latency), but you also have a long-term copy of all your transactional data in Topic 1 and Topic 2 that can be used offline for analytics and other needs.

uLinga for Kafka also supports user exits that can be invoked from anywhere within the internal routing logic.  User exit processes can be managed by uLinga, by Pathway, or by another NonStop-based mechanism.  Processes can also be hosted on another platform, and accessed by TCP/IP or HTTP.  This capability for a wide range of functionality, including data transformation, filtering of data to be forwarded to Kafka, and even Web service invocation for data enrichment before records are streamed to Kafka.

If you’ve been looking into ways to incorporate Kafka into your NonStop environment, chances are uLinga for Kafka can help.  Please contact us at productinfo@infrasoft.com.au for more information.