Azure Data Explorer Kafka Connector – new features with version 1.x

This post has been republished via RSS; it originally appeared at: Azure Data Explorer articles.

Azure Data Explorer has a mature, open source Kafka Connect sink connector that is in use in production by multiple customers. The connector was *gold* certified in September 2020 by Confluent and is now listed in Confluent Connector Hub.  The certification involved stringent testing by Confluent for quality, standards, best practices, feature completeness, delivery guarantees and performance.  To meet the certification requirements, we added a number of new features to our connector, renamed some, and removed support for some with version 1.x.  This article details the same. 

For comprehensive information about our connector, visit our Git repo.

Improved delivery guarantees


Our connector now offers "At least once" delivery guarantees (no data loss).

What's new

 

1. Comprehensive upfront validation

The connector validates for all required and dependent sink properties, access and grants and shuts down upfront with appropriate error messages.

 

2. Configurable behavior on error

A new sink property with possible values of fail (shut down connector task), ignore (ignore and continue processing), and log (log errors and continue processing).  

 

behavior.on.error

 

 

3. *New* configurable retries

A set of new sink properties-

 

errors.retry.max.time.ms errors.retry.backoff.time.ms

 

errors.retry.max.time.ms: Period of time in milliseconds to retry for transient errors

errors.retry.backoff.time.ms: Period of time in milliseconds to backoff before retry for transient errors

 

4. Additional converters supported

We have added support for the following converters-

 

org.apache.kafka.connect.json.JsonConverter io.confluent.connect.avro.AvroConverter io.confluent.connect.json.JsonSchemaConverter

 

JsonSchemaConverter is for Json with schema.  


4. Schema registry support

The connector now supports schema registry with the property-

 

value.converter.schema.registry.url

 

We support schema registry for avro and json, protobuf is on the roadmap.

 

5. Schema with message

The connector now supports messages with schema with the following sink property that accepts a boolean-

 

value.converter.schemas.enable

 


6. Kafka Connect deadletterqueue

The optional sink properties below can be provided for Kafka Connect to redirect messages that failed deserialization-

 

errors.deadletterqueue.bootstrap.servers errors.deadletterqueue.topic.name errors.deadletterqueue.security.protocol errors.deadletterqueue.sasl.mechanism errors.deadletterqueue.sasl.jaas.config errors.deadletterqueue.sasl.kerberos.service.name

 

The last 4 sink properties are required for secure clusters and*kerberos* property is specific to kerberized Kafka clusters.

 

7. Miscellaneous deadletterqueue

The optional sink properties below can be provided for the connector to redirect messages that failed besides deserialization, e.g. transient failures beyond retry period -

 

misc.deadletterqueue.bootstrap.servers misc.deadletterqueue.topic.name misc.deadletterqueue.security.protocol misc.deadletterqueue.sasl.mechanism misc.deadletterqueue.sasl.jaas.config misc.deadletterqueue.sasl.kerberos.service.name

 

The last 4 sink properties are required for secure clusters and*kerberos* property is specific to kerberized Kafka clusters.

 

What's renamed

For the purpose of standardization and consistency, we have renamed some properties-

Previous name New name
kusto.auth.authority aad.auth.authority
kusto.auth.appid aad.auth.appid
kusto.auth.appkey aad.auth.appkey
kusto.tables.topics_mapping kusto.tables.topics.mapping
kusto.sink.tempdir tempdir.path
kusto.sink.flush_interval_ms flush.interval.ms
kusto.sink.flush_size
flush.size.bytes

 

What's removed

The following properties are no longer supported with version 1.x-

Property name Rationale
kusto.auth.username Customers typically only use application IDs/service principals; We therefore removed support for user principals
kusto.auth.password Customers typically only use application IDs/service principals; We therefore removed support for user principals
eventDataCompression We now compress all data shipped to Azure Data Explorer, to improve performance


Connector documentation

Visit out Git repo


Connector download 

Visit out Git repo release page


Confluent Connector Hub

https://www.confluent.io/hub/microsoftcorporation/kafka-sink-azure-kusto


Need support?

Ran into a bug? Need hep getting started?  Have a feature request?  Open an issue.


Want to contribute?

Read our guidelines and get involved.  We welcome community contribution.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.