Commit f55dded7 authored by Shikhar Bhushan's avatar Shikhar Bhushan
Browse files

Doc update following #30

parent aa9fe36d
Loading
Loading
Loading
Loading
+5 −5
Original line number Diff line number Diff line
@@ -30,7 +30,7 @@ Configuration Options
  * Importance: medium

``max.in.flight.requests``
  The maximum number of incomplete batches each task will send before blocking. Note that if this is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries
  The maximum number of incomplete batches each task will send before blocking.

  * Type: int
  * Default: 5
@@ -51,14 +51,14 @@ Configuration Options
  * Importance: low

``max.buffered.records``
  Approximately the max number of records each task will buffer. This config controls the memory usage for each task. When the number of buffered records is larger than this value, the partitions assigned to this task will be paused.
  Approximately the max number of records each task will buffer. This config controls the memory usage for each task.

  * Type: long
  * Type: int
  * Default: 20000
  * Importance: low

``max.retries``
  The max allowed number of retries. Allowing retries will potentially change the ordering of records.
  The max allowed number of retries.

  * Type: int
  * Default: 5
+5 −6
Original line number Diff line number Diff line
@@ -111,12 +111,11 @@ The Elasticsearch connector offers a bunch of features:
Delivery Semantics
------------------
The connector supports batching and pipelined writes to Elasticsearch to boost throughput. It
accumulates messages in batches and allows concurrent processing of multiple batches. However,
when piplining is enabled, the ordering of batches written to Elasticsearch is not guaranteed.
A transient failure can cause a batch to be retried and because of this, a batch sent later may
be written to Elasticsearch first. Pipeplining should be turned off when order of batches is
important (set ``max.in.flight.requests=1``). Moreover, batching should be turned off when message
order is important (set ``batch.size=1``).
accumulates messages in batches and allows concurrent processing of multiple batches.

Document-level update ordering is ensured by using the partition-level Kafka offset as the
`document version <https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning>`_,
and using ``version_mode=external``.

Mapping Management
------------------
+3 −4
Original line number Diff line number Diff line
@@ -86,8 +86,7 @@ public class ElasticsearchSinkConnectorConfig extends AbstractConfig {

  public static final String MAX_IN_FLIGHT_REQUESTS_CONFIG = "max.in.flight.requests";
  private static final String MAX_IN_FLIGHT_REQUESTS_DOC =
      "The maximum number of incomplete batches each task will send before blocking. Note that if this is set to be greater "
      + "than 1 and there are failed sends, there is a risk of message re-ordering due to retries";
      "The maximum number of incomplete batches each task will send before blocking.";
  private static final int MAX_IN_FLIGHT_REQUESTS_DEFAULT = 5;
  private static final String MAX_IN_FLIGHT_REQUESTS_DISPLAY = "Max in Flight Requests";

@@ -99,7 +98,7 @@ public class ElasticsearchSinkConnectorConfig extends AbstractConfig {
  private static final String RETRY_BACKOFF_MS_DISPLAY = "Retry Backoff (ms)";

  public static final String MAX_RETRIES_CONFIG = "max.retries";
  private static final String MAX_RETRIES_DOC = "The max allowed number of retries. Allowing retries will potentially change the ordering of records.";
  private static final String MAX_RETRIES_DOC = "The max allowed number of retries.";
  private static final int MAX_RETRIES_DEFAULT = 5;
  private static final String MAX_RETRIES_DISPLAY = "Max Retries";