Chapter 6. Debezium connector for Db2

1

databaseName
schemaName

Identifies the database and the schema that contain the change.

2

ddl

Always null for the Db2 connector. For other connectors, this field contains the DDL responsible for the schema change. This DDL is not available to Db2 connectors.

3

tableChanges

An array of one or more items that contain the schema changes generated by a DDL command.

4

type

Describes the kind of change. The value is one of the following:

5

id

Full identifier of the table that was created, altered, or dropped.

6

table

Represents table metadata after the applied change.

7

primaryKeyColumnNames

List of columns that compose the table’s primary key.

8

columns

Metadata for each column in the changed table.

1

schema

The schema portion of the key specifies a Kafka Connect schema that describes what is in the key’s payload portion.

2

fields

Specifies each field that is expected in the payload, including each field’s name, type, and whether it is required.

3

optional

Indicates whether the event key must contain a value in its payload field. In this example, a value in the key’s payload is required. A value in the key’s payload field is optional when a table does not have a primary key.

4

mydatabase​.MYSCHEMA​.CUSTOMERS​.Key

Name of the schema that defines the structure of the key’s payload. This schema describes the structure of the primary key for the table that was changed. Key schema names have the format connector-name.database-name.table-name.Key. In this example:

5

payload

Contains the key for the row for which this change event was generated. In this example, the key, contains a single ID field whose value is 1004.

1

schema

The value’s schema, which describes the structure of the value’s payload. A change event’s value schema is the same in every change event that the connector generates for a particular table.

2

name

In the schema section, each name field specifies the schema for a field in the value’s payload.

mydatabase.MYSCHEMA.CUSTOMERS.Value is the schema for the payload’s before and after fields. This schema is specific to the customers table. The connector uses this schema for all rows in the MYSCHEMA.CUSTOMERS table.

Names of schemas for before and after fields are of the form logicalName.schemaName.tableName.Value, which ensures that the schema name is unique in the database. This means that when using the Avro converter, the resulting Avro schema for each table in each logical source has its own evolution and history.

3

name

io.debezium.connector.db2.Source is the schema for the payload’s source field. This schema is specific to the Db2 connector. The connector uses it for all events that it generates.

4

name

mydatabase.MYSCHEMA.CUSTOMERS.Envelope is the schema for the overall structure of the payload, where mydatabase is the database, MYSCHEMA is the schema, and CUSTOMERS is the table.

5

payload

The value’s actual data. This is the information that the change event is providing.

It may appear that JSON representations of events are much larger than the rows they describe. This is because a JSON representation must include the schema portion and the payload portion of the message. However, by using the Avro converter, you can significantly decrease the size of the messages that the connector streams to Kafka topics.

6

before

An optional field that specifies the state of the row before the event occurred. When the op field is c for create, as it is in this example, the before field is null since this change event is for new content.

7

after

An optional field that specifies the state of the row after the event occurred. In this example, the after field contains the values of the new row’s ID, FIRST_NAME, LAST_NAME, and EMAIL columns.

8

source

Mandatory field that describes the source metadata for the event. The source structure shows Db2 information about this change, which provides traceability. It also has information you can use to compare to other events in the same topic or in other topics to know whether this event occurred before, after, or as part of the same commit as other events. The source metadata includes:

9

op

Mandatory string that describes the type of operation that caused the connector to generate the event. In this example, c indicates that the operation created a row. Valid values are:

10

ts_ms

Optional field that displays the time at which the connector processed the event. The time is based on the system clock in the JVM running the Kafka Connect task.

In the source object, ts_ms indicates the time that the change was made in the database. By comparing the value for payload.source.ts_ms with the value for payload.ts_ms, you can determine the lag between the source database update and Debezium.

1

before

An optional field that specifies the state of the row before the event occurred. In an update event value, the before field contains a field for each table column and the value that was in that column before the database commit. In this example, note that the EMAIL value is john.doe@example.com.

2

after

An optional field that specifies the state of the row after the event occurred. You can compare the before and after structures to determine what the update to this row was. In the example, the EMAIL value is now noreply@example.com.

3

source

Mandatory field that describes the source metadata for the event. The source field structure contains the same fields as in a create event, but some values are different, for example, the sample update event has different LSNs. You can use this information to compare this event to other events to know whether this event occurred before, after, or as part of the same commit as other events. The source metadata includes:

4

op

Mandatory string that describes the type of operation. In an update event value, the op field value is u, signifying that this row changed because of an update.

5

ts_ms

Optional field that displays the time at which the connector processed the event. The time is based on the system clock in the JVM running the Kafka Connect task.

In the source object, ts_ms indicates the time that the change was made in the database. By comparing the value for payload.source.ts_ms with the value for payload.ts_ms, you can determine the lag between the source database update and Debezium.

1

before

Optional field that specifies the state of the row before the event occurred. In a delete event value, the before field contains the values that were in the row before it was deleted with the database commit.

2

after

Optional field that specifies the state of the row after the event occurred. In a delete event value, the after field is null, signifying that the row no longer exists.

3

source

Mandatory field that describes the source metadata for the event. In a delete event value, the source field structure is the same as for create and update events for the same table. Many source field values are also the same. In a delete event value, the ts_ms and LSN field values, as well as other values, might have changed. But the source field in a delete event value provides the same metadata:

4

op

Mandatory string that describes the type of operation. The op field value is d, signifying that this row was deleted.

5

ts_ms

Optional field that displays the time at which the connector processed the event. The time is based on the system clock in the JVM running the Kafka Connect task.

In the source object, ts_ms indicates the time that the change was made in the database. By comparing the value for payload.source.ts_ms with the value for payload.ts_ms, you can determine the lag between the source database update and Debezium.

1

The name of the connector.

2

Only one task should operate at any one time.

3

The connector’s configuration.

4

The database host, which is the address of the Db2 instance.

5

The logical name of the Db2 instance/cluster, which forms a namespace and is used in the names of the Kafka topics to which the connector writes, the names of Kafka Connect schemas, and the namespaces of the corresponding Avro schema when the Avro Connector is used.

6

Changes in only the public.inventory database are captured.

name

Unique name for the connector. Attempting to register again with the same name will fail. This property is required by all Kafka Connect connectors.

connector.class

The name of the Java class for the connector. Always use a value of io.debezium.connector.db2.Db2Connector for the Db2 connector.

tasks.max

1

The maximum number of tasks that should be created for this connector. The Db2 connector always uses a single task and therefore does not use this value, so the default is always acceptable.

database.hostname

IP address or hostname of the Db2 database server.

database.port

50000

Integer port number of the Db2 database server.

database.user

Name of the Db2 database user for connecting to the Db2 database server.

database.password

Password to use when connecting to the Db2 database server.

database.dbname

The name of the Db2 database from which to stream the changes

database.server​.name

Logical name that identifies and provides a namespace for the particular Db2 database server that hosts the database for which Debezium is capturing changes. Only alphanumeric characters and underscores should be used in the database server logical name. The logical name should be unique across all other connectors, since it is used as a topic name prefix for all Kafka topics that receive records from this connector.

database.history​.kafka.topic

The full name of the Kafka topic where the connector stores the database schema history.

database.history​.kafka.bootstrap​.servers

A list of host/port pairs that the connector uses to establish an initial connection to the Kafka cluster. This connection is used for retrieving database schema history previously stored by the connector, and for writing each DDL statement read from the source database. Each pair should point to the same Kafka cluster used by the Debezium Kafka Connect process.

table.whitelist

An optional, comma-separated list of regular expressions that match fully-qualified table identifiers for tables whose changes you want the connector to capture. Any table not included in the include list does not have its changes captured. Each identifier is of the form schemaName.tableName. By default, the connector captures changes in every non-system table. Do not also set the table.blacklist property.

table.blacklist

An optional, comma-separated list of regular expressions that match fully-qualified table identifiers for tables whose changes you do not want the connector to capture. The connector captures changes in each non-system table that is not included in the exclude list. Each identifier is of the form schemaName.tableName. Do not also set the table.whitelist property.

column.blacklist

empty string

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns to exclude from change event values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. Primary key columns are always included in the event’s key, even if they are excluded from the value.

column.mask​.hash.hashAlgorithm​.with.salt.salt

n/a

An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be pseudonyms in change event values. A pseudonym is a field value that consists of the hashed value obtained by applying the hashAlgorithm algorithm and the salt salt that you specify in the property name.

Based on the hash algorithm applied, referential integrity is kept while data is masked. Supported hash algorithms are described in the {link-java7-standard-names}[MessageDigest section] of the Java Cryptography Architecture Standard Algorithm Name Documentation. The hash value is automatically shortened to the length of the column.

You can specify multiple instances of this property with different algorthims and salts. Fully-qualified names for columns are of the form schemaName.tableName.columnName. For example:

column.mask.hash.SHA-256.with.salt.CzQMA0cB5K = + inventory.orders.customerName, inventory.shipment.customerName

where CzQMA0cB5K is a randomly selected salt.
Depending on the hashAlgorithm used, the salt selected, and the actual data set, the field value may not be completely masked.

time.precision.mode

adaptive

Time, date, and timestamps can be represented with different kinds of precision:

adaptive captures the time and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column’s type.

connect always represents time and timestamp values by using Kafka Connect’s built-in representations for Time, Date, and Timestamp, which uses millisecond precision regardless of the database columns' precision. See temporal values.

tombstones.on​.delete

true

Controls whether a tombstone event should be generated after a delete event.

true - delete operations are represented by a delete event and a subsequent tombstone event.

false - only a delete event is sent.

After a delete operation, emitting a tombstone event enables Kafka to delete all change event records that have the same key as the deleted row.

include.schema​.changes

true

Boolean value that specifies whether the connector should publish changes in the database schema to a Kafka topic with the same name as the database server ID. Each schema change is recorded with a key that contains the database name and a value that is a JSON structure that describes the schema update. This is independent of how the connector internally records database history.

column.truncate.to​.length.chars

n/a

An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Fully-qualified names for columns are of the form schemaName.tableName.columnName. In change event records, values in these columns are truncated if they are longer than the number of characters specified by length in the property name. You can specify multiple properties with different lengths in a single configuration. Length must be a positive integer, for example, column.truncate.to.20.chars.

column.mask​.with.length.chars

n/a

An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Fully-qualified names for columns are of the form schemaName.tableName.columnName. In change event values, the values in the specified table columns are replaced with length number of asterisk (*) characters. You can specify multiple properties with different lengths in a single configuration. Length must be a positive integer or zero. When you specify zero, the connector replaces a value with an empty string.

column.propagate​.source.type

n/a

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns. Fully-qualified names for columns are of the form databaseName.tableName.columnName, or databaseName.schemaName.tableName.columnName.

For each specified column, the connector adds the column’s original type and original length as parameters to the corresponding field schemas in the emitted change records. The following added schema parameters propagate the original type name and also the original length for variable-width types:

__debezium.source.column.type + __debezium.source.column.length + __debezium.source.column.scale

This property is useful for properly sizing corresponding columns in sink databases.

datatype.propagate​.source.type

n/a

An optional, comma-separated list of regular expressions that match the database-specific data type name for some columns. Fully-qualified data type names are of the form databaseName.tableName.typeName, or databaseName.schemaName.tableName.typeName.

For these data types, the connector adds parameters to the corresponding field schemas in emitted change records. The added parameters specify the original type and length of the column:

__debezium.source.column.type + __debezium.source.column.length + __debezium.source.column.scale

These parameters propagate a column’s original type name and length, for variable-width types, respectively. This property is useful for properly sizing corresponding columns in sink databases.

See Db2 data types for the list of Db2-specific data type names.

message.key​.columns

empty string

A semicolon separated list of tables with regular expressions that match table column names. The connector maps values in matching columns to key fields in change event records that it sends to Kafka topics. This is useful when a table does not have a primary key, or when you want to order change event records in a Kafka topic according to a field that is not a primary key.

Separate entries with semicolons. Insert a colon between the fully-qualified table name and its regular expression. The format is:

schema-name.table-name:_regexp_;…​

For example,

schemaA.table_a:regex_1;schemaB.table_b:regex_2;schemaC.table_c:regex_3

If table_a has a an id column, and regex_1 is ^i (matches any column that starts with i), the connector maps the value in table_a's id column to a key field in change events that the connector sends to Kafka.

snapshot.mode

initial

Specifies the criteria for performing a snapshot when the connector starts:

initial - For tables in capture mode, the connector takes a snapshot of the schema for the table and the data in the table. This is useful for populating Kafka topics with a complete representation of the data.

schema_only - For tables in capture mode, the connector takes a snapshot of only the schema for the table. This is useful when only the changes that are happening from now on need to be emitted to Kafka topics. After the snapshot is complete, the connector continues by reading change events from the database’s redo logs.

snapshot.isolation​.mode

repeatable_read

During a snapshot, controls the transaction isolation level and how long the connector locks the tables that are in capture mode. The possible values are:

read_uncommitted - Does not prevent other transactions from updating table rows during an initial snapshot. This mode has no data consistency guarantees; some data might be lost or corrupted.

read_committed - Does not prevent other transactions from updating table rows during an initial snapshot. It is possible for a new record to appear twice: once in the initial snapshot and once in the streaming phase. However, this consistency level is appropriate for data mirroring.

repeatable_read - Prevents other transactions from updating table rows during an initial snapshot. It is possible for a new record to appear twice: once in the initial snapshot and once in the streaming phase. However, this consistency level is appropriate for data mirroring.

exclusive - Uses repeatable read isolation level but takes an exclusive lock for all tables to be read. This mode prevents other transactions from updating table rows during an initial snapshot. Only exclusive mode guarantees full consistency; the initial snapshot and streaming logs constitute a linear history.

event.processing​.failure.handling​.mode

fail

Specifies how the connector handles exceptions during processing of events. The possible values are:

fail - The connector logs the offset of the problematic event and stops processing.

warn - The connector logs the offset of the problematic event and continues processing with the next event.

skip - The connector skips the problematic event and continues processing with the next event.

poll.interval.ms

1000

Positive integer value that specifies the number of milliseconds the connector should wait for new change events to appear before it starts processing a batch of events. Defaults to 1000 milliseconds, or 1 second.

max.queue.size

8192

Positive integer value for the maximum size of the blocking queue. The connector places change events that it reads from the database log into the blocking queue before writing them to Kafka. This queue can provide backpressure for reading change-data tables when, for example, writing records to Kafka is slower than it should be or Kafka is not available. Events that appear in the queue are not included in the offsets that are periodically recorded by the connector. The max.queue.size value should always be larger than the value of the max.batch.size connector configuration property.

max.batch.size

2048

Positive integer value that specifies the maximum size of each batch of events that the connector processes.

heartbeat.interval​.ms

0

Controls how frequently the connector sends heartbeat messages to a Kafka topic. The default behavior is that the connector does not send heartbeat messages.

Heartbeat messages are useful for monitoring whether the connector is receiving change events from the database. Heartbeat messages might help decrease the number of change events that need to be re-sent when a connector restarts. To send heartbeat messages, set this property to a positive integer, which indicates the number of milliseconds between heartbeat messages.

Heartbeat messages are useful when there are many updates in a database that is being tracked but only a tiny number of updates are in tables that are in capture mode. In this situation, the connector reads from the database transaction log as usual but rarely emits change records to Kafka. This means that the connector has few opportunities to send the latest offset to Kafka. Sending heartbeat messages enables the connector to send the latest offset to Kafka.

heartbeat.topics​.prefix

__debezium-heartbeat

Specifies the prefix for the name of the topic to which the connector sends heartbeat messages. The format for this topic name is <heartbeat.topics.prefix>.<server.name>.

snapshot.delay.ms

An interval in milliseconds that the connector should wait before performing a snapshot when the connector starts. If you are starting multiple connectors in a cluster, this property is useful for avoiding snapshot interruptions, which might cause re-balancing of connectors.

snapshot.fetch.size

2000

During a snapshot, the connector reads table content in batches of rows. This property specifies the maximum number of rows in a batch.

snapshot.lock​.timeout.ms

10000

Positive integer value that specifies the maximum amount of time (in milliseconds) to wait to obtain table locks when performing a snapshot. If the connector cannot acquire table locks in this interval, the snapshot fails. How the connector performs snapshots provides details. Other possible settings are:

0 - The connector immediately fails when it cannot obtain a lock.

-1 - The connector waits infinitely.

snapshot.select​.statement​.overrides

Controls which table rows are included in snapshots. This property affects snapshots only. It does not affect events that the connector reads from the log. Specify a comma-separated list of fully-qualified table names in the form schemaName.tableName.

For each table that you specify, also specify another configuration property: snapshot.select.statement.overrides.SCHEMA_NAME.TABLE_NAME. For example: snapshot.select.statement.overrides.customers.orders. Set this property to a SELECT statement that obtains only the rows that you want in the snapshot. When the connector performs a snapshot, it executes this SELECT statement to retrieve data from that table.

A possible use case for setting these properties is large, append-only tables. You can specify a SELECT statement that sets a specific point for where to start a snapshot, or where to resume a snapshot if a previous snapshot was interrupted.

sanitize.field​.names

true if connector configuration sets the key.converter or value.converter property to the Avro converter.

false if not.

Indicates whether field names are sanitized to adhere to Avro naming requirements.

provide.transaction​.metadata

false

Determines whether the connector generates events with transaction boundaries and enriches change event envelopes with transaction metadata. Specify true if you want the connector to do this. See Transaction metadata for details.