flowbber.components.sink

Module implementating the Sink and FilterSink base classes.

All custom Flowbber sinks must extend from the Sink class.

FilterSink Options

Any Sink that inherits from the FilterSink class will have available the following configuration options:

include

List of patterns of data to include.

Matching is performed using Python’s fnmatch.

• Default: ['*']

• Optional: True

• Schema:

  {
      'type': 'list',
      'schema': {
          'type': 'string',
      },
  }
  

• Secret: False

include_files

List of paths to files containing patterns of data to include.

Matching is performed using Python’s fnmatch.

All unique patterns parsed from these files will be added to the ones defined in the include configuration option.

• Default: []

• Optional: True

• Schema:

  {
      'type': 'list',
      'schema': {
          'type': 'string',
          'empty': False,
      },
  }
  

• Secret: False

exclude

List of patterns of data to exclude.

Matching is performed using Python’s fnmatch.

• Default: []

• Optional: True

• Schema:

  {
      'type': 'list',
      'schema': {
          'type': 'string',
      },
  }
  

• Secret: False

exclude_files

List of paths to files containing patterns of data to exclude.

Matching is performed using Python’s fnmatch.

All unique patterns parsed from these files will be added to the ones defined in the exclude configuration option.

• Default: []

• Optional: True

• Schema:

  {
      'type': 'list',
      'schema': {
          'type': 'string',
          'empty': False,
      },
  }
  

• Secret: False

Classes

• Sink: Main base class to implement a Sink.
• FilterSink: Common sink base class that adds several inclusion and exclusion

class flowbber.components.sink.Sink(index, type_, id_, optional=False, timeout=None, config=None)

Main base class to implement a Sink.

Inheritance

distribute(data)

Distribute the collected data.

All sinks subclasses must implement this abstract method.

Parameters

data (OrderedDict) – The collected data. This dictionary can be modified as required without consequences for the pipeline.

class flowbber.components.sink.FilterSink(index, type_, id_, optional=False, timeout=None, config=None)

Common sink base class that adds several inclusion and exclusion configuration options for filtering data before using it.

See FilterSink Options for more information.

Inheritance