flowbber.plugins.sinks.data_splitter

Data Splitter Sink

This sink creates files with certain information from the whole data collected. The user needs to specify what data he/she wants to extract, which unique id to use for each file, where the files will be stored and what format does he/she wants for the file.

Important

This class inherits several inclusion and exclusion configuration options for filtering data before using it. See FilterSink Options for more information.

Dependencies:

pip3 install flowbber[data_splitter]

Usage:

[[sinks]]
type = "data_splitter"
id = "..."

    [sinks.config]
    create_parents = true
    output = ".../hardware"
    id_selector = "hardware.suites.pytest.tests.*"
    data_selector = "hardware.suites.pytest.*"
    format = "str"

{
    "sinks": [
        {
            "type": "data_splitter",
            "id": "...",
            "config": {
                "create_parents": true,
                "output": ".../hardware",
                "id_selector": "hardware.suites.pytest.tests.*",
                "data_selector": "hardware.suites.pytest.*"
            }
        }
    ]
}

encoding

Encoding to use to encode the file.

• Default: utf-8

• Optional: True

• Schema:

  {
      'type': 'string',
      'empty': False,
  }
  

• Secret: False

output

Pattern to a path to where collected data files will be stored. The path should use the {{id}} replacement pattern to allow the sink to determine its final path using the IDs found.

For example:

path = "/tmp/myfiles/{{id}}/{{id}}.yaml"

• Default: N/A

• Optional: False

• Schema:

  {
      'type': 'string',
      'empty': False,
  }
  

• Secret: False

create_parents

Create output file parent directories if don’t exist.

• Default: False

• Optional: True

• Schema:

  {
      'type': 'boolean',
  }
  

• Secret: False

id_selector

Selector is a pattern using fnmatch that matches branches in the data tree. The branches found will be used as ID for the filenames.

• Default: N/A

• Optional: False

• Schema:

  {
      'type': 'list',
      'empty': False,
  }
  

• Secret: False

data_selector

Selector is a pattern using fnmatch that matches branches in the data tree. The branches found will be used as the data of the files.

• Default: N/A

• Optional: False

• Schema:

  {
      'type': 'list',
      'empty': False,
  }
  

• Secret: False

format

File format that will be used to store the data.

• Default: json

• Optional: true

• Schema:

  {
      'type': 'string',
      'empty': False,
      'allowed': ['yaml', 'json', 'toml', 'str']
  }
  

• Secret: False

Classes

• DataSplitterSink: Common sink base class that adds several inclusion and exclusion

class flowbber.plugins.sinks.data_splitter.DataSplitterSink(index, type_, id_, optional=False, timeout=None, config=None)

Inheritance