Aggregators

Aggregators are plugins that allows to delete, add or modify the data collected by the sources.

The following aggregators plugins are included as part of Flowbber.

Expander

Takes a value on data and move its contents to the root

For example, consider a collected data that looks as following:

OrderedDict([
    ('loaded', OrderedDict([
        ('my_source', {
            'my_value1': 1000,
            'my_value2': 2000,
            'my_value3': 3000,
            'other_value': 'hello'
        }),
    ])),
    ('other_source', {
        'its_value1': 1999,
        'its_value2': 2999,
        'its_value3': 3999,
        'other_value': 'bye'
    }),
])

And you want data inside loaded to be in the top level. If using the configuration:

[[aggregators]]
type = "expander"
id = "expander"

    [aggregators.config]
    key = "loaded"

The data will be filtered to:

OrderedDict([
    ('my_source', {
        'my_value1': 1000,
        'my_value2': 2000,
        'my_value3': 3000,
        'other_value': 'hello'
    }),
    ('other_source', {
        'its_value1': 1999,
        'its_value2': 2999,
        'its_value3': 3999,
        'other_value': 'bye'
    }),
])

Dependencies:

pip3 install flowbber[expander]

Usage:

[[aggregators]]
type = "expander"
id = "..."

    [aggregators.config]
    key = "..."

{
    "aggregators": [
        {
            "type": "expander",
            "id": "...",
            "config": {
                "key": "..."
            }
        }
    ]
}

key

Item to expand to root

• Optional: False

• Schema:

  {
      'type': 'string',
      'empty': False,
  }
  

• Secret: False

Filter

Filter the collected data structure using the provided include and exclude patterns.

For example, consider a collected data that looks as following:

OrderedDict([
    ('my_source', {
        'my_value1': 1000,
        'my_value2': 2000,
        'my_value3': 3000,
        'other_value': 'hello'
    }),
    ('coverage', {
        'files': {
            '__init__.py': {
                'line_rate': 1.0,
                'total_misses': 0,
                'total_statements': 4,
            },
            '__main__.py': {
                'line_rate': 0.5,
                'total_misses': 5,
                'total_statements': 10,
            },
        },
        'total': {
            'line_rate': 0.37275985663082434,
            'total_misses': 350,
            'total_statements': 558,
        },
    }),
])

And you want to remove all the my_value* keys from my_source and all files entries in coverage. If using the configuration:

[[aggregators]]
type = "filter"
id = "filter"

    [aggregators.config]
    include = ["*"]
    exclude = ["my_source.my_value*", "coverage.files"]

The data will be filtered to:

OrderedDict([
    ('my_source', {
        'other_value': 'hello'
    }),
    ('coverage', {
        'total': {
            'line_rate': 0.37275985663082434,
            'total_misses': 350,
            'total_statements': 558,
        },
    }),
])

Dependencies:

pip3 install flowbber[filter]

Usage:

[[aggregators]]
type = "filter"
id = "..."

    [aggregators.config]
    exclude = []
    include = ["*"]

{
    "aggregators": [
        {
            "type": "filter",
            "id": "...",
            "config": {
                "include": ["*"],
                "exclude": []
            }
        }
    ]
}

include

List of patterns of key paths to include.

Matching is performed using Python’s fnmatch.

• Default: ['*']

• Optional: True

• Schema:

  {
      'type': 'list',
      'schema': {
          'type': 'string',
          'empty': False,
      },
  }
  

• Secret: False

exclude

List of patterns of key paths to exclude.

Matching is performed using Python’s fnmatch.

• Default: []

• Optional: True

• Schema:

  {
      'type': 'list',
      'schema': {
          'type': 'string',
          'empty': False,
      },
  }
  

• Secret: False

Lcov merger

Merge two or more coverage tracefiles created with lcov into one new tracefile.

Data produced

{
    "files": {
        "my_source_code.c": {
            "total_statements": 40,
            "total_misses": 20,
            "branch_rate": 0.5,
            "total_hits": 8,
            "line_rate": 0.5
        },
        "another_source.c": {
            "total_statements": 40,
            "total_misses": 40,
            "branch_rate": 0.5,
            "total_hits": 8,
            "line_rate": 0.0
        }
    },
    "total": {
        "total_statements": 80,
        "total_misses": 20,
        "line_rate": 0.75
    },
    "tracefile": "<path-to-tracefile.info>"
}

Dependencies:

pip3 install flowbber[lcov_merger]

Usage:

[[aggregators]]
type = "lcov_merger"
id = "..."

    [aggregators.config]
    keys = ["lcov_source_1","lcov_source_2", "..."]
    allow_missing_keys = true
    rc_overrides = ["lcov_branch_coverage=1"]
    remove = ["*hello2*"]

{
    "aggregators": [
        {
            "type": "lcov_merger",
            "id": "...",
            "config": {
                "keys": ["lcov_source_1","lcov_source_2", "..."],
                "allow_missing_keys": true,
                "rc_overrides": ["lcov_branch_coverage=1"],
                "remove": ["*hello2*"]
            }
        }
    ]
}

keys

List of lcov sources.

• Default: N/A

• Optional: False

• Schema:

  schema={
      'type': 'list',
      'empty': False,
      'schema': {
          'type': 'string',
          'empty': False,
      },
  },
  

• Secret: False

allow_missing_keys

Ignore missing lcov source keys.

If True then any missing key, for example of an optional lcov source, will not trigger an aggregator failure and the aggregator will proceed with the remaining keys unaffected, unless no valid key remains (in which case you should mark the aggregator as optional). If False then any missing key will trigger an aggregator failure.

• Default: True

• Optional: True

• Schema:

  schema={
      'type': 'boolean',
  },
  

• Secret: False

rc_overrides

Override lcov configuration file settings.

Elements should have the form SETTING=VALUE.

• Default: []

• Optional: False

• Schema:

  {
      'type': 'list',
      'schema': {
          'type': 'string',
          'empty': False
      },
  }
  

• Secret: False

remove

List of patterns of files to remove from coverage computation.

Patterns will be interpreted as shell wild‐card patterns.

• Default: []

• Optional: True

• Schema:

  {
      'type': 'list',
      'schema': {
          'type': 'string',
          'empty': False,
      },
  }
  

• Secret: False