Sources

Sources are plugins that allow to collect data from different origins.

The following sources plugins are included as part of Flowbber.

Cobertura

This source parses standard Cobertura XML files.

A Cobertura file describes the coverage of source code executed by tests. This files can be generated for several programming languages, including:

  • Python, by using pytest-cov or coverage.py.
  • Java, by using the original Cobertura tool.
  • C, by using GNU gcov to generate .gcno and .gcda coverage files, then processing the source tree with lcov to generate a coverage.info which is finally converted to Cobertura using lcov_cobertura.
  • Go, by using gocov to convert a -coverprofile file and then processing the resulting coverage.json with gocov-xml.

Data collected:

{
    "files": {
        "my_source_code.c": {
            "total_statements": 40,
            "total_misses": 20,
            "line_rate": 0.5
        },
        "another_source.c": {
            "total_statements": 40,
            "total_misses": 40,
            "line_rate": 0.0
        }
    },
    "total": {
        "total_statements": 80,
        "total_misses": 20,
        "line_rate": 0.75
    }
}

Dependencies:

pip3 install flowbber[cobertura]

Usage:

[[sources]]
type = "cobertura"
id = "..."

    [sources.config]
    xmlpath = "coverage.xml"
    include = ["*"]
    exclude = []
{
    "sources": [
        {
            "type": "cobertura",
            "id": "...",
            "config": {
                "xmlpath": "coverage.xml",
                "include": ["*"],
                "exclude": []
            }
        }
    ]
}

xmlpath

Path to the Cobertura coverage.xml file to be parsed.

  • Default: N/A

  • Optional: False

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

include

List of patterns of files to include.

Matching is performed using Python’s fnmatch.

  • Default: ['*']

  • Optional: True

  • Schema:

    {
        'type': 'list',
        'schema': {
            'type': 'string',
        },
    }
    
  • Secret: False

exclude

List of patterns of files to exclude.

Matching is performed using Python’s fnmatch.

  • Default: []

  • Optional: True

  • Schema:

    {
        'type': 'list',
        'schema': {
            'type': 'string',
        },
    }
    
  • Secret: False

CPU

This source collects information about the system’s CPUs load.

Data collected:

{
    "num_cpus": 2,
    "system_load": 25.0,
    "per_cpu": [40.0, 10.0]
}

Dependencies:

pip3 install flowbber[cpu]

Usage:

[[sources]]
type = "cpu"
id = "..."
{
    "sources": [
        {
            "type": "cpu",
            "id": "...",
            "config": {}
        }
    ]
}

Env

This source collects environment variables.

Danger

This source can leak sensitive data. Please adjust include and exclude patterns accordingly.

By default include and exclude patterns are empty, so no data will be collected. See Usage below for examples.

Data collected:

{
    "pythonhashseed": "720667772"
}

Dependencies:

pip3 install flowbber[env]

Usage:

Variables will be collected if the name of the variable match any item in the include list but DOES NOT match any item in the exclude list.

Variables names will be stored in lowercase if the lowercase option is set to true (the default).

[[sources]]
type = "env"
id = "..."

    [sources.config]
    include = ["PYTHONHASHSEED"]
    exclude = []
    lowercase = true
{
    "sources": [
        {
            "type": "env",
            "id": "...",
            "config": {
                "include": [
                    "PYTHONHASHSEED"
                ],
                "exclude": [],
                "lowercase": true
            }
        }
    ]
}

Configuration examples:

Collect all environment variables

{
    "include": ["*"],
    "exclude": []
}

Collect all except a few

{
    "include": ["*"],
    "exclude": ["*KEY*", "*SECRET*"]
}

Collect only the ones specified

{
    "include": ["PYTHONHASHSEED"],
    "exclude": []
}

Using with Jenkins CI

This source is very helpful to collect information from Jenkins CI:

[[sources]]
type = "env"
id = "jenkins"

    [sources.config]
    include = [
        "BUILD_NUMBER",
        "JOB_NAME",
        "GIT_COMMIT",
        "GIT_URL",
        "GIT_BRANCH",
        "BUILD_TIMESTAMP"
    ]
    lowercase = false
{
    "sources": [
        {
            "type": "env",
            "id": "jenkins",
            "config": {
                "include": [
                    "BUILD_NUMBER",
                    "JOB_NAME",
                    "GIT_COMMIT",
                    "GIT_URL",
                    "GIT_BRANCH",
                    "BUILD_TIMESTAMP"
                ],
                "lowercase": false
            }
        }
    ]
}

Note

In the above example the BUILD_TIMESTAMP variable is provided by the Build Timestamp Plugin.

include

List of patterns of environment variables to include.

Matching is performed using Python’s fnmatch.

  • Default: []

  • Optional: True

  • Schema:

    {
        'type': 'list',
        'schema': {
            'type': 'string',
            'empty': False,
        },
    }
    
  • Secret: False

exclude

List of patterns of environment variables to exclude.

Matching is performed using Python’s fnmatch.

  • Default: []

  • Optional: True

  • Schema:

    {
        'type': 'list',
        'schema': {
            'type': 'string',
            'empty': False,
        },
    }
    
  • Secret: False

lowercase

Store variables names in lowercase.

  • Default: True

  • Optional: True

  • Schema:

    {
        'type': 'boolean',
    }
    
  • Secret: False

Git

This source collects information from a (this) local git repository:

  • The repository root path.
  • The current branch.
  • The current revision hash.
  • The (a) tag pointing to the current revision.
  • Name of the author of the current revision.
  • Email of the author of the current revision.
  • Commit message subject of current revision.
  • Commit message body of current revision.
  • Commit date in strict ISO 8601 format.

The directory option can be used to specify a path to a local git repository to get information from. In particular, if your CI system created an out-of-repository build directory you can always get the context of the repository the pipeline definition file is commited to using:

[[sources]]
type = "git"
id = "..."

    [sources.config]
    directory = "{git.root}"
{
    "sources": [
        {
            "type": "git",
            "id": "...",
            "config": {
                "directory": "{git.root}"
            }
        }
    ]
}

Data collected:

{
    "root": "/home/kuralabs/flowbber",
    "branch": "master",
    "rev": "9742e30",
    "tag": "2.0.0",
    "name": "The Author",
    "email": "author@domain.com",
    "subject": "Added a new Git source.",
    "body": "",
    "date": "2017-09-13T01:27:55-06:00"
}

Dependencies:

pip3 install flowbber[git]

Usage:

[[sources]]
type = "git"
id = "..."

    [sources.config]
    directory = "."
{
    "sources": [
        {
            "type": "git",
            "id": "...",
            "config": {
                "directory": "."
            }
        }
    ]
}

directory

Directory to run git from. This can be any subdirectory inside a git repository, not only the root.

  • Default: '.'

  • Optional: True

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

GitHub

This source collects pull requests and issues statistics from a GitHub repository using the GitHub API v3. This source requires a personal access token, you can create one in your settings. No particular scope is required.

Data collected:

{
    "issue": {
        "closed": 1,
        "open": 2
    },
    "pr": {
        "closed": 3,
        "open": 4
    }
}

Dependencies:

pip3 install flowbber[github]

Usage:

[[sources]]
type = "github"
id = "..."

    [sources.config]
    token = "abcdefabcdefabcdefabcdef"
    repository = "organization/repository"
    base_url = "https://api.github.com"
{
    "sources": [
        {
            "type": "github",
            "id": "...",
            "config": {
                "token": "abcdefabcdefabcdefabcdef",
                "repository": "organization/repository",
                "base_url": "https://api.github.com"
            }
        }
    ]
}

token

Personal access token to use to connect to the GitHub API. Keep this value safe and secret.

  • Default: N/A

  • Optional: False

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: True

repository

Name of the repository to query for in the form organization/repository.

  • Default: N/A

  • Optional: False

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

base_url

Base URL to connect to the GitHub v3 API.

For public GitHub, the default https://api.github.com should be used. For private GitHub Enterprise instances use https://github.yourdomain.com/api/v3 or similar.

  • Default: https://api.github.com

  • Optional: True

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

Google Test

This source parses the JUnit like results XML file generated by Google Test.

Data collected:

{
    "failures": 1,
    "disabled": 1,
    "errors": 1,
    "tests": 1,
    "time": 10.555,
    "timestamp": "2017-09-13T00:51:51",
    "properties": {
        "<propname1>": "<propvalue1>"
    },
    "suites": {
        "<suitename1>": {
            "cases": {
                "<casename1>": {
                    "status": "<PASS|FAIL|SKIP>",
                    "time": 0.05,
                    "properties": {
                        "<propname1>": "<propvalue1>"
                    }
                },
                "<casename2>": {
                    "status": "<PASS|FAIL|SKIP>",
                    "time": 0.05,
                    "properties": {
                        "<propname1>": "<propvalue1>"
                    }
                }
            },
            "properties": {
                "<propname1>": "<propvalue1>"
            },
            "failures": 1,
            "passed": 1,
            "disabled": 1,
            "errors": 1,
            "tests": 1,
            "time": 0.456
        }
    }
}

In addition to the previous data structure, if status is FAIL an additional key failures will be available with a list of failures found:

{
    # ...
    'failures': [
        '/home/kuralabs/googletest-example/tests/test2.cpp:12\n'
        'Expected: 0\n'
        'To be equal to: 1',
    ]
}

Dependencies:

pip3 install flowbber[gtest]

Usage:

[[sources]]
type = "gtest"
id = "..."

    [sources.config]
    xmlpath = "tests.xml"
{
    "sources": [
        {
            "type": "gtest",
            "id": "...",
            "config": {
                "xmlpath": "tests.xml"
            }
        }
    ]
}

xmlpath

Path to the JUnit like XML results tests.xml file to be parsed.

  • Default: N/A

  • Optional: False

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

JSON

This source fetch and parses a local or remote (http, https) json file.

Data collected:

Same as the source file

Dependencies:

pip3 install flowbber[json]

Usage:

[[sources]]
type = "json"
id = "..."

    [sources.config]
    file_uri = "file://{pipeline.dir}/file.json"
    encoding = "utf-8"
    ordered = true
    verify_ssl = true
{
    "sources": [
        {
            "type": "json",
            "id": "...",
            "config": {
                "file_uri": "file://{pipeline.dir}/file.json",
                "encoding": "utf-8",
                "ordered": true,
                "verify_ssl": true
            }
        }
    ]
}

file_uri

URI to the JSON file. If no schema is specified, file:// will be used.

Supported schemas:

  • file:// (the default): File system path to the JSON file.

    file://path/to/file.json
    

    Or if using Substitutions:

    file://{pipeline.dir}/file.json
    
  • http[s]://: URL to download the JSON file.

    https://mydomain.com/archive/file.json
    
  • Default: N/A

  • Optional: False

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

encoding

Encoding to use to decode the file.

  • Default: utf-8

  • Optional: True

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

ordered

Parses the file in order and returns it as a py:class:collections.OrderedDict instead of an unordered dictionary.

  • Default: False

  • Optional: True

  • Schema:

    {
        'type': 'boolean'
    }
    
  • Secret: False

verify_ssl

Enable or disable SSL verification. This option only applies if the https schema is used.

  • Default: True

  • Optional: True

  • Schema:

    {
        'type': 'boolean'
    }
    
  • Secret: False

Lcov

This source calls lcov on a specified directory and process its result file with lcov_cobertura to create a standard Cobertura XML file, which in turn is then parsed using the flowbber Cobertura source.

Note

This source requires the lcov executable to be available in your system to run.

Data collected:

{
    "files": {
        "my_source_code.c": {
            "total_statements": 40,
            "total_misses": 20,
            "branch_rate": 0.5,
            "total_hits": 8,
            "line_rate": 0.5
        },
        "another_source.c": {
            "total_statements": 40,
            "total_misses": 40,
            "branch_rate": 0.5,
            "total_hits": 8,
            "line_rate": 0.0
        }
    },
    "total": {
        "total_statements": 80,
        "total_misses": 20,
        "line_rate": 0.75
    },
    "tracefile": "<path-to-tracefile.info>"
}

Dependencies:

pip3 install flowbber[lcov]

Usage:

[[sources]]
type = "lcov"
id = "..."

    [sources.config]
    directory = "{pipeline.dir}"
    rc_overrides = ["lcov_branch_coverage=1"]
    remove = ["*hello2*"]
{
    "sources": [
        {
            "type": "lcov",
            "id": "...",
            "config": {
                "directory": "{pipeline.dir}",
                "rc_overrides": ["lcov_branch_coverage=1"],
                "remove": ["*hello2*"]
            }
        }
    ]
}

directory

Path to the directory containing gcov’s .gcda files .

  • Default: N/A

  • Optional: False

  • Schema:

    {
        'type': 'string',
        'empty': False
    }
    
  • Secret: False

rc_overrides

Override lcov configuration file settings.

Elements should have the form SETTING=VALUE.

  • Default: []

  • Optional: False

  • Schema:

    {
        'type': 'list',
        'schema': {
            'type': 'string',
            'empty': False
        },
    }
    
  • Secret: False

remove

List of patterns of files to remove from coverage computation.

Patterns will be interpreted as shell wild‐card patterns.

  • Default: []

  • Optional: True

  • Schema:

    {
        'type': 'list',
        'schema': {
            'type': 'string',
            'empty': False,
        },
    }
    
  • Secret: False

Pytest

This source parses the JUnit like results XML file generated by pytest.

Data collected:

{
    "errors": 0,
    "failures": 2,
    "skips": 0,
    "tests": 4,
    "passed": 6,
    "time": 0.047,
    "suites": {
        "<suite_name>": {
            "errors": 0,
            "failures": 2,
            "skips": 0,
            "tests": 4,
            "time": 0.047,
            "cases": {
                "<classname>.<name>": {
                    "status": "<PASS|FAIL|ERROR|SKIP>",
                    "file": "test/test_file.py",
                    "line": 19,
                    "classname": "test.test_file",
                    "name": "test_function",
                    "time": 0.0012459754943847656,
                    "properties": [
                        {"<propname1>": "<propvalue1>"},
                        {"<propname2>": "<propvalue2>"}
                    ]
                }
            }
        }
    }
}

Warning

Most of the time the <suite_name> will be set to pytest, but do not hardwire your code to that value. It can be changed by changing your pytest.ini file:

In addition to the previous data structure, if status is FAIL, ERROR or SKIP an additional key failure, error or skipped will be available with information of the problem:

{
    # ...
    'error': {
        'code': (
            '@pytest.fixture\n'
            '    def dofail():\n'
            '>       raise RuntimeError()\n'
            'E       RuntimeError\n\nconftest.py:14: RuntimeError'
        ),
        'message': 'test setup failure',
    }
}

The information object always have a message and a code key describing what the issue is and where it happened.

Dependencies:

pip3 install flowbber[pytest]

Usage:

[[sources]]
type = "pytest"
id = "..."

    [sources.config]
    xmlpath = "tests.xml"
{
    "sources": [
        {
            "type": "pytest",
            "id": "...",
            "config": {
                "xmlpath": "tests.xml"
            }
        }
    ]
}

xmlpath

Path to the JUnit like XML results tests.xml file to be parsed.

  • Default: N/A

  • Optional: False

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

SLOC

This source count source lines of code. It scans a directory for source code files, identify their language and count the number of code, comments and empty lines. It is highly tunable using include and exclude patterns as explained below. It is implemented on top of pygount, which is very accurate, but not the fastest. Benchmark on HUGE code bases.

Data collected:

{
    "sloc": {
        "html": {
            "string": 0,
            "empty": 59,
            "code": 129,
            "documentation": 7
        },
        "python": {
            "string": 362,
            "empty": 1532,
            "code": 2262,
            "documentation": 4291
        },
        "restructuredtext": {
            "string": 73,
            "empty": 619,
            "code": 1133,
            "documentation": 19
        },
    },
    "files": {
        "setup.py": "python",
        "lib/flowbber/main.py": "python",
        "... more collected files ...": "<language detected>"
    }
}

Dependencies:

pip3 install flowbber[sloc]

Usage:

[[sources]]
type = "sloc"
id = "..."

    [sources.config]
    directory = "{git.root}"
    include = ["*"]
    exclude = []
{
    "sources": [
        {
            "type": "sloc",
            "id": "...",
            "config": {
                "directory": "{git.root}",
                "include": ["*"],
                "exclude": []
            }
        }
    ]
}

directory

Root directory to search for files.

  • Default: '.'

  • Optional: True

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

include

List of patterns of files to include.

Matching is performed using Python’s fnmatch.

  • Default: ['*']

  • Optional: True

  • Schema:

    {
        'type': 'list',
        'schema': {
            'type': 'string',
        },
    }
    
  • Secret: False

exclude

List of patterns of files to exclude.

Matching is performed using Python’s fnmatch.

  • Default:

    [
        '.tox', '.git', '.cache',
        '__pycache__', '**/__pycache__',
        '*.pyc', '*.egg-info'
    ]
    
  • Optional: True

  • Schema:

    {
        'type': 'list',
        'schema': {
            'type': 'string',
        },
    }
    
  • Secret: False

Speed

This source collects the network bandwidth the system curretly has using Speedtest.net servers.

Three metrics are collected:

  • ping: Latency, in milliseconds.
  • download: Download speed, in bits per second.
  • upload: Upload speed, in bits per second.

Data collected:

{
    "ping": 9.306252002716064,
    "download": 42762976.92544772,
    "upload": 19425388.307319913
}

Dependencies:

pip3 install flowbber[speed]

Usage:

[[sources]]
type = "speed"
id = "..."

    [sources.config]
    runs = 2
{
    "sources": [
        {
            "type": "speed",
            "id": "...",
            "config": {
                "host": null,
                "runs": 2
            }
        }
    ]
}

host

Use a specific server.

  • Default: None

  • Optional: True

  • Schema:

    {
        'type': 'string',
        'empty': False,
        'nullable': True,
    }
    
  • Secret: False

runs

Number of runs to perform.

  • Default: 2

  • Optional: True

  • Schema:

    {
        'type': 'list',
        'schema': {
            'type': 'integer',
            'min': 1,
        },
    }
    
  • Secret: False

Timestamp

This source allows to collect timestamps in several formats.

It is recommended to include at least one timestamp source on every pipeline in order to have a unique and consistent timestamp for the whole pipeline to use.

Data collected:

{
    "timezone": null,
    "epoch": 1502852229,
    "epochf": 1502852229.427491,
    "iso8601": "2017-08-15T20:57:09",
    "strftime": "2017-08-15 20:57:09"
}
{
    "timezone": 0,
    "epoch": 1507841667,
    "epochf": 1507841667.9304831028,
    "iso8601": "2017-10-12T20:54:27+00:00",
    "strftime": "2017-10-12 20:54:27"
}

Dependencies:

pip3 install flowbber[timestamp]

Usage:

[[sources]]
type = "timestamp"
id = "..."

    [sources.config]
    epoch = true
    epochf = true
    iso8601 = true
    strftime = "%Y-%m-%d %H:%M:%S"
{
    "sources": [
        {
            "type": "timestamp",
            "id": "...",
            "config": {
                "timezone": null,
                "epoch": true,
                "epochf": true,
                "iso8601": true,
                "strftime": "%Y-%m-%d %H:%M:%S"
            }
        }
    ]
}

timezone

Specify the timezone for which the timestamp should be calculated. If None is provided (the default), the timestamp will be calculated using the local timezone (current time). Use 0 for a UTC timestamp, and a +/-12 integer for any other timezone.

Note that this doesn’t affect the epoch or epochf timestamps, as those values are always in POSIX time which is the number of seconds since the Epoch (1970-01-01 UTC) not counting leap seconds.

  • Default: None

  • Optional: True

  • Schema:

    {
       'type': 'integer',
       'nullable': True,
       'max': 12,
       'min': -12,
    }
    
  • Secret: False

epoch

Include seconds since the EPOCH, as integer.

  • Default: True

  • Optional: True

  • Schema:

    {
        'type': 'boolean',
    }
    
  • Secret: False

epochf

Include seconds since the EPOCH, as float.

  • Default: False

  • Optional: True

  • Schema:

    {
        'type': 'boolean',
    }
    
  • Secret: False

iso8601

Include current time as a ISO 8601 string using the seconds time specification.

  • Default: False

  • Optional: True

  • Schema:

    {
        'type': 'boolean',
    }
    
  • Secret: False

strftime

Include a string timestamp using a custom format supported by Python’s strftime function.

For example: "%Y-%m-%d %H:%M:%S"

  • Default: None

  • Optional: True

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False

User

This source allows to collect information about the current user.

Data collected:

{
    "uid": 1000,
    "user": "kuralabs"
}

Dependencies:

pip3 install flowbber[user]

Usage:

[[sources]]
type = "user"
id = "..."
{
    "sources": [
        {
            "type": "user",
            "id": "...",
            "config": {}
        }
    ]
}

Valgrind Memcheck

This source parses and collects information from the XML generated by Valgrind’s Memcheck tool.

Such XML file can be generated with:

$ valgrind \
    --tool=memcheck \
    --xml=yes \
    --xml-file=memcheck.xml \
    --leak-check=full \
    ./executable

Data collected:

{
    "status":[
        {
            "state":"RUNNING",
            "time":"00:00:00:00.268"
        },
        {
            "state":"FINISHED",
            "time":"00:00:00:59.394"
        }
    ],
    "ppid":"4242",
    "preamble":{
        "line":[
            "Memcheck, a memory error detector",
            "Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.",
            "Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info",
            "Command: ./binary"
        ]
    },
    "suppcounts":{
        "pair":[
            {
                "name":"reachable memory from libstdc++ pool",
                "count":"1"
            }
        ]
    },
    "pid":"424242",
    "errorcounts":null,
    "protocoltool":"memcheck",
    "protocolversion":"4",
    "tool":"memcheck",
    "error":[
        {
            "kind":"Leak_DefinitelyLost",
            "stack":{
                "frame":[
                    {
                        "obj":"/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so",
                        "ip":"0x4C2FB55",
                        "fn":"calloc"
                    },
                    {
                        "dir":"/home/library",
                        "obj":"/home/library/binary",
                        "line":"76",
                        "ip":"0xED467F",
                        "fn":"hello_world()",
                        "file":"hello.cpp"
                    },
                ]
            },
            "xwhat":{
                "leakedblocks":"1",
                "text":"8 bytes in 1 blocks are definitely lost in loss record 1 of 5",
                "leakedbytes":"8"
            },
            "tid":"1",
            "unique":"0x0"
        }
    ],
    "args":{
        "vargv":{
            "exe":"/usr/bin/valgrind.bin",
            "arg":[
                "--track-origins=yes",
                "--leak-check=full",
                "--show-leak-kinds=all",
                "--errors-for-leak-kinds=definite",
                "--error-exitcode=1",
                "--xml=yes",
                "--xml-file=memcheck.xml",
                "--suppressions=/home/library/suppressions.supp"
            ]
        },
        "argv":{
            "exe":"./binary",
            "arg":[]
        }
    }
}

Dependencies:

pip3 install flowbber[valgrind_memcheck]

Usage:

[[sources]]
type = "valgrind_memcheck"
id = "..."

    [sources.config]
    xmlpath = "memcheck.xml"
{
    "sources": [
        {
            "type": "valgrind_memcheck",
            "id": "...",
            "config": {
                "xmlpath": "memcheck.xml"
            }
        }
    ]
}

xmlpath

Path to Valgrind’s Memcheck XML output.

  • Default: N/A

  • Optional: False

  • Schema:

    {
        'type': 'string',
        'empty': False,
    }
    
  • Secret: False