Posted on

Execute tests in parallel with Behave BDD

Note (Sep 2021) :

I used this Behave BDD framework few years ago and no longer use it. I highly recommend the new Python framework called Pytest . It is very powerful, natively support running tests in parallel with pytest-xdist plugin. It also supports BDD style with pytest-bdd plugin.

By default, Behave does not have any option to run tests parallel. We need to create a custom behave runner to support parallel by take advantage of 2 packages named multiprocessing and subprocess in Python which able to handle multiple processes.

It is combined with Selenium Grid to run the regression suite in parallel. for how to setup Docker Selenium Grid, refer Setup Docker Selenium Grid for Parallel Test Execution on Ubuntu 16.04 LTS

Ideally , all scenarios in test suite should be run in parallel. But It will depend a lot to the product you test, the environment, your database… which leads some tests need to run independently. So you will see the script below have both options, run parallel and consequential. pass the relevant argument value, you decide which type the suite run. Of course , before that you need to add corresponding tag for each scenario.

Python file for Parallel test execution.

We need to execute test parallel. For doing that we need to create a python file named as ‘behave_parallel.py‘  this file will find all the feature files, step definitions and run them parallel.

In this python file, you can customize what arguments and default value for parallel execution. the arguments list I used below:

def parse_arguments():
"""
Parses commandline arguments
:return: Parsed arguments
"""
parser = argparse.ArgumentParser('Running in parallel mode. Do not use features and tags argument at the same time')
parser.add_argument('--suite', '-s', help='Please specify the suite you want to run. Default suite is regression',
default='regression_suite')
parser.add_argument('--feature_list', '-l', help='Please specify file path of features or features location you want to run.')
parser.add_argument('--feature', '-f', help='Please specify feature you want to run.')
parser.add_argument('--processes', '-p', type=int, help='Maximum number of processes. Default = 5', default=5)
parser.add_argument('--tags', '-t', help='Please specify behave tags to run')
parser.add_argument('--outfile_prefix', '-o', help='Please specify outfile prefix to run')
return parser.parse_args()

full script will be:

from multiprocessing import Pool
from subprocess import call, Popen, PIPE
from functools import partial
from glob import glob
import logging
import argparse
import json

logging.basicConfig(level=logging.INFO, format="[%(levelname)-8s %(asctime)s] %(message)s")
logger = logging.getLogger(__name__)


def parse_arguments():
    """
    Parses commandline arguments
    :return: Parsed arguments
    """
    parser = argparse.ArgumentParser('Running in parallel mode. Do not use features and tags argument at the same time')
    parser.add_argument('--suite', '-s', help='Please specify the suite you want to run. Default suite is regression',
                        default='regression_suite')
    parser.add_argument('--feature_list', '-l', help='Please specify file path of features or features location you want to run.')
    parser.add_argument('--feature', '-f', help='Please specify feature you want to run.')
    parser.add_argument('--processes', '-p', type=int, help='Maximum number of processes. Default = 5', default=5)
    parser.add_argument('--tags', '-t', help='Please specify behave tags to run')
    parser.add_argument('--outfile_prefix', '-o', help='Please specify outfile prefix to run')
    return parser.parse_args()


def _run_parallel_feature(feature):
    """
    Runs features without tags @sequential
    :param feature: Feature will be run
    :type feature: str
    """
    logger.debug('Processing feature: {}'.format(feature))
    feature_test_log = feature.replace('/', '-')
    cmd = 'behave {feature} --tags ~@sequential>> ~/log/{feature_test_log}.txt'.format(feature=feature,
                                                                                       feature_test_log=feature_test_log)
    r = call(cmd, shell=True)
    status = 'Passed' if r == 0 else 'Failed'
    print('{0:50}: {1}!!'.format(feature, status))


def _run_sequential_feature(feature, outfile_prefix):
    """
    Runs features with tags @sequential
    :param feature: Feature will be run
    :type feature: str
    """
    logger.debug('Processing feature: {}'.format(feature))
    feature_test_log = feature.replace('/', '-')
    cmd = 'behave {feature} --tags @sequential --tags {outfile_prefix} >> ~/log/{outfile_prefix}_{feature_test_log}.txt'.format(
        feature=feature, outfile_prefix=outfile_prefix, feature_test_log=feature_test_log)
    r = call(cmd, shell=True)
    status = 'Passed' if r == 0 else 'Failed'
    print('{0:50}: {1}!!'.format(feature, status))


def main():
    """
    Runner
    """
    args = parse_arguments()
    pool = Pool(args.processes)
    if not args.feature_list and not args.feature and not args.tags:
        """
        Run all features in system
        """
        features = glob('{suite}/*.feature'.format(suite=args.suite))
    elif args.feature_list and not args.tags:
        """
        Run feature list defined by users
        """
        file = open(args.feature_list)
        features = []
        for feature in file:
            feature = feature.replace('\n', '')
            features.append(feature) if '/' in feature else features.append(args.suite + '/' + feature)
    elif args.feature_list and args.tags:
        """
        Run feature list with specific tag defined by users
        """
        file = open(args.feature_list)
        features = []
        for feature in file:
            feature = feature.replace('\n', '')
            feature = args.suite + '/' + feature if '/' not in feature else feature
            cmd = 'behave {feature} --tags {tag} -d -f json --no-summary'.format(feature=feature, tag=args.tags)
            p = Popen(cmd, stdout=PIPE, shell=True)
            out, err = p.communicate()
            if json.loads(out.decode()):
                scenarios = json.loads(out.decode())[0]['elements']
                for scenario in scenarios:
                    features.append(scenario['location'])
    else:
        cmd = ''
        if args.feature and args.tags:
            """
            Run a feature with specific tag
            """
            cmd = 'behave {suite}/{feature} -d -f json --no-summary -t {tags}'.format(
                suite=args.suite, feature=args.feature, tags=args.tags)

        elif args.tags:
            """
            Run tags defined by user input
            """
            cmd = 'behave {suite}/. -d -f json --no-summary -t {tags}'.format(suite=args.suite, tags=args.tags)
        p = Popen(cmd, stdout=PIPE, shell=True)
        out, err = p.communicate()
        scenarios = json.loads(out.decode())[0]['elements']
        features = [scenario['location'] for scenario in scenarios]

    if args.outfile_prefix:
        pool.map(partial(_run_sequential_feature, outfile_prefix=args.outfile_prefix), features)
    else:
        pool.map(_run_parallel_feature, features)

if __name__ == '__main__':
    main()

Then you can trigger the execution by runnung below command:

cd ~/thachhoang/myProject/tests/behave
python ./behave_parallel.py --feature_list ~/thachhoang/myProject/tests/feature_list.txt --tags ~sequential --processes 12

Here is the structure of my behave framework :
├ behave
├─├ conf // user and environment configuration
├─├ regression_suite // regression suite directory
├─├── ├steps // steps directory which contains step definition
├ ────├─ login_steps.py
├─├───login.feature
├─├ page_objects
├─├ behave_parallel.py // custom runner to trigger tests in parallel
├─├ .behaverc

22 thoughts on “Execute tests in parallel with Behave BDD”

  1. Do you have any project of example?
    how can i run behave_parallel.py and in witch part of proyect I can put it?

    Like

    Reply

    1. Sorry, I don’t have a public example project.
      You can put the behave_parallel.py in the root of your project. basically, it is like a wrapper of behave runner. you need to pass the feature list, appropriate tags and the number of concurrent processes, the behave_parallel.py file will read the feature and tags , splitting them into scenarios and deliver them to each process to archive parallel execution.I added the full command to trigger the script.

      Like

      Reply

  2. I have tried to use but it asks for a JSON File and its structure or example is not specified in the above text. Can you give an hint on how it should be structured?

    Like

    Reply

    1. Here is the sample structure:
      ├ behave
      ├─├ conf // user and environment configuration
      ├─├ regression_suite // regression suite directory
      ├─├── ├steps // steps directory which contains step definition
      ├ ────├─ login_steps.py
      ├─├───login.feature
      ├─├ page_objects
      ├─├ behave_parallel.py // custom runner to trigger tests in parallel
      ├─├ .behaverc

      cd behave
      python ./behave_parallel.py –feature_list ~/path_to_feature_list/feature_list.txt –tags ~sequential –processes 12

      “I have tried to use but it asks for a JSON File”
      if you are getting error about json file, it means something wrong with this command:
      cmd = ‘behave {feature} –tags {tag} -d -f json –no-summary’.format(feature=feature, tag=args.tags)
      this command will execute dryrun and return a list scenario in json format, from this json file, it will collect all scenario locations to execute tests in parallel. make sure you pass correct tag, located in correct directory.
      above sample structure, I use custom feature directory, so I set this line in .behaverc file.
      [behave]
      paths=regression_suite/.
      Hope it help.

      Like

      Reply

  3. ‘behave {suite}/{feature} -d -k -f json –no-summary -t {tags}’.format(
    suite=args.suite, feature=args.feature, tags=args.tags)…
    Adding -k resolved my JSON error

    Like

    Reply

  4. Hi,

    I am getting error as below. Can you please advice

    ./behave_parallel.py –feature features\adstore.feature
    Traceback (most recent call last):
    File “./behave_parallel.py”, line 122, in
    main()
    File “./behave_parallel.py”, line 112, in main
    scenarios = json.loads(out.decode())[0][‘elements’]
    File “C:\Users\akumar\AppData\Local\Programs\Python\Python38-32\lib\json\__init__.py”, line 357, in loads
    return _default_decoder.decode(s)
    File “C:\Users\akumar\AppData\Local\Programs\Python\Python38-32\lib\json\decoder.py”, line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File “C:\Users\akumar\AppData\Local\Programs\Python\Python38-32\lib\json\decoder.py”, line 355, in raw_decode
    raise JSONDecodeError(“Expecting value”, s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

    Like

    Reply

      1. Hi hoang,
        Thanks for the article.
        I am using below directory structure for my project and made changes as you suggested but I am still seeing error that script can’t find my step definitions with json decode error.

        Project
        –Test
        –features (all the features are listed here)
        –steps (all step definitions)

        behave_parallel.py

        Can you please help me with change so that my script can find step definitions.
        Thanks

        Like

      2. Steps folder is inside features folder. And I got it done using –suites option in command but I am now getting eroor with json.loads (out.decode())[0] [‘elements’]
        Line.

        Like

      3. –suites is the argument that I created to run all features in a directory. in your case, it is “features” directory. And if you use this argument, use only it, DO NOT use other args like –feature or –tags. the right command would be:
        cd ~/thachhoang/myProject/tests/behave
        python ./behave_parallel.py –suites features –processes 12

        If you have to combine those arguments (suite, feature, tags), you will need to update my test script to do that.

        Like

  5. Hi! Thanks for the article, it was very helpful!
    Did you adjust your tests somehow to make them thread safe? I implemented similar solution, but I paralleling by scenarios and when I run tests sequentially they all green, but when I run in parallel 15-20% of them fail. Browser instance and all dynamic context variables are created before each scenario. And I also removed all before_feature steps and moved them to features as backgrounds, but results are still the same.
    We’re running tests now on one machine in 4 threads and we have also sequential tests which can not be run in parallel.
    If you could come up with any idea, it would be very helpful 🙂

    Like

    Reply

    1. Hi, Firstly, I suppose you already used a customized runner to support run tests in parallel. You may use multiprocessing like I did in this post or you can use multi-threading to enable parallel. And both options are thread safe. Each sub-process in multiprocessing will have it own memory space copy, so it is safe for running tests concurrently.

      Next, targeting to the problem you are encountering which many tests are failed when running concurrently. You did good job when moving code from before_feature into feature background, or you also can move it to before_scenario.

      But I think your tests are still inter-dependent on each others. The common reason I have seen is that these tests are accessing the same test data. eg: scenario 1 try to login User1 while scenario2 try to reset User1 password or some scenarios try to edit the same test data. If that test data is mandatory for some tests, tests should only read that data, not changing it.

      So, you need to scan all tests and make sure they are independent.

      This answer is kind of unspecific but I think if you done this, you certantly can run parallel effectively.

      Like

      Reply

      1. Sorry man, I completely forgot to reply 😦
        Thanks for answering!
        Yes, I’m using a multiprocessing similar to what you showed in this example. I’m looking deeply into tests to find dependencies now I’m still new to the project and can’t see all dependencies from the first sight, I still have sometimes random tests failed with Wrong webdriver session id exception, maybe you had such problem trying to implement your solution?

        Like

      2. Nothing special on what I am using. Basically, I only create a WebDriver instance in before_scenario and quit all browser windows in after_scenario.
        def before_scenario(context, scenario):
        context.browser = init_browser_session(context)

        def after_scenario(context, scenario):
        context.browser.quit()

        I think you should post your problem with full log and exception stack trace on stackoverflow. You will need to provide more info in order to find its root cause.

        Like

  6. Hi hoang,
    My project exact folder structure is:
    Project
    –Test
    —–features
    —–demo.feature
    ———steps
    ———step_impl_demo.py
    –behave_parallel.py

    I want to execute this particular feature and I am executing below command:
    python ./behave_parallel.py –suite Test/features –feature demo.feature –tag test –processes 12

    I am getting Json decode error here.

    ” raise JSONDecodeError(“Expecting value”, s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)”

    I think I am somehow messing withthe folder structure and base path which being used here to read all the steps.
    I have checked the json output of Out variable. It is having 3 statements before the json

    Supplied path : “”
    Trying base directory: “D:\Abhishek\project\Test\featured”

    Like

    Reply

    1. I have this exact same issue. it only happens when I try to define features with specific tags. My Output goes as follows:
      b’Supplied path: “features/feature/01_simple_tests.feature”\nPrimary path is to a file so using its directory\nTrying base directory: /Users/robmeyer/Projects/Python/Traject-Automated-UI-Tests/features/feature\nTrying base directory: /Users/robmeyer/Projects/Python/Traject-Automated-UI-Tests/features\n[\n{\n “description”: “”,\n “elements”: [\n {\n “description”: “”,\n “id”: “vpl-traject-simple-tests;validate-a-user-can-login-as-superuser”,\n “keyword”: “Scenario”,\n “line”: 5,\n “location”: “features/feature/01_simple_tests.feature:5”,\n “name”: “Validate a user can login as Superuser”,\n “steps”: [\n {\n “keyword”: “Given”,\n “line”: 6,\n “match”: {\n “location”: “features/steps/login.py:16″\n },\n “name”: “the browser is open to VPL Traject”,\n “result”: {\n “duration”: 0,\n “status”: “skipped”\n },\n “step_type”: “given”\n },\n {\n “keyword”: “When”,\n “line”: 7,\n “match”: {\n “location”: “features/steps/login.py:34″\n },\n “name”: “the user is logged in as Superuser”,\n “result”: {\n “duration”: 0,\n “status”: “skipped”\n },\n “step_type”: “when”\n }\n ],\n “tags”: [\n {\n “line”: 4,\n “name”: “pr_smoke”\n }\n ],\n “type”: “scenario”\n },\n {\n “description”: “”,\n “id”: “vpl-traject-simple-tests;validate-a-user-can-login-as-customer-admin”,\n “keyword”: “Scenario”,\n “line”: 10,\n “location”: “features/feature/01_simple_tests.feature:10”,\n “name”: “Validate a user can login as Customer Admin”,\n “steps”: [\n {\n “keyword”: “Given”,\n “line”: 11,\n “match”: {\n “location”: “features/steps/login.py:16″\n },\n “name”: “the browser is open to VPL Traject”,\n “result”: {\n “duration”: 0,\n “status”: “skipped”\n },\n “step_type”: “given”\n },\n {\n “keyword”: “When”,\n “line”: 12,\n “match”: {\n “location”: “features/steps/login.py:34″\n },\n “name”: “the user is logged in as Customer Admin”,\n “result”: {\n “duration”: 0,\n “status”: “skipped”\n },\n “step_type”: “when”\n }\n ],\n “tags”: [\n {\n “line”: 9,\n “name”: “prod_smoke”\n },\n {\n “line”: 9,\n “name”: “pr_smoke”\n }\n ],\n “type”: “scenario”\n }\n ],\n “id”: “vpl-traject-simple-tests”,\n “keyword”: “Feature”,\n “line”: 3,\n “name”: “VPL Traject Simple Tests”,\n “status”: “skipped”,\n “tags”: [\n {\n “line”: 2,\n “name”: “simple”\n }\n ],\n “uri”: “features/feature/01_simple_tests.feature”\n}\n]\n0 features passed, 0 failed, 0 skipped, 1 untested\n0 scenarios passed, 0 failed, 0 skipped, 2 untested\n0 steps passed, 0 failed, 0 skipped, 0 undefined, 4 untested\nTook 0m0.000s\n’

      Any help is much appreciated.

      Like

      Reply

  8. python ./behave_parallel.py –feature abc.feature –tags @jsonschema –processes 5.
    zsh: no such user or named directory: jsonschema

    please help

    Like

    Reply

  9. Hello, how much work is it to convert your project that uses behave to pytest? For the parallel behave run, did you run into tests that took over an hour to complete that should only take less than 10 minutes?

    Like

    Reply

    1. Hi Esther,
      If the framework or test scripts themselves are not designed well for running in parallel, you may encounter that issue.

      How much work to move from behave to pytest-bdd depends on how your framework is designed, the number of existing tests you have, so I can’t give the exact number, but I think it will not required too much work.
      Here are some thoughts I can share:
      – First, You should build a POC using pytest-bdd, migrate one or some tests from behave fw and see how it works.
      – Then optimize your existing behave fw, steps definition should depends on Page Object , not calling webdriver api(or any libs you are using for ui/api testing) or using locators in step files. the idea is abstracting the interactions with the ui elements. Then it will be easier for you to bring all the behave feature files and steps definition to pytest-bdd fw as they are similar.
      – in behave, to share data between steps including webdriver instance, we use “context”. when moving to pytest-bdd , it has a powerful thing called fixtures. you can create a fixture, “scenario_context” for example to share data and create fixture “browser” to manage WD.
      – you may need to check the differences in the step parameterization, regular expression, hooks between behave and pytest-bdd.

      Like

      Reply

Leave a comment