How to use pytest_runtest_call method in Pytest

Best Python code snippet using pytest

pytest_raises.py

Source:pytest_raises.py

...27       failed.28    **Parameters**29    ``item``30        The ``pytest`` test item, e.g., what is supplied to31        ``pytest_runtest_setup(item)`` or ``pytest_runtest_call(item)``.32    ``outcome``33        The ``pytest`` test outcome for the ``@pytest.hookimpl(hookwrapper=True)`` hook34        wrappers, where ``outcome = yield``.35    ``marker_name``36        The string marker name.  Values are **assumed** to be ``'setup_raises'`` or37        ``'raises'`` **only**.38        - ``'setup_raises'``: call originates from ``pytest_runtest_setup`` hook wrapper.39        - ``'raises'``: call originates from ``pytest_runtest_call`` hook wrapper.40    ``ExceptionClass``41        The exception class to re-raise.  Expected to be :class:`ExpectedException` or42        :class:`ExpectedMessage`, but not strictly required.43    ``failure_message``44        The string failure message to mark with or re-raise, depending on the value45        of ``marker_name``.46    ``traceback``47        The traceback information if available, ``None`` otherwise.48    """49    # pylint: disable=unused-variable50    __tracebackhide__ = True51    if marker_name == 'setup_raises':52        # In the later stage when `fail` is called, it is nice to "simulate" an53        # exception by putting the expected exception class's name as a prefix.54        failure_message = '{}: {}'.format(ExceptionClass.__name__, failure_message)55        item.add_marker(pytest.mark.setup_raises_expected_exc_or_message_not_found(failure_message))56    else:  # marker_name == 'raises'57        # Avoid "while handling exception another exception occurred" scenarios.58        if issubclass(ExceptionClass, PytestRaisesUsageError):59            failure_message = '{}: {}'.format(ExceptionClass.__name__, failure_message)60            pytest.fail(failure_message, pytrace=False)61        else:62            try:63                raise ExceptionClass(failure_message)64            except(ExceptionClass):65                # 1. Try and set ``outcome.excinfo``.66                # 2. Sometimes (unknown when) ``outcome.excinfo`` will trigger an67                #    AttributeError even when the test raised.  So try and set the68                #    undocumented ``outcome._excinfo`` attribute instead.69                # 3. If setting ``outcome._excinfo`` fails, fallback on ``pytest.fail``.70                excinfo = sys.exc_info()71                if traceback:72                    excinfo = excinfo[:2] + (traceback, )73                try:74                    outcome.excinfo = excinfo75                # pylint: disable=bare-except76                except:77                    try:78                        # pylint: disable=protected-access79                        outcome._excinfo = excinfo80                    # pylint: disable=bare-except81                    except:  # pragma: no cover (no tests hit this, kept for safety).82                        pytest.fail(failure_message, pytrace=False)83def _pytest_raises_validation(item, outcome, marker_name):84    """85    Validate that the test ``item`` and corresponding ``outcome`` raised an exception86    of the correct class, and if supplied the exception message was as expected.  A87    given test that has been marked with either ``@pytest.mark.setup_raises`` or88    ``@pytest.mark.raises`` can fail in one of three ways:89    1. The test raised an exception of the correct exception class, but the exception90       message did not match what was specified using either ``message`` or ``match``91       parameters.92    2. The test raised an exception of the incorrect exception class (as specified by93       the ``exception`` argument).94    3. The test was marked with either ``@pytest.mark.setup_raises`` or95       ``@pytest.mark.raises``, but no exception was raised.96    In order to support hook wrappers for both ``pytest_runtest_setup`` and97    ``pytest_runtest_call``, a "handshake" must be performed using a "secret" marker.98    This handshake is only possible because this extension implements a hook wrapper99    for both ``pytest_runtest_setup`` and ``pytest_runtest_call``.  To better explain100    the handshake, we first examine the ``pytest_runtest_call`` hook wrapper.101    ``@pytest.mark.raises(...)`` execution:102        1. The test is run (``outcome = yield``).103        2. This method is called.  If any of the three cases above that indicate failure104           happen, the test is failed.105        3. The test is failed by calling :func:`_pytest_fail_by_mark_or_set_excinfo`,106           which in this case will set ``outcome.excinfo``.107        4. By setting ``outcome.excinfo``, ``pytest`` will take over at a later stage108           and report the test as failed with our message.109    ``@pytest.mark.setup_raises(...)`` execution:110        1. The test *setup* is run (``outcome = yield``).111        2. This method is called, If any of the three cases above that indicate failure112           happen, the test is *marked* for failure.113        3. The test is failed by calling :func:`_pytest_fail_by_mark_or_set_excinfo`,114           which adds a "secret" marker that includes the failure message.115        4. Officially, the entire ``pytest_runtest_setup`` phase is completed without116           any formal failure by this extension.117        5. The ``pytest_runtest_call`` is triggered by ``pytest``, and this method is118           called again.119        6. The "secret" marker is detected, and an explicit invocation of120           ``pytest.fail`` is issued, ultimately failing the test.121    This process is unfortunately a little contrived.  However, it is done this way122    because this extension needs to be able to mark tests as failed, not error.  For123    reasons unknown to the author, any of the following issued during the124    ``pytest_runtest_setup`` hook wrapper will cause the test to **ERROR** rather than125    **FAIL**:126    - Setting ``outcome.excinfo``: during the setup phase this is a write protected127      attribute.128    - Issuing ``pytest.fail(...)``: a call to ``pytest.fail(...)`` during the setup129      phase will trigger a test **error** rather than a failure.130    .. note::131        The use of this handshake has an important implication!  Since the "secret"132        marker must be checked for first in order to fail out early, this means that133        marking a test case with **both** ``@pytest.mark.setup_raises`` and134        ``@pytest.mark.raises`` **cannot** be supported.  In practice, this should not135        be done (it does not make sense, if your setup fails you cannot run the test136        reliably).137    **Parameters**138    ``item``139        The ``pytest`` test item, e.g., what is supplied to140        ``pytest_runtest_setup(item)`` or ``pytest_runtest_call(item)``.141    ``outcome``142        The ``pytest`` test outcome for the ``@pytest.hookimpl(hookwrapper=True)`` hook143        wrappers, where ``outcome = yield``.144    ``marker_name``145        The string marker name.  Values are **assumed** to be ``'setup_raises'`` or146        ``'raises'`` **only**.147        - ``'setup_raises'``: call originates from ``pytest_runtest_setup`` hook wrapper.148        - ``'raises'``: call originates from ``pytest_runtest_call`` hook wrapper.149    """150    # pylint: disable=unused-variable151    __tracebackhide__ = True152    # Pytest 3.5+ has a new function for getting a maker from a node153    # In order to maintain compatability, prefer the newer function154    # (get_closest_marker) but use the old function (get_marker) if it155    # doesn't exist.156    marker_get_func = item.get_closest_marker if hasattr(item, 'get_closest_marker') else item.get_marker157    # Short-circuit: if the "secret" marker is found, then this test failed during setup158    # and it is now safe to ``pytest.fail`` without causing an ERROR.159    secret_marker = marker_get_func('setup_raises_expected_exc_or_message_not_found')160    if secret_marker:161        # NOTE: pytrace=False because the existing call stack is unrelated to the162        # original failure processed during `pytest_runtest_setup` hook wrapper.163        pytest.fail(secret_marker.args[0], pytrace=False)164    raises_marker = marker_get_func(marker_name)165    if raises_marker:166        exception = raises_marker.kwargs.get('exception', Exception)167        try:168            if not issubclass(exception, BaseException):169                failure_message = '@pytest.mark.{0}: supplied `exception={1}` is not a subclass of `BaseException`.'.format(170                    marker_name, exception171                )172                _pytest_fail_by_mark_or_set_excinfo(173                    item, outcome, marker_name, PytestRaisesUsageError, failure_message, None174                )175                return176        except TypeError:177            failure_message = '@pytest.mark.{}: supplied `exception` argument must be a Class, e.g., `exception=RuntimeError`.'.format(178                marker_name179            )180            _pytest_fail_by_mark_or_set_excinfo(181                item, outcome, marker_name, PytestRaisesUsageError, failure_message, None182            )183            return184        message = raises_marker.kwargs.get('message', None)185        match_pattern = raises_marker.kwargs.get('match', None)186        match_flags = raises_marker.kwargs.get('match_flags', 0)  # 0 means no flags for `re.match`187        # Only `message` or `match` should be supplied at a time, not both.188        if message and match_pattern:189            failure_message = '@pytest.mark.{}: only `message="{}"` *OR* `match="{}"` allowed, not both.'.format(190                marker_name, message, match_pattern191            )192            _pytest_fail_by_mark_or_set_excinfo(193                item, outcome, marker_name, PytestRaisesUsageError, failure_message, None194            )195            return196        raised_exception = outcome.excinfo[1] if outcome.excinfo else None197        traceback = outcome.excinfo[2] if outcome.excinfo else None198        # This plugin needs to work around the other hooks, see:199        # https://docs.pytest.org/en/latest/writing_plugins.html#hookwrapper-executing-around-other-hooks200        outcome.force_result(None)201        # Case 1: test raised exception is correct class (or derived type), check202        # message if provided by user.203        if isinstance(raised_exception, exception):204            raised_message = str(raised_exception)205            failure_message = None206            if message is not None:207                if message not in raised_message:208                    failure_message = '"{}" not in "{}"'.format(message, raised_message)209            elif match_pattern is not None:210                if not re.match(match_pattern, raised_message, match_flags):211                    failure_message = '"{}" does not match raised message "{}"'.format(match_pattern, raised_message)212            if failure_message:213                _pytest_fail_by_mark_or_set_excinfo(214                    item, outcome, marker_name, ExpectedMessage, failure_message, traceback215                )216        # Case 2: test raised exception, but it was of an unexpected type.217        elif raised_exception:218            failure_message = 'Expected exception of type {}, but got exception of type {} with message: {}'.format(219                exception, type(raised_exception), str(raised_exception)220            )221            _pytest_fail_by_mark_or_set_excinfo(222                item, outcome, marker_name, ExpectedException, failure_message, traceback223            )224        # Case 3: test did _not_ raise exception, but was expected to.225        else:226            failure_message = 'Expected exception {}, but it did not raise'.format(exception)227            _pytest_fail_by_mark_or_set_excinfo(228                item, outcome, marker_name, ExpectedException, failure_message, traceback229            )230@pytest.hookimpl(hookwrapper=True)231def pytest_runtest_setup(item):232    # pylint: disable=unused-variable233    __tracebackhide__ = True234    outcome = yield235    _pytest_raises_validation(item, outcome, 'setup_raises')236@pytest.hookimpl(hookwrapper=True)237def pytest_runtest_call(item):238    # pylint: disable=unused-variable239    __tracebackhide__ = True240    outcome = yield241    _pytest_raises_validation(item, outcome, 'raises')242# NOTE: this gets evaluated by consuming packages only.243def pytest_configure(config):  # pragma: no cover244    """245    Register the markers with pytest.246    See: https://docs.pytest.org/en/latest/writing_plugins.html#registering-markers247    """248    config.addinivalue_line(249        'markers',250        'setup_raises: expect pytest_runtest_setup phase to raise.'251    )...

test_pytest.py

Source:test_pytest.py

1pytest_plugins = ['pytester']2def test_group_runs_appropriate_tests(testdir):3    testdir.makepyfile("""4        def test_x(): pass5        def test_y(): pass6        def test_z(): pass7    """)8    result = testdir.runpytest_subprocess('--test-group-count', '2', '--test-group', '1')9    result.assert_outcomes(passed=2)10    result.stdout.fnmatch_lines([11        'Running test group #1 (2 tests)'12    ])13    result = testdir.runpytest_subprocess('--test-group-count', '2', '--test-group', '2')14    result.assert_outcomes(passed=1)15    result.stdout.fnmatch_lines([16        'Running test group #2 (1 tests)'17    ])18def test_group_runs_all_test(testdir):19    """Given a large set of tests executed in random order, assert that all20    tests are executed.21    """22    testdir.makepyfile("""23        def test_b(): pass24        def test_c(): pass25        def test_d(): pass26        def test_e(): pass27        def test_f(): pass28        def test_g(): pass29        def test_h(): pass30        def test_i(): pass31        def test_j(): pass32        def test_k(): pass33        def test_l(): pass34        def test_m(): pass35        def test_n(): pass36        def test_o(): pass37        def test_p(): pass38        def test_q(): pass39        def test_r(): pass40        def test_s(): pass41        def test_t(): pass42        def test_u(): pass43        def test_v(): pass44        def test_w(): pass45        def test_x(): pass46        def test_y(): pass47        def test_z(): pass48    """)49    result = testdir.inline_run('--test-group-count', '2',50                                '--test-group', '1',51                                '--test-group-random-seed', '5')52    group_1 = [x.item.name for x in result.calls if x._name == 'pytest_runtest_call']53    result.assertoutcome(passed=13)54    result = testdir.inline_run('--test-group-count', '2',55                                '--test-group', '2',56                                '--test-group-random-seed', '5')57    group_2 = [x.item.name for x in result.calls if x._name == 'pytest_runtest_call']58    result.assertoutcome(passed=12)59    result = testdir.inline_run('--test-group-count', '1',60                                '--test-group', '1',61                                '--test-group-random-seed', '5')62    all_tests = [x.item.name for x in result.calls if x._name == 'pytest_runtest_call']...

test_runtest_hook.py

Source:test_runtest_hook.py

1from __future__ import absolute_import, division, print_function2import logging3import pytest4logger = logging.getLogger('pytest_catchlog.test.perf')5@pytest.fixture(autouse=True)6def bench_runtest(request, benchmark):7    # Using benchmark.weave to patch a runtest hook doesn't seem to work with8    # pytest 2.8.3; for some reason hook gets called more than once, before9    # running benchmark cleanup finalizer, resulting in the10    # "FixtureAlreadyUsed: Fixture can only be used once" error.11    #12    # Use plain old monkey patching instead.13    ihook = request.node.ihook14    saved_hook = ihook.pytest_runtest_call15    def patched_hook(*args, **kwargs):16        ihook.pytest_runtest_call = saved_hook  # restore early17        return benchmark(saved_hook, *args, **kwargs)18    ihook.pytest_runtest_call = patched_hook19    benchmark.group = 'runtest'20@pytest.yield_fixture  # because 'caplog' is also a yield_fixture21def stub():22    """No-op stub used in place of 'caplog'.23    Helps to measure the inevitable overhead of the pytest fixture injector to24    let us exclude it later on.25    """26    yield27def test_fixture_stub(stub):28    logger.info('Testing %r hook performance: %s',29                'catchlog', 'pure runtest hookwrapper overhead')30def test_caplog_fixture(caplog):31    logger.info('Testing %r hook performance: %s',...

test-plugin.py

Source:test-plugin.py

...11class MyPlugin(object):12    def __init__(self):13        self.__name__ = "myplugin"14        self.collected = set()15    def pytest_runtest_call(self, item):16        print('pytest_runtest_call')17        print('item: %s' % item)18    def pytest_xdist_node_collection_finished(self, node, ids):19        self.collected.update(set(ids))20        print('pytest_xdist_node_collection_finished')21    def pytest_collection_modifyitems(self, items):22        print('pytest_collection_modifyitems')23        for item in items:24            self.collected.update({item.nodeid})25            26# Execute Tests27def main():28    # This run causes `pytest_runtest_call` to fire29    args = ["test-plugin.py", "-s",]...

Pytest Tutorial

Looking for an in-depth tutorial around pytest? LambdaTest covers the detailed pytest tutorial that has everything related to the pytest, from setting up the pytest framework to automation testing. Delve deeper into pytest testing by exploring advanced use cases like parallel testing, pytest fixtures, parameterization, executing multiple test cases from a single file, and more.

Chapters

What is pytest
Pytest installation: Want to start pytest from scratch? See how to install and configure pytest for Python automation testing.
Run first test with pytest framework: Follow this step-by-step tutorial to write and run your first pytest script.
Parallel testing with pytest: A hands-on guide to parallel testing with pytest to improve the scalability of your test automation.
Generate pytest reports: Reports make it easier to understand the results of pytest-based test runs. Learn how to generate pytest reports.
Pytest Parameterized tests: Create and run your pytest scripts while avoiding code duplication and increasing test coverage with parameterization.
Pytest Fixtures: Check out how to implement pytest fixtures for your end-to-end testing needs.
Execute Multiple Test Cases: Explore different scenarios for running multiple test cases in pytest from a single file.
Stop Test Suite after N Test Failures: See how to stop your test suite after n test failures in pytest using the @pytest.mark.incremental decorator and maxfail command-line option.