体育投注开户

Salt is ending a journey that started three years ago with the  to add . In the upcoming weeks, Salt  to PyTest as its sole test runner, as the previous and heavily customized unit test runner, , finally reaches EOL.

Why?

runtests.py served us well, but every time the Salt test suite got a new sub-directory, runtests.py had to be updated in order to pick up those new tests. Often enough, those tests or sub-directories, would be added and never actually run on the CI pipelines because the needed updates to runtests.py were lacking.

When we needed additional daemons running for the test suite, for example the multimaster tests – again, it required an update to runtests.py and  a  of code duplication.

There needed to be a better way moving forward.

What’s The Gain?

If you know PyTest, you already know what it brings to the table. If you’re not familiar, here are a few highlights:

  • If you need “something” added to your test suite, it’s likely that someone already  implementing it.
  • Selecting a group of tests, a single test, or a test class is , especially when compared to runtests.py.
    • Running all fileserver related tests:nox -e 'pytest-zeromq-3(coverage=False)' -- -k fileserver
    • Running all proxy related tests:nox -e 'pytest-zeromq-3(coverage=False)' -- -k proxy
    • Running Windows specific tests, which commonly contain win in either the test module name or test name:nox -e 'pytest-zeromq-3(coverage=False)' -- -k win
    • Running the failed tests after a test run, stopping at first failure:nox -e 'pytest-zeromq-3(coverage=False)' -- --lf --maxfail=1

  • Fixtures are powerful and bring code de-duplication to the table. The less code you have to maintain, the better, especially when talking about the test suite. Imagine you have a big setup routine to prepare for a test. In the  world you would use  or . Now, what if that setup routine had to be used in multiple test classes?
    Likely you would either duplicate code or create a base class to reuse the code. In PyTest, you create a function, the fixture, and every test that needs that prep work just needs to add the function name(fixture name) as an , and
  • it “Just Works!” You can even  and provide  for the fixtures!

  • PyTest is smart enough that all you need to do on your test function is  something. It knows how to handle python’s builtin types and  if needed. Also, in case of failure, it adds useful information on the error output to hint as to why it failed. For example:$ pytest test_assert2.py =========================== test session starts ============================ platform linux -- Python 3.x.y, pytest-6.x.y, py-1.x.y, pluggy-0.x.y cachedir: $PYTHON_PREFIX/.pytest_cache rootdir: $REGENDOC_TMPDIR collected 1 item test_assert2.py F [100%] ================================= FAILURES ================================= ___________________________ test_set_comparison ____________________________ def test_set_comparison(): set1 = set("1308") set2 = set("8035") > assert set1 == set2 E AssertionError: assert {'0', '1', '3', '8'} == {'0', '3', '5', '8'} E Extra items in the left set: E '1' E Extra items in the right set: E '5' E Use -v to get the full diff test_assert2.py:6: AssertionError ========================= short test summary info ========================== FAILED test_assert2.py::test_set_comparison - AssertionError: assert {'0'... ============================ 1 failed in 0.12s =============================

体育投注开户There’s a lot more to gain by using PyTest and also a lot of resources on the internet that you can read to know more.

What Changes?

体育投注开户To begin with, most of what changes is the way we run the test suite.

  • Run Unit Tests:nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/ using the deprecated runtests.py the command would have been:nox -e 'runtests-zeromq-3(coverage=False)' -- --unit-tests
  • Run unit and integration tests for states:nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/states/ tests/integration/states/ using the deprecated runtests.py the command would have been:nox -e 'runtests-zeromq-3(coverage=False)' -- --state-tests
  • Run integration tests for an individual module:nox -e 'pytest-zeromq-3(coverage=False)' -- tests/integration/modules/test_virt.py using the deprecated runtests.py the command would have been:nox -e 'runtests-zeromq-3(coverage=False)' -- -n integration.modules.test_virt
  • Run unit tests for an individual module:nox -e 'pytest-zeromq-3(coverage=False)' -- tests/unit/modules/test_virt.py using the deprecated runtests.py the command would have been:nox -e 'runtests-zeromq-3(coverage=False)' -- -n unit.modules.test_virt
  • Run an individual test by using the class and test name:nox -e 'pytest-zeromq-3(coverage=False)' -- 'tests/integration/modules/test_virt.py::VirtTest::test_default_kvm_profile' using the deprecated runtests.py the command would have been:nox -e 'runtests-zeromq-3(coverage=False)' -- -n integration.modules.test_virt.VirtTest.test_default_kvm_profile
  • Run all macOS related tests(which commonly contain mac on the test module or test name):nox -e 'pytest-zeromq-3(coverage=False)' -- -k mac using the deprecated runtests.py the command would have been, impossible
  • Run all macOS and Windows tests:nox -e 'pytest-zeromq-3(coverage=False)' -- -k 'mac or win' using the deprecated runtests.py the command would have been, impossible
  • Run all MySQL related tests(which commonly contain mysql on the test module or test name):nox -e 'pytest-zeromq-3(coverage=False)' -- -k mysql using the deprecated runtests.py the command would have been, impossible.

As the Salt test suite evolves,  are going to be written for PyTest. Existing tests will likely be  for PyTest (most, likely not all, since we have +11k tests).

What About My Open Pull Requests?

体育投注开户At this stage, the Salt project will not require open pull requests to rewrite their tests for PyTest.

For new code submissions, PyTest tests are preferred. In the event that tests are being added to an existing unittest test module, it’s really not a problem. We’re already super exited that you’re contributing code with tests!

However, please note that  tests only get a  of PyTest’s mighty power.

What’s Next?

We will continue to adapt the test suite for PyTest by creating any necessary foundational code required and rewriting almost all体育投注开户 of the testing documentation.