Waiting without pytest and without starting the server #251

Activity

1. csernazs commented on May 22, 2023

   

   csernazs

   on May 22, 2023

   Owner

   More actions

   Hi Boldizsár,

   For the first sight I thing doing server stop/start from the wait() method would be a layering violation, as wait() operates at how the requests are handled, while the server stopping and starting manages the server itself.

   In pytest we also don't stop the server after each test as it causes some time penalty (0.1 sec or similar), so instead of stopping/starting we are clearing the server state instead. Because of this, it would be complicated to implement the stopping logic in the wait method.

   I think with the unittest code you have, there are still possibilities.

   1. you can run unittest code with pytest (although I have no experience about how the fixture works in that case)
   2. you can put server starting to setUp and stopping to tearDown methods, so it will be started and stopped. You can store the httpserver instance to an attribute in such case. In this, case the server will be started/stopped for earch test.
   3. you can put server starting and stopping to setUpClass and tearDownClass (which are classmethods) so the server will be started for the class and it will survive the tests. In such case you need to cleanup the server state by calling server.clear() call, this ensures that there's no cross-talk between the tests (removes the handlers, etc).
   4. if there's no reason to use the wait method, you don't have to. If your code you test will finish, you can still examine the log attribute of the server object for the requests (and raise an error if there were none).
   5. you can start the httpserver as a module variable, and then use it for the whole process, and let the test process to do the cleanup (this is not the nicest one, but still would work). In such case you would just need to call the clear() method descibed in step 3 between the tests.

   What do you think, would any of these work for you?
2. palotasb commented on May 23, 2023

   

   palotasb

   on May 23, 2023

   Author

   More actions

   put server starting to setUp and stopping to tearDown methods, so it will be started and stopped. You can store the httpserver instance to an attribute in such case. In this, case the server will be started/stopped for earch test.

   This is what I actually have currently but maybe I'll switch to the classmethods and .clear()-ing to speed up things a bit. I agree that making the server a module level object wouldn't be very nice so I think I'll avoid that.

   I haven't considered not wait()-ing at all, and looking at the .log instead, but that might make the test methods even simpler so I'll give it a shot. Actually we have a few places where our client code is supposed to not make any requests at all, so we might need to inspect the log anyway.

   I think I mostly wanted to .wait() because of the stop_on_nohandler=True, raise_assertions=True options, i.e., turning unexpected requests into exceptions.

   We do run the unittest test code via pytest (and tox) but as far as I know we can't use pytest test fixtures in this case.
3. csernazs commented on May 23, 2023

   

   csernazs

   on May 23, 2023

   Owner

   More actions

   I found that there's setUpModule and tearDownModule also, for module level setup and teardown.
   So this would allow you to define more classes in the module, all receiving httpserver.

   Example:

   import unittest
   from pytest_httpserver import HTTPServer
   import requests
   
   httpserver = HTTPServer()
   
   
   def setUpModule():
       httpserver.start()
   
   
   def tearDownModule():
       httpserver.stop()
   
   
   class TestMyStuff(unittest.TestCase):
       def test_foo(self):
           httpserver.expect_request("/foo").respond_with_data("OK")
           self.assertEqual(requests.get(httpserver.url_for("/foo")).text, "OK")
   
       def test_bar(self):
           # no cross-talk
           self.assertNotEqual(requests.get(httpserver.url_for("/foo")).text, "OK")
   
       def tearDown(self):
           httpserver.clear()

   There's some limited support for fixtures: https://docs.pytest.org/en/7.1.x/how-to/unittest.html#mixing-pytest-fixtures-into-unittest-testcase-subclasses-using-marks

   But using setUp and tearDown (module and class levels, as well) is the idiomatic way to use the unittest module, I think.
4. palotasb commented on May 23, 2023

   

   palotasb

   on May 23, 2023

   Author

   More actions

   For some reason, having this code without any server.wait() hangs tox/pytest/unittest indefinitely after the tests have completed (the process doesn't quit).

   # test_pytest_httpserver.py
   # python -m unittest test_pytest_httpserver
   
   import unittest
   import requests
   from pytest_httpserver import HTTPServer
   
   class TestMyStuff(unittest.TestCase):
       def setUp(self):
           self.server = HTTPServer()
           self.server.start()
   
       def tearDown(self):
           self.server.check()
           self.server.stop()
   
       def test_the_test_server_setup(self):
           self.server.expect_oneshot_request("/foobar").respond_with_json({"foo": "bar"})
           requests.get(self.server.url_for("/UNEXPECTED"))
           self.assertEqual(len(self.server.log), 1)

   $ python -m unittest test_pytest_httpserver
   127.0.0.1 - - [23/May/2023 11:10:53] "GET /UNEXPECTED HTTP/1.1" 500 -
   F
   ======================================================================
   FAIL: test_the_test_server_setup (test_pytest_httpserver.TestMyStuff)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/example/test_pytest_httpserver.py", line 11, in tearDown
       self.server.check()
     File "/Users/example/venv/venv-pytest-httpserver/lib/python3.8/site-packages/pytest_httpserver/httpserver.py", line 776, in check
       self.check_assertions()
     File "/Users/example/venv/venv-pytest-httpserver/lib/python3.8/site-packages/pytest_httpserver/httpserver.py", line 795, in check_assertions
       raise AssertionError(assertion)
   AssertionError: No handler found for request <Request 'http://localhost:53222/UNEXPECTED' [GET]>.
   Ordered matchers:
       none
   
   Oneshot matchers:
       <RequestMatcher uri='/foobar' method='__ALL' query_string=None headers={} data=None json=<UNDEFINED>>
   
   Persistent matchers:
       none
   
   ----------------------------------------------------------------------
   Ran 1 test in 0.017s
   
   FAILED (failures=1)
   ^CException ignored in: <module 'threading' from '/opt/homebrew/Caskroom/miniconda/base/lib/python3.8/threading.py'>
   Traceback (most recent call last):
     File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.8/threading.py", line 1388, in _shutdown
       lock.acquire()
   KeyboardInterrupt:

   Is there something I'm doing wrong here? Note that after the line FAILED (failures=1) the process hung and I hit ^C to quit.
5. palotasb commented on May 23, 2023

   

   palotasb

   on May 23, 2023

   Author

   More actions

   Now I think I remember better, this is why I started using .wait() in the first place, to avoid the process hanging in some test cases (even if they are failed test cases). Maybe I can wrap each test case in a .wait() automatically by creating a waiter in the setUp() and finishing it in the tearDown().
6. palotasb commented on May 23, 2023

   

   palotasb

   on May 23, 2023

   Last edited by palotasb

   Author

   More actions

   This is the current best setup I have:

   # test_pytest_httpserver.py
   # python -m unittest test_pytest_httpserver
   
   import contextlib
   import unittest
   import requests
   from pytest_httpserver import HTTPServer
   
   class TestMyStuff(unittest.TestCase):
       def setUp(self):
           self.server = HTTPServer()
           self._exit_stack = contextlib.ExitStack()
           self.server.start()
           self._exit_stack.callback(self.server.stop)
           self._exit_stack.callback(self.server.check)
           waiter = self.server.wait()
           self._exit_stack.enter_context(waiter)
   
       def tearDown(self):
           self._exit_stack.close()
   
       def test_the_test_server_setup(self):
           self.server.expect_oneshot_request("/foobar").respond_with_json({"foo": "bar"})
           response = requests.get(self.server.url_for("/foobar"))
           response.raise_for_status()
           self.assertDictEqual(response.json(), {"foo": "bar"})
   
       @unittest.expectedFailure
       def test_the_test_server_setup_for_failed_test_cases(self):
           self.server.expect_oneshot_request("/foobar").respond_with_json({"foo": "bar"})
           requests.get(self.server.url_for("/UNEXPECTED"))
   
           # We need this call, otherwise the @unittest.expectedFailure won't have any effect
           # on the exception coming from the tearDown() after this test case
           self._exit_stack.close()

   The actual test case code is very minimal, only server.expect_...(...).respond_...(...) and requests.request(...) calls, and if I expect one-shot requests, and they don't happen, the test case will still fail without needing to assert the server logs len. The test case doesn't hang.