Addressing Security Regression Through Unit Testing – Part 1
EHLO again! I had the pleasure of speaking at QCon NYC last week and I must say it was a pretty damn good conference. Unlike most of the conferences I’ve spoken at, this one was a developer conference. For anyone that likes speaking on security-related topics, I can’t recommend speaking at developer conferences strongly enough. It’s great to speak with one another in the security industry about all the problems plaguing the state of security in the world, but nine times out of 10 we are not the ones with boots on the ground responsible for fixing the myriad holes that we find. This responsibility quite often falls on the shoulders of developers, and as such we should see it as our responsibility to work closely with the software development community to equip them with the knowledge required to improve the general security posture of software.
But I digress – the talk that I gave was entitled Addressing Security Regression Through Unit Testing. This post is a write-up of the topic and a run-through of the software that I wrote to support the talk. I hope this content serves to inspire some of you to implement similar methods in your own codebases!!
The slides for the talk can be found on my Slideshare here:
The code for the talk can be found here:
https://github.com/lavalamp-/security-unit-testing
Once the talk recording is up, I’ll post a link as well.
Security Regression
Regression in codebases is a known and largely addressed subject. The problem of regression is, simply put, that there is no guarantee that the integrity of code is maintained as new code is added. In order to address this problem developers commonly rely upon unit tests – they write unit tests that check whether or not small components of code are working as intended. These tests are then run prior to deployment as new functionality is added to ensure that the new functionality has not broken the older, unit tested functionality.
Through my time in penetration testing, I’ve come to the conclusion that regression with respect to security is a similarly large (if not larger) problem. The number of times that I’ve been on an engagement, found a number of issues in software, counseled the software owners through what the problems were and how to properly fix them, verified that the proper fixes were in place, and then came back six months later to find that the problems were back is far more than I would like to admit. Sometimes the vulnerabilities were back in the same place that they had been found previously. Sometimes the same type of vulnerability was present in new functionality. In all cases though, it seemed that there was little lasting improvement to application security posture as a result of the engagement I had conducted. We can have a much longer conversation around the shortcomings of offensive security testing (which I may reserve for another blog post), but regardless of the details it appeared that even when teams were able to properly address vulnerabilities in their codebases/networks/environments there was no guarantee that those fixes would have any bearing on the continued improvement of the affected organization’s security posture.
So what can we do to start addressing this problem? Well that’s the purpose of this talk and blog post. It turns out we can use the same techniques that developers use to address integrity regression to address security regression. Furthermore, by using a technique that so much infrastructure has already been built around, we get additional improvements to the security posture of the affected codebase by leveraging that infrastructure (continuous integration/deployment infrastructure for example). Not only that but we can use introspection to dynamically generate security unit tests that will provide us with guarantees around the security posture of code that hasn’t even been written yet. Put altogether, we can use unit tests to address security regression to a significant extent.
Dynamically Generating Unit Tests
The code for this blog post is written using the Django Web Framework. The techniques discussed here will certainly work with other frameworks (and perhaps even compiled languages), but one core component that we’ll be leveraging here is the presence of an explicit mapping of URL routes to the views that handle requests to those routes. For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
urlpatterns = [ # Admin url(r"^admin/", admin.site.urls), # Posts url(r"^$", views.PostListView.as_view(), name="post-list"), url(r"^my-posts/?$", views.MyPostsListView.as_view(), name="my-posts"), url(r"^new-post/?$", views.CreatePostView.as_view(), name="new-post"), url(r"^view-post/(?P<pk>[-\w]+)/?", views.PostDetailView.as_view(), name="view-post"), url(r"^edit-post/(?P<pk>[-\w]+)/?", views.EditPostView.as_view(), name="edit-post"), url(r"^delete-post/(?P<pk>[-\w]+)/?", views.DeletePostView.as_view(), name="delete-post"), url(r"^post-successful/(?P<pk>[-\w]+)/?", views.SuccessfulPostDetailView.as_view(), name="post-successful"), url(r"^get-posts-by-title/?$", views.GetPostsByTitleView.as_view(), name="get-post-image"), # Authentication url(r"^login/?$", auth_views.login, {"template_name": "pages/login.html"}, name="login"), url(r"^logout/?$", auth_views.logout, {"template_name": "pages/logout.html"}, name="logout"), url(r"^register/?$", views.CreateUserView.as_view(), name="register"), url(r"^register-success/?$", views.CreateUserSuccessView.as_view(), name="register-success"), # Error Handling url(r"^error-details/?$", views.ErrorDetailsView.as_view(), name="error-info"), # Redirection url(r"^redirect/?$", views.RedirectView.as_view(), name="redirect"), ] |
The reason that this approach requires explicit mapping is that we will use introspection to look into these URL routes and use them to dynamically generate unit tests for all of the views registered in the application.
While we can use introspection to enumerate all of these views, one thing I did not particularly want to do dynamically was figure out how to invoke all of the different HTTP verb functionality (e.g. GET, POST, PUT, DELETE, etc) for all of the registered views with valid HTTP requests. For instance, sending a POST request to the CreatePostView requires title, description, and image parameters. Dynamically figuring out how to create a valid request is certainly possible but would take a significant amount of effort. And so, the approach that I’m using requires a small amount of additional code written for every view in the application in the form of a Requestor class. The base Requestor class is shown below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 |
class BaseRequestor(object): """ This is a base class for all requestor classes used by the Street Art project. """ # Class Members requires_auth = False supported_verbs = [] # Instantiation # Static Methods # Class Methods # Public Methods def get_delete_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP DELETE requests to the view. :param user: A string depicting the user to get DELETE data for. :return: A dictionary containing data to submit in HTTP DELETE requests to the view. """ return None def get_get_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP GET requests to the view. :param user: A string depicting the user to get POST data for. :return: A dictionary containing data to submit in HTTP GET requests to the view. """ return None def get_patch_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP PATCH requests to the view. :param user: A string depicting the user to get PATCH data for. :return: A dictionary containing data to submit in HTTP PATCH requests to the view. """ return None def get_post_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP POST requests to the view. :param user: A string depicting the user to get POST data for. :return: A dictionary containing data to submit in HTTP POST requests to the view. """ return None def get_put_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP PUT requests to the view. :param user: A string depicting the user to get PUT data for. :return: A dictionary containing data to submit in HTTP PUT requests to the view. """ return None def get_trace_data(self, user="user_1"): """ Get a dictionary containing data to submit in HTTP TRACE requests to the view. :param user: A string depicting the user to get TRACE data for. :return: A dictionary containing data to submit in HTTP TRACE requests to the view. """ return None def get_url_path(self, user="user_1"): """ Get the URL path to request through the methods found in this class. :param user: A string depicting the user that the requested URL should be generated off of. :return: A string depicting the URL path to request. """ return None def send_delete(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a DELETE request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.delete. :param kwargs: Keyword arguments for client.delete. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.delete( self.get_url_path(user=user_string), data=self.get_delete_data(user=user_string), *args, **kwargs ) def send_get(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a GET request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.get. :param kwargs: Keyword arguments for client.get. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.get( self.get_url_path(user=user_string), data=self.get_get_data(user=user_string), *args, **kwargs ) def send_head(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a HEAD request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.head. :param kwargs: Keyword arguments for client.head. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.head( self.get_url_path(user=user_string), *args, **kwargs ) def send_options(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send an OPTIONS request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.options. :param kwargs: Keyword arguments for client.options. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.options( self.get_url_path(user=user_string), *args, **kwargs ) def send_patch(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a PATCH request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.patch. :param kwargs: Keyword arguments for client.patch. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.patch( self.get_url_path(user=user_string), data=self.get_patch_data(user=user_string), *args, **kwargs ) def send_post(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a POST request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.post. :param kwargs: Keyword arguments for client.post. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.post( self.get_url_path(user=user_string), data=self.get_post_data(user=user_string), *args, **kwargs ) def send_put(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a PUT request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.put. :param kwargs: Keyword arguments for client.put. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.put( self.get_url_path(user=user_string), data=self.get_put_data(user=user_string), *args, **kwargs ) def send_request_by_verb(self, verb, *args, **kwargs): """ Send a request to the configured view based on the given verb. :param verb: The verb to send the request as. :param args: Positional arguments for the send method. :param kwargs: Keyword arguments for the send method. :return: The HTTP response. """ verb = verb.lower() if verb == "get": return self.send_get(*args, **kwargs) elif verb == "post": return self.send_post(*args, **kwargs) elif verb == "options": return self.send_options(*args, **kwargs) elif verb == "delete": return self.send_delete(*args, **kwargs) elif verb == "put": return self.send_put(*args, **kwargs) elif verb == "head": return self.send_head(*args, **kwargs) elif verb == "patch": return self.send_patch(*args, **kwargs) elif verb == "trace": return self.send_trace(*args, **kwargs) else: raise ValueError( "Unsure of how to handle HTTP verb of %s." % (verb.upper(),) ) def send_trace(self, user_string="user_1", do_auth=True, enforce_csrf_checks=False, *args, **kwargs): """ Send a TRACE request to the configured URL endpoint on behalf of the given user. :param user_string: The user to send the request as. :param do_auth: Whether or not to log the user in if the view requires authentication. :param enforce_csrf_checks: Whether or not to enforce CSRF checks in the HTTP client. :param args: Positional arguments for client.trace. :param kwargs: Keyword arguments for client.trace. :return: The HTTP response. """ client = Client(enforce_csrf_checks=enforce_csrf_checks) if self.requires_auth and do_auth: user = SaFaker.get_user(user_string) client.force_login(user) return client.trace( self.get_url_path(user=user_string), data=self.get_trace_data(user=user_string), *args, **kwargs ) # Protected Methods # Private Methods # Properties # Representation and Comparison def __repr__(self): return "<%s - %s>" % (self.__class__.__name__, self.get_url_path()) |
The Requestor class contains all of the functionality required for our unit tests to invoke all of the functionality for a view in the application. The class contains methods for sending requests for each of the supported HTTP verbs, as well as methods for retrieving the data that should be supplied alongside a given HTTP verb request in order to invoke the functionality successfully. The class also contains a list of the HTTP verbs that the view supports as well as whether or not the view requires authentication. Creating a Requestor for a particular view is quite simple. For example, the Requestor class for the CreatePostView view is shown below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
class CreatePostViewRequestor(BaseRequestor): """ This is a requestor class for sending requests to the CreatePostView view. """ supported_verbs = ["HEAD", "OPTIONS", "GET", "POST", "PUT"] def get_post_data(self, user="user_1"): return SaFaker.get_create_post_kwargs() def get_put_data(self, user="user_1"): return SaFaker.get_create_post_kwargs() def get_url_path(self, user="user_1"): return "/new-post/" |
So now that we have the ability to enumerate all of the views found within the application, and we have the Requestor classes for invoking the functionality for all of these views, the last thing we need is to establish a mapping from the views to the requestors written for them. To accomplish this I introduce the notion of the Registry, which contains a dictionary mapping views to the Requestor classes written to invoke the relevant view functionality:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 |
@Singleton class TestRequestorRegistry(object): """ This is a class that maintains mappings from views to test cases that are configured to send HTTP requests to the view in question. """ # Class Members # Instantiation def __init__(self): self._registry = {} # Static Methods # Class Methods # Public Methods def add_mapping(self, requestor_path=None, requested_view=None): """ Add a mapping from the view to the given requestor class specified by requestor_path. :param requestor_path: The path to the requestor class configured for the given view. :param requested_view: The view that the requestor is meant to send requests for. :return: None """ try: requestor_class = self.__import_class(requestor_path) except (ImportError, AttributeError) as e: raise RequestorNotFoundException( "Unable to load requestor at %s: %s." % (requestor_path, e.message) ) if not issubclass(requestor_class, BaseRequestor): raise InvalidRequestorException( "Class of %s is not a valid requestor class." % (requestor_class.__name__,) ) self._registry[requested_view] = requestor_class def does_view_have_mapping(self, view): """ Check to see if a mapping exists between the given view and a requestor class. :param view: The view to check a mapping for. :return: Whether or not a mapping exists for the given view. """ return view in self.registry def get_requestor_for_view(self, view): """ Get the requestor configured to send requests to the given view. :param view: The view to retrieve the requestor for. :return: The requestor configured to send requests to the given view. """ return self.registry[view] def print_mappings(self): """ Print all of the mappings currently stored within the registry. :return: None """ for k, v in self.registry.iteritems(): print("%s --> %s" % (k, v)) # Protected Methods # Private Methods def __import_class(self, class_path): """ Import the class at the given class path and return it. :param class_path: The class path to the class to load. :return: The loaded class. """ components = class_path.split(".") mod = __import__(components[0]) for component in components[1:]: mod = getattr(mod, component) return mod # Properties @property def registry(self): """ Get the registry mapping functions and classes to the test classes that are configured to submit HTTP requests to them. :return: the registry mapping functions and classes to the test classes that are configured to submit HTTP requests to them. """ return self._registry |
In order to establish the mapping from views to their related Requestor classes, I use the following decorator:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
def requested_by(requestor_path): """ This is a decorator for views that maps a requestor class to the view that it is configured to submit requests to. :param requestor_path: A string depicting the local file path to the requestor class for the given view. :return: A function that maps the view class to the requestor path and returns the called function or class. """ def decorator(to_wrap): registry = TestRequestorRegistry.instance() registry.add_mapping(requestor_path=requestor_path, requested_view=to_wrap) return to_wrap return decorator |
This decorator can then be used to decorate the view classes, and the only argument to the decorator is the import path of the Requestor related to the view. For instance, the CreatePostView class is decorated as follows:
|
1 2 3 4 5 |
@requested_by("streetart.tests.requestors.pages.CreatePostViewRequestor") class CreatePostView(BaseFormView): """ This is a view for creating new street art posts. """ |
With this approach, now every view will automatically be mapped to its related Requestor as soon as it is imported. To demonstrate this, run the following from the sectesting directory:
python manage.py shell -c "from sectesting import urls; from streetart.tests import TestRequestorRegistry; registry = TestRequestorRegistry.instance(); registry.print_mappings()"
The result of running this command is shown below:
With the mapping in place, we now have the ability to:
- Enumerate all views in the application
- Retrieve a requestor for every view that has the ability to invoke all of the view’s functionality
Great! With this framework we can now dynamically generate unit tests for all of the HTTP verbs for all of the views in the application. This dynamic generation is handled by the StreetArtTestRunner class found in tests/runner.py, the contents of which are shown below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 |
class StreetArtTestRunner(DiscoverRunner): """ This is a custom discover runner for populating unit tests for the Street Art project. """ # Class Members ALL_HTTP_VERBS = [ "GET", "HEAD", "POST", "PUT", "DELETE", "OPTIONS", "TRACE", "PATCH", ] CSRF_VERBS = [ "POST", "PUT", "DELETE", "PATCH", ] # Instantiation def __init__(self, *args, **kwargs): self._url_patterns = None super(StreetArtTestRunner, self).__init__(*args, **kwargs) # Static Methods # Class Methods # Public Methods def build_suite(self, test_labels=None, extra_tests=None, **kwargs): """ Build the test suite to run for this discover runner. :param test_labels: A list of strings describing the tests to be run. :param extra_tests: A list of extra TestCase instances to add to the suite that is executed by the test runner. :param kwargs: Additional keyword arguments. :return: The test suite. """ extra_tests = extra_tests if extra_tests is not None else [] extra_tests.extend(self.__get_generated_test_cases()) return super(StreetArtTestRunner, self).build_suite( test_labels=test_labels, extra_tests=extra_tests, **kwargs ) def run_suite(self, suite, **kwargs): """ Override the run_suite functionality to populate the database. :param suite: The suite to run. :param kwargs: Keyword arguments. :return: The rest suite result. """ self.__populate_database() return super(StreetArtTestRunner, self).run_suite(suite, **kwargs) # Protected Methods # Private Methods def __get_authentication_enforcement_tests(self): """ Get a list of test cases that will test whether or not authentication is correctly enforced on a given view. :return: A list of test cases that will test whether or not authentication is correctly enforced on a given view. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) if not requestor.requires_auth: continue for supported_verb in requestor.supported_verbs: class AnonTestCase(AuthenticationEnforcementTestCase): pass to_return.append(AnonTestCase(view=view, verb=supported_verb)) return to_return def __get_csrf_enforcement_tests(self): """ Get a list of test cases that check to make sure that CSRF checks are being correctly enforced. :return: A list of test cases that check to make sure that CSRF checks are being correctly enforced. """ to_return = [] csrf_verbs = [x.lower() for x in self.CSRF_VERBS] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) supported_verbs = [x.lower() for x in requestor.supported_verbs] supported_csrf_verbs = filter(lambda x: x in csrf_verbs, supported_verbs) for supported_csrf_verb in supported_csrf_verbs: class AnonTestCase1(CsrfEnforcementTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_csrf_verb)) return to_return def __get_dos_class_tests(self): """ Get a list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. :return: A list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) for supported_verb in requestor.supported_verbs: class AnonTestCase1(RegularViewRequestIsSuccessfulTestCase): pass class AnonTestCase2(AdminViewRequestIsSuccessfulTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_verb)) to_return.append(AnonTestCase2(view=view, verb=supported_verb)) return to_return def __get_generated_test_cases(self): """ Get a list containing the automatically generated test cases to add to the test suite this runner is configured to run. :return: A list containing the automatically generated test cases to add to the test suite this runner is configured to run. """ # Ensure that all views are loaded import sectesting.urls to_return = [] if settings.TEST_FOR_REQUESTOR_CLASSES: to_return.extend(self.__get_requestor_class_tests()) if settings.TEST_FOR_DENIAL_OF_SERVICE: to_return.extend(self.__get_dos_class_tests()) if settings.TEST_FOR_UNKNOWN_METHODS: to_return.extend(self.__get_unknown_methods_tests()) if settings.TEST_FOR_AUTHENTICATION_ENFORCEMENT: to_return.extend(self.__get_authentication_enforcement_tests()) if settings.TEST_FOR_RESPONSE_HEADERS: to_return.extend(self.__get_response_header_tests()) if settings.TEST_FOR_OPTIONS_ACCURACY: to_return.extend(self.__get_options_accuracy_tests()) if settings.TEST_FOR_CSRF_ENFORCEMENT: to_return.extend(self.__get_csrf_enforcement_tests()) return to_return def __get_options_accuracy_tests(self): """ Get a list of test cases that will test to ensure that no verbs other than those specified in OPTIONS responses are present on all views. :return: A list of test cases that will test to ensure that no verbs other than those specified in OPTIONS responses are present on all views. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) supported_verbs = [x.lower() for x in requestor.supported_verbs] for http_verb in self.ALL_HTTP_VERBS: if http_verb.lower() not in supported_verbs: class AnonTestCase1(RegularVerbNotSupportedTestCase): pass class AnonTestCase2(AdminVerbNotSupportedTestCase): pass to_return.append(AnonTestCase1(view=view, verb=http_verb)) to_return.append(AnonTestCase2(view=view, verb=http_verb)) return to_return def __get_response_header_tests(self): """ Get a list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. :return: A list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) for k, v in settings.EXPECTED_RESPONSE_HEADERS["included"].iteritems(): for supported_verb in requestor.supported_verbs: class AnonTestCase1(HeaderKeyExistsTestCase): pass class AnonTestCase2(HeaderValueAccurateTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_verb, header_key=k)) to_return.append(AnonTestCase2(view=view, verb=supported_verb, header_key=k, header_value=v)) for excluded_header in settings.EXPECTED_RESPONSE_HEADERS["excluded"]: for supported_verb in requestor.supported_verbs: class AnonTestCase3(HeaderKeyNotExistsTestCase): pass to_return.append(AnonTestCase3(view=view, verb=supported_verb, header_key=excluded_header)) return to_return def __get_requestor_class_tests(self): """ Get a list of test cases that will test the views associated with the Street Art project to ensure that the view has a requestor class associated with it. :return: A list of test cases that will test the views associated with the Street Art project to ensure that the view has a requestor class associated with it. """ to_return = [] for _, _, callback in self.url_patterns: class AnonTestCase(ViewHasRequestorTestCase): pass to_return.append(AnonTestCase(self.__get_view_from_callback(callback))) return to_return def __get_unknown_methods_tests(self): """ Get a list of test cases that will test whether or not views return the expected HTTP verbs through OPTIONS requests. :return: A list of test cases that will test whether or not views return the expected HTTP verbs through OPTIONS requests. """ to_return = [] for _, _, callback in self.url_patterns: view = self.__get_view_from_callback(callback) class AnonTestCase1(RegularUnknownMethodsTestCase): pass class AnonTestCase2(AdminUnknownMethodsTestCase): pass to_return.append(AnonTestCase1(view)) to_return.append(AnonTestCase2(view)) return to_return def __get_view_from_callback(self, callback): """ Get the view associated with the given callback. :param callback: The callback to get the view from. :return: The view associated with the given callback. """ if hasattr(callback, "view_class"): return callback.view_class else: return callback def __get_view_and_requestor_from_callback(self, callback): """ Get a tuple containing (1) the view and (2) the requestor associated with the given URL pattern callback. :param callback: The URL pattern callback to process. :return: A tuple containing (1) the view and (2) the requestor associated with the given URL pattern callback. """ registry = TestRequestorRegistry.instance() view = self.__get_view_from_callback(callback) requestor = registry.get_requestor_for_view(view) return view, requestor def __populate_database(self): """ Populate the database with dummy database models. :return: None """ print("Now populating test database...") SaFaker.create_users() # Properties @property def url_patterns(self): """ Get a list of tuples containing (1) the URL pattern regex, (2) the pattern name, and (3) the callback function for the views that this runner should generate automated tests for. :return: a list of tuples containing (1) the URL pattern regex, (2) the pattern name, and (3) the callback function for the views that this runner should generate automated tests for. """ if self._url_patterns is None: self._url_patterns = UrlPatternHelper.get_all_streetart_views( include_admin_views=settings.INCLUDE_ADMIN_VIEWS_IN_TESTS, include_auth_views=settings.INCLUDE_AUTH_VIEWS_IN_TESTS, include_generic_views=settings.INCLUDE_GENERIC_VIEWS_IN_TESTS, include_contenttype_views=settings.INCLUDE_CONTENTTYPE_VIEWS_IN_TESTS, ) return self._url_patterns # Representation and Comparison def __repr__(self): return "<%s>" % (self.__class__.__name__,) |
To demonstrate how we dynamically generate tests, let’s take the RegularViewRequestIsSuccessfulTestCase as an example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
class RegularViewRequestIsSuccessfulTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a view returns a HTTP response indicating that the request was successful for a regular user. """ def runTest(self): """ Tests that the given view returns a successful HTTP response for the given verb. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="user_1") self._assert_response_successful( response, "%s did not return a successful response for %s verb (%s status, regular user)" % (self.view, self.verb, response.status_code) ) |
This test case inherits from the BaseViewVerbTestCase class, which takes a view and a verb argument in the constructor. In order to generate instances of this test case for all of the verbs and views in the application, let’s take a look at the __get_dos_class_tests method in the StreetArtTestRunner class:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
def __get_dos_class_tests(self): """ Get a list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. :return: A list of test cases that will test to ensure that all of the configured URL routes return successful HTTP status codes. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) for supported_verb in requestor.supported_verbs: class AnonTestCase1(RegularViewRequestIsSuccessfulTestCase): pass class AnonTestCase2(AdminViewRequestIsSuccessfulTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_verb)) to_return.append(AnonTestCase2(view=view, verb=supported_verb)) return to_return |
As shown above, we:
- Iterate over all of the views found in the application (line 9)
- Get the view and the
Requestorassociated with the view (line 10) - Iterate over all of the supported verbs listed in the
Requestor(line 11) - For each of the supported verbs, we create an anonymous subclass of
RegularViewRequestIsSuccessfulTestCase(line 13) - We add an instance of the anonymous subclass instantiated with the verb and view that we are iterating over (line 19)
This may look a bit odd – why are we creating an anonymous subclass? Well the way that the Python unit testing framework works we cannot have two instances of the same test case class in a test suite (it will only run one of them). As such, we create unique classes for each of the test cases that we need to run. It’s a bit of a quirk, but nothing we can’t handle!
And with that, we have the ability to dynamically generate unit tests for all of the functionality in our application. Let’s now take a look at how we can make good use of this capability.
Testing For Adherence To The Requestor Architecture
Since we are relying on our developers to add a little bit of extra functionality for all of the views that they author, a logical first step to test is that all of the code within our codebase does, in fact, follow our architecture. This is tested by the ViewHasRequestorTestCase:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
class ViewHasRequestorTestCase(BaseViewTestCase): """ This is a test case for testing whether or not a view has a corresponding requestor mapped to it. """ def runTest(self): """ Tests that the given view has a requestor mapped to it. :return: None """ registry = TestRequestorRegistry.instance() self.assertTrue( registry.does_view_have_mapping(self.view), "No requestor found for view %s." % self.view, ) |
This test is rather simple – it takes the view that the test case was instantiated with and checks the Registry to make sure that a requestor for the view exists. To see the results of this test, check out the v0.1 tag:
git checkout tags/v0.1
Once checked out, modify the settings.py file so that only the Requestor check tests are enabled:
|
1 2 3 4 5 6 7 |
TEST_FOR_REQUESTOR_CLASSES = True TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False |
We can now run the following to verify that all of the views have a Requestor mapped to them:
python manage.py test
The result of running this command is shown below:
That’s great and all that all of our unit tests are passing, but let’s make sure that they’re testing what we intend them to. To do so, we remove the requested_by decorator from the MyPostsListView view as shown below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
#@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor") class MyPostsListView(BaseListView): """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get_queryset(self): """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self.request.user.posts.all() |
After commenting out requested_by as shown in line 1 above, we run the tests again:
Sure enough, as expected one of our unit tests fails indicating that the MyPostsListView does not have a Requestor mapped to it. Great! Let’s go ahead and uncomment the requested_by decorator and continue.
Testing For Denial Of Service
Now that we know all of our views have the necessary Requestor class mapped to them, let’s test to make sure that all of the functionality within our application is working properly (ie: returns an HTTP status code indicating a successful request). Testing this is handled by the RegularViewRequestIsSuccessfulTestCase and AdminViewRequestIsSuccessfulTestCase test cases:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
class RegularViewRequestIsSuccessfulTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a view returns a HTTP response indicating that the request was successful for a regular user. """ def runTest(self): """ Tests that the given view returns a successful HTTP response for the given verb. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="user_1") self._assert_response_successful( response, "%s did not return a successful response for %s verb (%s status, regular user)" % (self.view, self.verb, response.status_code) ) class AdminViewRequestIsSuccessfulTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a view returns a HTTP response indicating that the request was successful for an admin user. """ def runTest(self): """ Tests that the given view returns a successful HTTP response for the given verb. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="admin_1") self._assert_response_successful( response, "%s did not return a successful response for %s verb (%s status, admin user)" % (self.view, self.verb, response.status_code) ) |
These test cases simply send a request to the related view using the configured verb and check to see that the HTTP status code of the response indicates that the request was successful. To run these tests, configure the application to only run the denial of service test cases in settings.py:
|
1 2 3 4 5 6 7 |
TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = True TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False |
We now run the unit tests:
python manage.py test
The result of running these tests is shown below:
As we did before, let’s modify our code to introduce a failing test case just to make sure that our unit tests are working as intended. To do this, raise a PermissionDenied exception in the EditPostView view’s get method:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
@requested_by("streetart.tests.requestors.pages.EditPostViewRequestor") class EditPostView(BaseUpdateView): """ This is the page for editing the contents of a street art post object. """ template_name = "pages/edit_streetart_post.html" form_class = EditStreetArtPostForm model = StreetArtPost def get(self, request, *args, **kwargs): """ Handle a GET request and ensure the requesting user owns the given post. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: super.get. """ raise PermissionDenied # if request.user != self.get_object().user and not request.user.is_superuser: # raise PermissionDenied # return super(EditPostView, self).get(request, *args, **kwargs) def post(self, request, *args, **kwargs): """ Handle a POST request and ensure the requesting user owns the given post. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: super.post. """ if request.user != self.get_object().user and not request.user.is_superuser: raise PermissionDenied return super(EditPostView, self).post(request, *args, **kwargs) |
We then run the tests again:
As expected, we see two failed unit tests indicating that the EditPostView view is returning an HTTP status code indicating a request error. Great! As before, let’s modify the EditPostView back to what it was beforehand and continue.
Testing For Unknown HTTP Verbs
We’ve now got some guarantees that our code is following the Requestor framework and that all of the tested functionality is indicating success, but what if our Requestor classes aren’t testing all of the HTTP verbs supported by our views? It is incredibly common for frameworks (especially ones like Django and Rails) to have all sorts of crazy functionality under the hood, and I’ve abused functionality that developers didn’t know about to nefarious ends on more than one occasion. To ensure that we are testing all of the functionality within our application, let’s make sure that the supported verbs reported by our views match the verbs that our Requestor classes are configured to invoke. Testing this is handled by the RegularUnknownMethodsTestCase:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
class RegularUnknownMethodsTestCase(BaseViewTestCase): """ This is a test case for testing whether or not a view returns the expected HTTP verbs through an OPTIONS request from a regular user. """ def runTest(self): """ Tests that the HTTP verbs returned by an OPTIONS request match the expected values. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_options(user_string="user_1") allowed_verbs = response._headers.get("allow", None) if not allowed_verbs: raise ValueError("No allow header returned by view %s." % self.view) allowed_verbs = [x.strip().lower() for x in allowed_verbs[1].split(",")] supported_verbs = [x.lower() for x in requestor.supported_verbs] self.assertTrue( all([x.lower() in supported_verbs for x in allowed_verbs]), "Unexpected verbs found for view %s. Expected %s, got %s." % (self.view, [x.upper() for x in supported_verbs], [x.upper() for x in allowed_verbs]) ) |
As shown above, we issue an HTTP OPTIONS request to the view, parse the contents of the Allow HTTP response header (which contains a comma-separated list of the verbs supported by the endpoint), and then check those verbs against the verbs listed in the related Requestor. To run these tests, configure settings.py to only enable the unknown methods test cases:
|
1 2 3 4 5 6 7 |
TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = True TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False |
We then run the tests:
python manage.py test
The result of running these tests is shown below:
As shown above, we have multiple Requestor classes that are not testing all of the HTTP verbs associated with their views! Funnily enough – this output was exactly what I saw when I was first authoring the test codebase for this talk. I had no idea that views that subclassed DeleteView supported both the GET and DELETE HTTP verbs!
Taking a look at the DeletePostViewRequestor class, we see that the class only tests for the POST verb:
|
1 2 3 4 5 6 7 8 9 10 11 |
class DeletePostViewRequestor(BaseRequestor): """ This is a requestor class for sending requests to the DeletePostView view. """ supported_verbs = ["POST"] requires_auth = True def get_url_path(self, user="user_1"): post = SaFaker.get_post_for_user(user) return "/delete-post/%s/" % (post.uuid,) |
Let’s go ahead and check out the next tag:
git checkout tags/v0.2
And taking a look at the DeletePostViewRequestor class again, we see that it is now updated to support the GET and DELETE verbs:
|
1 2 3 4 5 6 7 8 9 10 11 |
class DeletePostViewRequestor(BaseRequestor): """ This is a requestor class for sending requests to the DeletePostView view. """ supported_verbs = ["GET", "POST", "DELETE"] requires_auth = True def get_url_path(self, user="user_1"): post = SaFaker.get_post_for_user(user) return "/delete-post/%s/" % (post.uuid,) |
Let’s run the tests again and check to make sure we are now testing all of the HTTP verbs supported by our application’s views:
Boom! We now know that all of our Requestor classes are properly configured to test all of the HTTP verbs associated with all of our views!
Testing For Authentication Enforcement
Darn near every application contains functionality that is only available to users that are authenticated to the application. Wouldn’t it be great to ensure that all of our post-auth functionality is properly enforcing authentication checks? Well this case is tested for by the AuthenticationEnforcementTestCase. To see this test case, first checkout the v0.3 tag:
git checkout tags/v0.3
The contents of the AuthenticationEnforcementTestCase are shown below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
class AuthenticationEnforcementTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not authentication is properly enforced on a view. """ def runTest(self): """ Tests that the given view returns the expected HTTP response value when an unauthenticated request is submitted to it. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, do_auth=False) self._assert_response_redirect( response, "Response from unauthenticated %s request to view %s was %s. Expected %s." % (self.verb, self.view, response.status_code, [301, 302]) ) |
When generating instances of this unit test, we generate them only for endpoints that require authentication as shown in the __get_authentication_enforcement_tests method in the StreetArtTestRunner class:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
def __get_authentication_enforcement_tests(self): """ Get a list of test cases that will test whether or not authentication is correctly enforced on a given view. :return: A list of test cases that will test whether or not authentication is correctly enforced on a given view. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) if not requestor.requires_auth: continue for supported_verb in requestor.supported_verbs: class AnonTestCase(AuthenticationEnforcementTestCase): pass to_return.append(AnonTestCase(view=view, verb=supported_verb)) return to_return |
To run these tests we configure settings.py to enable only the authentication check test cases:
|
1 2 3 4 5 6 7 |
TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = True TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False |
We then run the unit tests:
python manage.py test
The results of running these tests are shown below:
We see that an error is being thrown indicating that an attribute is being accessed on a user object when that user is not authenticated. The offending code can be found in the MyPostsListView view on line 16:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor") class MyPostsListView(BaseListView): """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get_queryset(self): """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self.request.user.posts.all() |
Let’s check out the next tag to see what has changed in the MyPostsListView:
git checkout tags/v0.4
The new contents of MyPostsListView are shown below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor") class MyPostsListView(BaseListView): """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get(self, request, *args, **kwargs): """ Handle the processing of an HTTP GET request to this endpoint to ensure that the requesting user has sufficient permissions. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: super.get. """ if not request.user.is_authenticated: raise PermissionDenied return super(MyPostsListView, self).get(request, *args, **kwargs) def get_queryset(self): """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self.request.user.posts.all() |
Running the tests again, we see that all of our authentication checks are now being properly enforced:
Great! We now know that all of our post-auth endpoints are properly enforcing authentication checks.
Testing For HTTP Response Headers
For the uninitiated, there are a number of HTTP response headers that can greatly improve the security posture of the browsers used by your application’s clients. Even better, many of these headers require little-to-no configuration in your application to just work! To test our application for the presence of these headers, lets first check out the next tag:
git checkout tags/v0.5
Once checked out, we can find the test functionality for checking response headers in the HeaderKeyExistsTestCase and the HeaderValueAccurateTestCase test cases found in tests/cases/headers.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
class HeaderKeyExistsTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a header key is contained within all of the HTTP responses returned by the Street Art project. """ def __init__(self, header_key=None, *args, **kwargs): self.header_key = header_key super(HeaderKeyExistsTestCase, self).__init__(*args, **kwargs) def runTest(self): """ Tests that the HTTP response received from the view contains a header corresponding to self.header_key. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="user_1") self._assert_response_has_header_key( response=response, header_key=self.header_key, message="Response from view %s with verb %s did not contain header key of %s. Keys were %s." % (self.view, self.verb, self.header_key, response._headers.keys()) ) class HeaderValueAccurateTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a response header has the expected value. """ def __init__(self, header_key=None, header_value=None, *args, **kwargs): self.header_key = header_key self.header_value = header_value super(HeaderValueAccurateTestCase, self).__init__(*args, **kwargs) def runTest(self): """ Tests that the HTTP response received from the view contains a header key and value corresponding to self.header_key and self.header_value. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="user_1") self._assert_response_has_header_value( response=response, header_key=self.header_key, header_value=self.header_value, message="Response from view %s with verb %s did not contain expected header value of %s: %s. " "Headers were %s." % (self.view, self.verb, self.header_key, self.header_value, response._headers) ) |
These two test cases test (1) that a header key exists in every one of the HTTP responses and (2) that the value associated with each header key has the expected content. To generate these unit tests, we first have a list of the headers that we want to see in the application configured in settings.py:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
EXPECTED_RESPONSE_HEADERS = { "included": { "X-Frame-Options": "deny", "X-XSS-Protection": "1; mode=block", "X-Content-Type-Options": "nosniff", "X-Permitted-Cross-Domain-Policies": "none", }, "excluded": [ "X-Supah-Secret", "X-Supah-Dupah-Secret", ] } |
When generating the unit tests, we iterate over the contents of the expected HTTP response headers and the views/verbs in our application as shown in the __get_response_header_tests method in the StreetArtTestRunner class:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
def __get_response_header_tests(self): """ Get a list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. :return: A list of test cases that will test the views associated with the Street Art project to ensure that the expected response headers are found in all responses. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) for k, v in settings.EXPECTED_RESPONSE_HEADERS["included"].iteritems(): for supported_verb in requestor.supported_verbs: class AnonTestCase1(HeaderKeyExistsTestCase): pass class AnonTestCase2(HeaderValueAccurateTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_verb, header_key=k)) to_return.append(AnonTestCase2(view=view, verb=supported_verb, header_key=k, header_value=v)) return to_return |
In order to properly run these tests, configure the middleware in settings.py to exclude the middleware used for populating the headers:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
MIDDLEWARE = [ # 'streetart.middleware.HeaderCleaningMiddleware', 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', # 'streetart.middleware.SecurityHeadersMiddleware', # 'streetart.middleware.BadHeadersMiddleware', ] |
With the relevant middleware commented out, configure settings.py to only test for the presence of HTTP response headers:
|
1 2 3 4 5 6 7 |
TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = True TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = False |
We can now run the tests:
python manage.py test
The results of running these tests are shown below:
As shown above, we have failing and erroring unit tests indicating which views are lacking which HTTP response headers. It turns out that the only security-related response header that we currently have in the application is the X-Frame-Options header! Let’s go ahead and fix this by checking out the next tag:
git checkout tags/v0.6
This tag introduces the SecurityHeadersMiddleware middleware which populates the relevant security headers in all responses that pass through it:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
class SecurityHeadersMiddleware(object): """ This is a middleware class for handling the addition of HTTP security headers to all responses. """ def __init__(self, get_response): self.get_response = get_response def __call__(self, request): response = self.get_response(request) response["X-Frame-Options"] = "deny" response["X-Content-Type-Options"] = "nosniff" response["X-XSS-Protection"] = "1; mode=block" response["X-Permitted-Cross-Domain-Policies"] = "none" return response |
Let’s change the settings.py file to include this middleware class in the middleware used by the application:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
MIDDLEWARE = [ # 'streetart.middleware.HeaderCleaningMiddleware', 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'streetart.middleware.SecurityHeadersMiddleware', # 'streetart.middleware.BadHeadersMiddleware', ] |
With the middleware now enabled, we run the tests again:
Fantastic – we now know that all of the HTTP verbs supported by all of our views are returning the HTTP response security headers that we want!
Testing For OPTIONS Accuracy
Sure we know that all of the functionality in our application is currently working, and that we are testing all of the HTTP verbs reported by OPTIONS responses returned by our views, but what if the OPTIONS responses weren’t being entirely honest? What if there were HTTP verbs that could be invoked but were not included in the Allow headers returned via OPTIONS requests? Well let’s test for it! First, let’s check out the next tag:
git checkout tags/v0.9
Once checked out, we can find the test cases that test for this functionality in the RegularVerbNotSupportedTestCase and the AdminVerbNotSupportedTestCase test cases:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
class RegularVerbNotSupportedTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a given HTTP verb is denied when submitted against a view by a regular user. """ def runTest(self): """ Tests that self.verb does not work against self.view. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="user_1") self._assert_response_not_allowed( response, "HTTP verb %s returned %s status code when it should have been 405 (regular user)." % (self.verb, response.status_code) ) class AdminVerbNotSupportedTestCase(BaseViewVerbTestCase): """ This is a test case for testing whether or not a given HTTP verb is denied when submitted against a view by an admin user. """ def runTest(self): """ Tests that self.verb does not work against self.view. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb(self.verb, user_string="admin_1") self._assert_response_not_allowed( response, "HTTP verb %s returned %s status code when it should have been 405 (admin user)." % (self.verb, response.status_code) ) |
These tests check to make sure that the relevant view and HTTP verb return a 405 HTTP status indicating that the verb is not supported. To generate these tests, we iterate over all the views in the application and create test cases for every HTTP verb that is supposedly not supported as found in the __get_options_accuracy_tests method in the StreetArtTestRunner class:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
def __get_options_accuracy_tests(self): """ Get a list of test cases that will test to ensure that no verbs other than those specified in OPTIONS responses are present on all views. :return: A list of test cases that will test to ensure that no verbs other than those specified in OPTIONS responses are present on all views. """ to_return = [] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) supported_verbs = [x.lower() for x in requestor.supported_verbs] for http_verb in self.ALL_HTTP_VERBS: if http_verb.lower() not in supported_verbs: class AnonTestCase1(RegularVerbNotSupportedTestCase): pass class AnonTestCase2(AdminVerbNotSupportedTestCase): pass to_return.append(AnonTestCase1(view=view, verb=http_verb)) to_return.append(AnonTestCase2(view=view, verb=http_verb)) return to_return |
We then configure settings.py to only enable the unknown verb tests:
|
1 2 3 4 5 6 7 |
TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = True TEST_FOR_CSRF_ENFORCEMENT = False |
We can then run the tests:
python manage.py test
The result of running these tests is shown below:
What’s this?! It looks like the OPTIONS response from one of our views wasn’t being entirely honest. The offending code can be found in the MyPostsListView view:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor") class MyPostsListView(LoginRequiredMixin, BaseListView): """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get_queryset(self): """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self.request.user.posts.all() def options(self, request, *args, **kwargs): """ Override the OPTIONS request handler to hide the presence of the TRACE backdoor. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: The HTTP response. """ to_return = super(MyPostsListView, self).options(request, *args, **kwargs) allowed_verbs = [x.strip() for x in to_return._headers["allow"][1].split(",")] allowed_verbs.remove("TRACE") to_return._headers["allow"] = ("Allow", ", ".join(allowed_verbs)) return to_return def trace(self, request, *args, **kwargs): """ Handle the TRACE backdoor. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: The HTTP response. """ if request.user.is_superuser: return HttpResponse("You found the secret backdoor!") else: return HttpResponseNotAllowed(["GET", "HEAD", "OPTIONS"]) |
It looks like some sneaky devil has put an (albeit completely useless) backdoor into our code! Let’s check out the next tag and verify that the backdoor has been removed:
git checkout tags/v0.10
We take a look at the MyPostsListView view to see that the backdoor has been removed:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
@requested_by("streetart.tests.requestors.pages.MyPostsListViewRequestor") class MyPostsListView(LoginRequiredMixin, BaseListView): """ This is a page for displaying all of the posts associated with the logged-in user. """ template_name = "pages/streetart_post_list.html" model = StreetArtPost paginate_by = 2 def get_queryset(self): """ Get all of the posts that are associated with the requesting user. :return: All of the posts that are associated with the requesting user. """ return self.request.user.posts.all() |
Sure enough, the backdoor has been removed, and when we run the tests now we see that there is no hidden functionality within our application:
Testing For CSRF Protection
Cross-site request forgery (CSRF) is a rather tricky vulnerability to describe, so I’ll let the good folks over at OWASP do the heavy lifting here. Long story short, for any non-idempotent HTTP verbs handled by our application we should have something called a CSRF token required in the request. If this token is either not present or not accurate, the request should be considered invalid and dropped immediately. To test that our application is properly guarded against CSRF, let’s check out the next tag:
git checkout tags/v0.11
The test case that will be checking for CSRF enforcement is entitled CsrfEnforcementTestCase:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
class CsrfEnforcementTestCase(BaseViewVerbTestCase): """ This is a test case for testing that a CSRF token is properly enforced on a view with a given verb via a request that is submitted by a regular user. """ def runTest(self): """ Test to ensure that that the CSRF token is properly enforced on the referenced view. :return: None """ requestor = self._get_requestor_for_view(self.view) response = requestor.send_request_by_verb( self.verb, user_string="user_1", enforce_csrf_checks=True, ) self._assert_response_permission_denied( response, "Response from %s indicated that CSRF protection was not enabled (verb was %s, status %s)." % (self.view, self.verb, response.status_code) ) |
This test case sends a request to the view and tells the Django unit testing framework to enforce CSRF checks. If the HTTP response indicates that the request was successful, the test case fails (as the request does not have a CSRF token)! To generate these unit tests, we iterate over all of the views in our application and create a test case for every non-idempotent HTTP verb as shown in the __get_csrf_enforcement_tests method in the StreetArtTestRunner class:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
def __get_csrf_enforcement_tests(self): """ Get a list of test cases that check to make sure that CSRF checks are being correctly enforced. :return: A list of test cases that check to make sure that CSRF checks are being correctly enforced. """ to_return = [] csrf_verbs = [x.lower() for x in self.CSRF_VERBS] for _, _, callback in self.url_patterns: view, requestor = self.__get_view_and_requestor_from_callback(callback) supported_verbs = [x.lower() for x in requestor.supported_verbs] supported_csrf_verbs = filter(lambda x: x in csrf_verbs, supported_verbs) for supported_csrf_verb in supported_csrf_verbs: class AnonTestCase1(CsrfEnforcementTestCase): pass to_return.append(AnonTestCase1(view=view, verb=supported_csrf_verb)) return to_return |
We then modify settings.py to enable only the CSRF enforcement tests:
|
1 2 3 4 5 6 7 |
TEST_FOR_REQUESTOR_CLASSES = False TEST_FOR_DENIAL_OF_SERVICE = False TEST_FOR_UNKNOWN_METHODS = False TEST_FOR_AUTHENTICATION_ENFORCEMENT = False TEST_FOR_RESPONSE_HEADERS = False TEST_FOR_OPTIONS_ACCURACY = False TEST_FOR_CSRF_ENFORCEMENT = True |
We can then run the unit tests:
python manage.py test
The results of running these unit tests are shown below:
Whoa! It looks like there are a handful of endpoints in our application that are not properly enforcing CSRF protections. Taking a look at the CreatePostView view, we can see exactly where the protections are being disabled:
|
1 2 3 4 5 6 7 8 9 |
@method_decorator(csrf_exempt, name="dispatch") @requested_by("streetart.tests.requestors.pages.CreatePostViewRequestor") class CreatePostView(BaseFormView): """ This is a view for creating new street art posts. """ template_name = "pages/new_streetart_post.html" form_class = NewStreetArtPostForm |
On line 1 above we see that a csrf_exempt method is wrapping the dispatch method in the view. This is effectively disabling CSRF protections for the view. Let’s go ahead and check out the next tag to fix this issue:
git checkout tags/v0.12
Once checked out, we take another look at the CreatePostView and confirm that the decorator is no longer present:
|
1 2 3 4 5 6 7 8 |
@requested_by("streetart.tests.requestors.pages.CreatePostViewRequestor") class CreatePostView(BaseFormView): """ This is a view for creating new street art posts. """ template_name = "pages/new_streetart_post.html" form_class = NewStreetArtPostForm |
We can then run the unit tests again to confirm that CSRF protections are properly enabled in our application:
The Benefits Of Dynamic Generation
At this point, our application now has the following security guarantees:
- All of our views have requestors mapped to them
- All of the verbs supported by our views are working properly (or at least they are returning HTTP status codes indicating success)
- All endpoints that require authentication are properly enforcing authentication checks
- None of our views support any HTTP verbs other than what they report in the OPTIONS response
- CSRF tokens are properly being enforced for non-idempotent requests
- HTTP response security headers are present in all responses from all of the verbs from all of our views
That’s great and all, but perhaps you’re wondering why we should use dynamic generation to test for any of this. We could just write traditional unit tests to check for all of this functionality, right?
That’s right! We definitely could. However, this would require developers to write those unit tests for all of the new views that they author. As we discussed at the beginning of this post, just because we have passing unit tests for the functionality that we’ve written already doesn’t mean anything about code that we have yet to write/bring into our codebase. This is where dynamic generation truly shines – all of our dynamically-generated unit tests will automatically be applied to all of the views that are added to our application (even the ones that haven’t been written yet). As such, the same guarantees that we have about our application right now will hold true for all of the functionality that we implement moving forward. Let’s take a look at what I mean here by checking out the next tag:
git checkout tags/v0.15
In this version, some devious developer has introduced a new view into our application that has quite a few problems with it. This view, entitled NewCreatePostView, is shown below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
@method_decorator(csrf_exempt, name="dispatch") class NewCreatePostView(BaseFormView): """ This is a view for demonstrating all of the bad things that dynamic test generation can protect against. """ template_name = "pages/new_streetart_post.html" form_class = NewStreetArtPostForm def options(self, request, *args, **kwargs): """ Override the OPTIONS request handler to hide the presence of the TRACE backdoor. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: The HTTP response. """ to_return = super(NewCreatePostView, self).options(request, *args, **kwargs) allowed_verbs = [x.strip() for x in to_return._headers["allow"][1].split(",")] allowed_verbs.remove("TRACE") to_return._headers["allow"] = ("Allow", ", ".join(allowed_verbs)) return to_return def trace(self, request, *args, **kwargs): """ Handle the TRACE backdoor. :param request: The request to process. :param args: Positional arguments. :param kwargs: Keyword arguments. :return: The HTTP response. """ if request.user.is_superuser: return HttpResponse("You found the secret backdoor!") else: return HttpResponseNotAllowed(["GET", "HEAD", "OPTIONS"]) |
With this view added, lets configure settings.py to enable all of the test case types that we’ve written:
|
1 2 3 4 5 6 7 |
TEST_FOR_REQUESTOR_CLASSES = True TEST_FOR_DENIAL_OF_SERVICE = True TEST_FOR_UNKNOWN_METHODS = True TEST_FOR_AUTHENTICATION_ENFORCEMENT = True TEST_FOR_RESPONSE_HEADERS = True TEST_FOR_OPTIONS_ACCURACY = True TEST_FOR_CSRF_ENFORCEMENT = True |
Let’s then run unit tests and see what happens:
We immediately see an error indicating that a Requestor class does not exist for this troublesome view! Ok, so we go and talk to the developer and get them to add a Requestor and they commit to the next tag:
git checkout tags/v0.16
We check the contents of the NewCreatePostView and confirm that a Requestor is now configured for the view:
|
1 2 3 4 5 6 7 8 9 10 |
@method_decorator(csrf_exempt, name="dispatch") @requested_by("streetart.tests.requestors.pages.NewCreatePostViewRequestor") class NewCreatePostView(BaseFormView): """ This is a view for demonstrating all of the bad things that dynamic test generation can protect against. """ template_name = "pages/new_streetart_post.html" form_class = NewStreetArtPostForm |
Let’s run the tests again and see what happens:
Boom – we have failing test cases that indicate:
- This view does not have CSRF protections enabled
- This view is not properly enforcing authentication checks
- This view has hidden functionality in the TRACE verb when an admin user requests it
Without writing a single additional unit test, we have the same security guarantees around this new view that we’ve had for all of the other views in our application. This is powerful.
In addition to providing protections around code that hasn’t even been authored yet, we have a significant return on investment for the amount of effort we put into writing unit tests. In exchange for writing the plumbing for the test generation as well as writing the test case itself, we get a large number of tests for all of our views and verbs. Specifically in this demo application, in exchange for writing 12 unit tests we have a total of 693 data points indicating that the security posture of our application has not regressed to a prior state:
693 data points in exchange for 12 unit tests seems like a pretty great trade off to me.
Where To From Here
So we now have a pretty solid security foundation for our codebase through the use of dynamic unit test generation, but does this cover everything?
Absolutely not! We can really only check for some basic security controls through dynamic generation. What about instances where someone discovers a specific vulnerability in a specific view? Well that’s something that we can write traditional unit tests for, and then we can use those traditional unit tests in conjunction with these dynamic tests to provide not only a solid foundation of security posture but also guarantees that discovered vulnerabilities do not re-emerge in our codebase. This will be the topic of Addressing Security Regression Through Unit Testing – Part 2, coming to my blog in the near future!
I hope you all enjoyed this, and keep your eyes peeled for part two soon!
