Swagger Jenkins Validation

The Mega Spec Problem

Swagger Spec is a great way to define and document API projects. Your code and documentation live and ship together, allowing for rapid documentation and development. However, you might say it’s too good.

When a team embraces Swagger, documenting every endpoint in great detail, the Swagger files swell to untenable dimensions1. What worked for the PetStore Spec no longer runs against the Swagger Editor webapp2, so developers don’t bother to validate their spec edits before checkin. CI/CD auto updates your Swagger docs on deploy, auto breaking the documentation developers and customers rely upon, ultimately impeding productivity.

Thus, an automatic quality gate on Swagger Specs can mitigate this path to rapid documentation corruption.

Automating Swagger Spec Validation

You can catch two types of spec defects:

  1. Format errors (e.g. JSON or YAML defects)
  2. Invalid spec definitions

Something as simple as running JSON through jq will catch format errors, but we’ll focus on #2, since it’s a catch all.

Yelp’s Swagger Spec Validator is just the ticket3.

#!/bin/bash
set -e
FULL_SPEC_PATH="/path/to/spec.json" # Absolute, please

# Idempotent, local library install
virtualenv .
source bin/activate
pip install swagger-spec-validator

# Test the Spec
python -c "from swagger_spec_validator import validate_spec_url; validate_spec_url('file://${FULL_SPEC_PATH}')"

Expect no output on success, or a trace on error.

For example, let’s corrupt the Pet spec file, but keep it in valid JSON format.

"ApiResponse_WRONG_NAME": {
  "type": "object",
  "properties": {
    ...
  }
}

Validation outputs the following helpful message, noting the ApiResponse entity references are no longer valid.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  ...
  File "/usr/local/lib/python2.7/dist-packages/jsonschema/validators.py", line 387, in resolve_from_url
    return self.resolve_fragment(document, fragment)
  File "/usr/local/lib/python2.7/dist-packages/jsonschema/validators.py", line 421, in resolve_fragment
    "Unresolvable JSON pointer: %r" % fragment
swagger_spec_validator.common.SwaggerValidationError: Unresolvable JSON pointer: u'definitions/ApiResponse'

Jenkins Wireup

Validation is great, but it will only reliably occur when fully automatic (developers are busy people!). Integration with Jenkins will ensure every Swagger Spec commit is valid, and prevent bad specs from getting into master.

Simply add an Execute Shell step and pass the above example (gist link). The non-zero exit code on error will trigger an informative message to the offending developer.

Note: I prefer to have this as the first build step to “fail fast” on checkin, especially for long build processes, but you may prefer to do it last, so all build errors are reported.

Example Jenkins Config Screen:

  1. I had to wrangle an 11K spec file. Use $ref includes if possible. [return]
  2. Running Swagger Editor locally is slow (e.g. 6min npm install/start), consider the prebuilt Docker image. [return]
  3. Aside from the “read the docs” link to the empty docs site. [return]