Contributing

First off, thank you for taking the time to contribute.

Note

This repository is a docker image wrapping a python Rest API around the aioswitcher pypi module.

If your contribution is a more of a core contribution, please consider maybe it belongs to the integrating module and not the wrapping docker. You can find the module repository here.

Contributing is pretty straight-forward:

  • Fork the repository

  • Commit your changes

  • Create a pull request against the dev branch

Please feel free to contribute, even to this contributing guideline file, if you see fit.

Items description

Configuration files

Docker

  • Dockerfile is the instruction file for building the docker image.

Python

  • pyscripts is where python scripts are stored.

Shell

  • shellscripts is where shell scripts are stored.

Ignore files

  • .dockerignore used for controlling what goes in the docker image.

  • .gitignore used for controlling what will not be pushed to github.

  • .remarkignore used for ignoring specific files or folders from remark-lint.

Requirement files

  • requirements.txt is a list of python requirements constraints.

  • requirements_prod.txt is a list of python requirements for running the solution.

  • requirements_docs.txt is a list of python requirements for testing and building the documentation.

  • requirements_test.txt is a list of python requirements for testing the solution.

Package management

The package.json file specified by npm manages our dependencies, scripts and some metadata.

Documentation

  • docs/sources is where the rst files for creating the Sphinx Documentation are stored for build, deployment and hosting by Read the Docs.

  • docs/sources/vale_styles is where the vale styles for vale linter are stored. Out of the box, vale comes with three pre-configured plugins: proselint, write-good and joblint. The docs/sources/vale_styles/18F and docs/sources/vale_styles/docs plugins were manually added from the vale repository.

  • docs/Makefile the basic Makefile for Sphinx documentation generator. From the docs path, type make html and sphinx will create the documentation site locally in docs/build.

Continuous Integration

CircleCi

By hook configuration, for every pull request, CircleCi will execute the workflows described in .circleci/config.yml and update the PR conversation with the results.

As a final step, CircleCi will push the Coverage.py XML Report to both CodeCov for code coverage analysis and Codacy for code quality analysis. Both will of course push their results into the PR conversation.

Some of the steps are considered required and may prevent the PR from being merged. But no worries, everything is fixable.

CodeCov

CodeCov is keeping tabs on our code coverage. When a report is uploaded (by CircleCi), CodeCov will check our code coverage and push its conclusions to github.

Codacy

Codacy is here to check the quality of our code. When a report is uploaded (by CircleCi), Codacy will check our code quality and push its conclusions to github.

Requires-io

Requires.io is keeping an eye for versions updates upon the python requirements listed in the various requirements files and in tox.ini file.

David-DM

David-DM is keeping an eye for versions updates upon the npm requirements listed in the package.json file.

Snyk

Snyk is keeping an eye out for vulnerabilities in our npm dependencies, our pypi requirements and our docker image dependencies.

Continuous Deployment

Docker Hub

When a git-tag with the regex of /^[0-9.]+$/ is set, Docker Hub Cloud will build the image based on the Dockerfile instructions file and tag it twice: - <git-tag> - latest

Read the Docs

By hook configuration, for every git-release-tag and push to the dev branch Read the Docs will build the documentation site based on docs/source and host it with the following tags: * stable tag is for the release snapshot. * latest tag is for the dev branch.

By hook configuration, Read the Docs will build the documentation site based on docs/source and host it: * stable tag will be built for every release snapshot. * latest tag will be built for every push the dev branch, so it’ll reflect unreleased changes.

Metadata

By hook configuration, for every docker image build by Docker Hub, MicroBadger will receive a notification and publish the image metadata.

Environments and Tools

Note

The following (Python, virtualenv, nodeenv and Tox) needs to be pre-installed before local testing with tox.

  • The Python scripts in pyscripts was written with Python 3.7 in mind, which added a few tweaks and adjustments, especially in regards to asyncio.

  • Python’s virtualenv, a tool for segregating Python environments.

  • Python’s nodeenv, a tool that enables us to create Node.js virtual environment in resemblance to virtualenv, the tool also allows combining nodeenv inside virtualenv, which is exactly what we’re doing with tox.

  • Docker, as some of the testing automation are performed within a run-once docker container.

  • Tox for automating unit testing in your local environment. * Tox utilizes Python’s virtualenv.

    • Tox is configured with tox.ini.

    • To run tox, simply execute tox from tox.ini’s path. It is recommended that you also run tox --help to get familiar with the various options such as -e and -r that will help you perform faster and better tests.)

Note

Please note: the rest of the steps require no installation on your behalf, but knowing them is important seeing they are key elements for testing with Tox and/or CircleCi.

  • NPM Package: package-json-validator for validating the package.json file.

  • Python Module: doc8 for checking restructured text (rst) files residing in docs/source and used to create the documentation site.

    • doc8 is configured with doc8.ini.

  • Docker Image: jdkato/vale for linting restructured text files residing in docs/source for spelling/syntax mistakes.

  • Python Module: sphinx for building the documentation site from the restructured Text (rst) files residing in docs/source.

  • NPM Package: remark-lint which is a plugin for Remark and the remark-cli command line tool for linting markdown files residing at the base path and in .github.

    • remark-lint uses a couple of presets and tools, all can be found under the dependencies key in package.json.

    • remark-lint ignore list is the file .remarkignore.

    • remark-lint is configured with .remarkrc.

  • NPM Package: markdown-spellcheck for checking the project markdown files for spelling errors.

  • Python Package: yamllint for linting the project yml files. * yamllint is configured with .yamllint.

  • Docker Image: koalaman/shellcheck is used for checking shell script residing in shellscripts.

  • Docker Image: hadolint/hadolint is used for linting the instruction file Dockerfile.

  • Linux Tool: container-structure-test for verifying the docker image content.

    • The tool runs with the helper script shellscripts/container-structure-test-verify.sh, it will not fail if the tool is not present when running tox locally. But this will probably come up with CircleCi so please consider installing the tool manually.

    • container-structure-test is configured with container_structure.yml.

  • Python Package: isort for sorting imports. isort runs automatically with tox in no-fail mode for recommendations only.

  • Python Package: bandit for finding common security issues with against the scripts residing in pyscripts. * bandit is configured with bandit.yml.

  • Python Package: isort for sorting python imports. * isort is configured with pyproject.toml.

  • Python Package: flake8 for checking python scripts residing in pyscripts.

  • Python Package: black for formatting python scripts residing in pyscripts. * black is configured with pyproject.toml.

  • Python Package: mypy for checking static typing tests against python scripts residing in pyscripts. * mypy is configured with mypy.ini.

  • Python Package: pytest as testing framework for running test-cases written in pyscripts/test_server.py. * pytest uses a bunch of awesome plugins listed in requirements_test.txt.

  • Docker Image: circleci/circleci-cli for validating the .circleci/config.yml file.

Testing

Testing is performed with Pytest, Full-featured Python testing tool. The various Rest Http request test-cases is in pyscripts/test_server.py.

For automated local tests, use tox.

Guidelines

Note

The project’s semvar is being handled in both VERSION file for creating the docker image with Makefile and in package.json for packaging handling.

Here are some guidelines (recommendations) for contributing to the switcher_webapi project: * Code docstrings documentation is here.

  • If you add a python dependency, for order keeping and for Snyk’s sake, Please add the dependency with the fixed version to requirements.txt, And add with no version statement in any or all of the other requirements file based on the dependency use case.

  • If you add a new file, please consider is it should be listed within any or all of the ignore files.

  • If you change something inside the docker image it is strongly recommended verifying it with the container-structure-test

  • While not all the test steps in CircleCi and in Tox are parallel to each other, most of them are, so tests failing with Tox will probably also fail with CircleCi.

  • If you’re writing python code, please remember to static type your code or else it will probably fail mypy tests.

  • You can run npm’s script spell-md-interactive for handling all spelling mistakes before testing. You can also choose to run spell-md-report to print a full report instead of handling the spelling mistakes one-by-one. * markdown-spellcheck dictionary is the file .spelling.

NPM Scripts

Before using the scrips, you need to install the dependencies. From the package.json file path, run npm install, Then you can execute the scripts from the same path.

  • npm run lint-md will run remark against markdown files.

  • npm run validate-pkg will run package-json-validator against the package.json file.

  • npm run spell-md-interactive will run markdown-spellcheck against markdown files in an interactive manner allowing us to select the appropriate action.

  • npm run spell-md-report will run markdown-spellcheck against markdown files and print the report to stdout.

Shell Scripts

Note

The shell scripts in shellscripts were written for bash and not for sh.

  • bash shellscripts/container-structure-test-verify.sh will verify the existence of container-structure-test and execute it. The script will exit 0 if the tool doesn’t exists so it will not fail tox.

  • bash shellscripts/push-docker-description.sh allows the deployment of the local README.md file as a docker image description in Docker Hub. Please use it with Makefile as arguments are required.

  • bash shellscripts/run-once-docker-operations.sh <add-argument-here> will verify the existence of Docker before executing various run-once docker operations based on the following arguments. If the script find that Docker is not installed, it will exit 0 so it will not fail tox:

    • argument: lint-dockerfile will execute the docker image hadolint/hadolint linting the local Dockerfile.

    • argument: check-shellscripts will execute the docker image koalaman/shellcheck for checking the shell scripts residing in shellscripts.

    • argument: circleci-validate will execute the docker image circleci/circleci-cli for validating the .circleci/config.yml file.

    • argument vale-rstdocs will execute the docker image jdkato/vale checking for spelling or syntax mistakes in restructured text file residing in docs/source.

Makefile

Using the Makefile is highly recommended, especially in regards to docker operations. Try make help to list all the available tasks: * make docker-build will build image from relative Dockerfile.

  • make docker-build-testing-image will build image from relative Dockerfile using a testing tag.

  • make docker-remove-testing-image will remove the testing image (must be build first).

  • make docker-build-no-cache will build image from Dockerfile with no caching.

  • make structure-test will run the container-structure-test tool against the built image (must be build first) using the relative container_structure.yml file.

  • make docker-build-structure-test will build the image and test the container structure.

  • make docker-build-no-cache-structure-test will build the image and test the container structure.

  • make docker-full-structure-testing` will build the image with the testing tag and remove after structure test.

  • make docker-tag-latest will add latest tag before pushing the latest version.

  • make docker-run will run the built image as a container (must be built first).

  • make docker-build-and-run will build image from Dockerfile and run as container.

  • make docker-build-no-cache-and-run will build image from Dockerfile with no caching and run as container.

  • make push-description will push the relative README.md file as full description to docker hub, requires username and password arguments.

  • make verify-environment-file will verify the existence of the required environment variables file and its content.

Chat

Feel free to join the project’s public Slack Channel. GitHub, Codacy Docker Hub and Snyk are integrated with the channel and keeping its members updated.

Best Practices

This project tries to follow the CII Best Practices guidelines. That’s not an easy task and I’m not sure achieving 100% is even possible for this specific project. At the time writing this, the project has achieved 42%. (The writing of this file was actually according one to those guidelines).

Any contribution bumping up this percentage will be gladly embraced.