Contributing¶
First off, thank you for taking the time to contribute.
Note
This repository is a docker image wrapping a python Rest API around the aioswitcher pypi module.
If your contribution is a more of a core contribution, please consider maybe it belongs to the integrating module and not the wrapping docker. You can find the module repository here.
Contributing is pretty straight-forward:
Fork the repository
Commit your changes
Create a pull request against the dev branch
Please feel free to contribute, even to this contributing guideline file, if you see fit.
TOC
Items description¶
Configuration files¶
.circle/config.yml
is the configuration file for CircleCi Continuous Integration and Deployment Services..codecov.yml
is the configuration file for CodeCov Code Coverage..coveragerc
is the configuration file for Coverage.py creating coverage reports with the pytest-cov plugin..yamllint
is the configuration for yamllint A Linter for YAML Files linting yml files..remarkrc
is the configuration file for remark-lint plugin for Remark linting md files.bandit.yml
is the configuration file for Bandit common security issues finder checking python scripts.container_structure.yml
is the configuration file for GoogleContainerTools container-structure-test validating the container content.doc8.ini
is the configuration file for Doc8 Style Checker checking rst file types.mypy.ini
is the configuration file for MyPy Static Type Checker for checking static types in python scripts.package.json
is npm’s package manager file for managing dependencies, scripts and etc.pyproject.toml
is designated to be the main configuration file for python based on PEP518, not fully operative in this project yet..spelling
is the dictionary/ignore file used by both markdown-spellcheck and vale. Case-insensitive words in this file will not raise a spelling mistake error..vale.ini
is the configuration for vale.tox.ini
is the configuration file for Tox Testing Automation testing the python code.
Ignore files¶
.dockerignore
used for controlling what goes in the docker image..gitignore
used for controlling what will not be pushed to github..remarkignore
used for ignoring specific files or folders from remark-lint.
Requirement files¶
requirements.txt
is a list of python requirements constraints.requirements_prod.txt
is a list of python requirements for running the solution.requirements_docs.txt
is a list of python requirements for testing and building the documentation.requirements_test.txt
is a list of python requirements for testing the solution.
Package management¶
The package.json
file specified by npm manages our dependencies, scripts and some metadata.
Documentation¶
docs/sources
is where the rst files for creating the Sphinx Documentation are stored for build, deployment and hosting by Read the Docs.docs/sources/vale_styles
is where the vale styles for vale linter are stored. Out of the box, vale comes with three pre-configured plugins: proselint, write-good and joblint. Thedocs/sources/vale_styles/18F
anddocs/sources/vale_styles/docs
plugins were manually added from the vale repository.docs/Makefile
the basic Makefile for Sphinx documentation generator. From thedocs
path, typemake html
and sphinx will create the documentation site locally indocs/build
.
Continuous Integration¶
CircleCi¶
By hook configuration, for every pull request, CircleCi will execute the workflows described in
.circleci/config.yml
and update the PR conversation with the results.
As a final step, CircleCi will push the Coverage.py XML Report to both CodeCov for code coverage analysis and Codacy for code quality analysis. Both will of course push their results into the PR conversation.
Some of the steps are considered required and may prevent the PR from being merged. But no worries, everything is fixable.
CodeCov¶
CodeCov is keeping tabs on our code coverage. When a report is uploaded (by CircleCi), CodeCov will check our code coverage and push its conclusions to github.
Codacy¶
Codacy is here to check the quality of our code. When a report is uploaded (by CircleCi), Codacy will check our code quality and push its conclusions to github.
Requires-io¶
Requires.io is keeping an eye for versions updates upon the python requirements listed in the
various requirements files
and in tox.ini
file.
David-DM¶
David-DM is keeping an eye for versions updates upon the npm requirements listed in the package.json file.
Snyk¶
Snyk is keeping an eye out for vulnerabilities in our npm dependencies, our pypi requirements and our docker image dependencies.
Continuous Deployment¶
Docker Hub¶
When a git-tag with the regex of /^[0-9.]+$/
is set, Docker Hub Cloud will build the
image based on the Dockerfile
instructions file and tag it twice:
- <git-tag>
- latest
Read the Docs¶
By hook configuration, for every git-release-tag
and push to the dev
branch
Read the Docs will build the documentation site based on docs/source
and host
it with the following tags:
* stable tag is for the release snapshot.
* latest tag is for the dev branch.
By hook configuration, Read the Docs will build the documentation site based on docs/source
and host it:
* stable tag will be built for every release snapshot.
* latest tag will be built for every push the dev branch, so it’ll reflect unreleased changes.
Metadata¶
By hook configuration, for every docker image build by Docker Hub, MicroBadger will receive a notification and publish the image metadata.
Environments and Tools¶
Note
The following (Python, virtualenv, nodeenv and Tox) needs to be pre-installed before local
testing with tox
.
The Python scripts in
pyscripts
was written with Python 3.7 in mind, which added a few tweaks and adjustments, especially in regards to asyncio.Python’s virtualenv, a tool for segregating Python environments.
Python’s nodeenv, a tool that enables us to create Node.js virtual environment in resemblance to virtualenv, the tool also allows combining nodeenv inside virtualenv, which is exactly what we’re doing with
tox
.Docker, as some of the testing automation are performed within a run-once docker container.
Tox for automating unit testing in your local environment. * Tox utilizes Python’s virtualenv.
Tox is configured with
tox.ini
.To run tox, simply execute
tox
fromtox.ini
’s path. It is recommended that you also runtox --help
to get familiar with the various options such as-e
and-r
that will help you perform faster and better tests.)
Note
Please note: the rest of the steps require no installation on your behalf,
but knowing them is important seeing they are key elements for testing with Tox
and/or
CircleCi
.
NPM Package: package-json-validator for validating the
package.json
file.Python Module: doc8 for checking restructured text (rst) files residing in
docs/source
and used to create the documentation site.doc8 is configured with
doc8.ini
.
Docker Image: jdkato/vale for linting restructured text files residing in
docs/source
for spelling/syntax mistakes.jdkato/vale ignore file is
.spelling
.jdkato/vale is configured with
.vale.ini
.
Python Module: sphinx for building the documentation site from the restructured Text (rst) files residing in
docs/source
.It’s worth mentioning that the documentation site hosted with Read the Docs is based upon the theme sphinx-rtd-theme
NPM Package: remark-lint which is a plugin for Remark and the remark-cli command line tool for linting markdown files residing at the base path and in
.github
.remark-lint uses a couple of presets and tools, all can be found under the dependencies key in
package.json
.remark-lint ignore list is the file
.remarkignore
.remark-lint is configured with
.remarkrc
.
NPM Package: markdown-spellcheck for checking the project markdown files for spelling errors.
markdown-spellcheck dictionary file is
.spelling
.
Python Package: yamllint for linting the project yml files. * yamllint is configured with
.yamllint
.Docker Image: koalaman/shellcheck is used for checking shell script residing in
shellscripts
.Docker Image: hadolint/hadolint is used for linting the instruction file
Dockerfile
.Linux Tool: container-structure-test for verifying the docker image content.
The tool runs with the helper script
shellscripts/container-structure-test-verify.sh
, it will not fail if the tool is not present when runningtox
locally. But this will probably come up with CircleCi so please consider installing the tool manually.container-structure-test is configured with
container_structure.yml
.
Python Package: isort for sorting imports. isort runs automatically with tox in
no-fail
mode for recommendations only.Python Package: bandit for finding common security issues with against the scripts residing in
pyscripts
. * bandit is configured withbandit.yml
.Python Package: isort for sorting python imports. * isort is configured with
pyproject.toml
.Python Package: flake8 for checking python scripts residing in
pyscripts
.Python Package: black for formatting python scripts residing in
pyscripts
. * black is configured withpyproject.toml
.Python Package: mypy for checking static typing tests against python scripts residing in
pyscripts
. * mypy is configured withmypy.ini
.Python Package: pytest as testing framework for running test-cases written in
pyscripts/test_server.py
. * pytest uses a bunch of awesome plugins listed inrequirements_test.txt
.Docker Image: circleci/circleci-cli for validating the
.circleci/config.yml
file.
Testing¶
Testing is performed with Pytest, Full-featured Python testing tool.
The various Rest Http request test-cases is in pyscripts/test_server.py
.
For automated local tests, use tox
.
Guidelines¶
Note
The project’s semvar is being handled in both VERSION
file for creating the docker image
with Makefile
and in package.json
for packaging handling.
Here are some guidelines (recommendations) for contributing to the switcher_webapi
project:
* Code docstrings documentation is here.
If you add a python dependency, for order keeping and for Snyk’s sake, Please add the dependency with the fixed version to
requirements.txt
, And add with no version statement in any or all of the other requirements file based on the dependency use case.If you add a new file, please consider is it should be listed within any or all of the
ignore files
.If you change something inside the
docker image
it is strongly recommended verifying it with the container-structure-testWhile not all the test steps in
CircleCi
and inTox
are parallel to each other, most of them are, so tests failing withTox
will probably also fail withCircleCi
.If you’re writing python code, please remember to static type your code or else it will probably fail
mypy
tests.You can run npm’s script
spell-md-interactive
for handling all spelling mistakes before testing. You can also choose to runspell-md-report
to print a full report instead of handling the spelling mistakes one-by-one. * markdown-spellcheck dictionary is the file.spelling
.
NPM Scripts¶
Before using the scrips, you need to install the dependencies.
From the package.json
file path, run npm install
,
Then you can execute the scripts from the same path.
npm run lint-md
will run remark against markdown files.npm run validate-pkg
will run package-json-validator against thepackage.json
file.npm run spell-md-interactive
will run markdown-spellcheck against markdown files in an interactive manner allowing us to select the appropriate action.npm run spell-md-report
will run markdown-spellcheck against markdown files and print the report to stdout.
Shell Scripts¶
Note
The shell scripts in shellscripts
were written for bash
and not for sh
.
bash shellscripts/container-structure-test-verify.sh
will verify the existence of container-structure-test and execute it. The script willexit 0
if the tool doesn’t exists so it will not failtox
.bash shellscripts/push-docker-description.sh
allows the deployment of the localREADME.md
file as a docker image description in Docker Hub. Please use it withMakefile
as arguments are required.bash shellscripts/run-once-docker-operations.sh <add-argument-here>
will verify the existence of Docker before executing various run-once docker operations based on the following arguments. If the script find that Docker is not installed, it willexit 0
so it will not failtox
:argument:
lint-dockerfile
will execute the docker image hadolint/hadolint linting the localDockerfile
.argument:
check-shellscripts
will execute the docker image koalaman/shellcheck for checking the shell scripts residing inshellscripts
.argument:
circleci-validate
will execute the docker image circleci/circleci-cli for validating the.circleci/config.yml
file.argument
vale-rstdocs
will execute the docker image jdkato/vale checking for spelling or syntax mistakes in restructured text file residing indocs/source
.
Makefile¶
Using the Makefile
is highly recommended, especially in regards to docker operations.
Try make help
to list all the available tasks:
* make docker-build
will build image from relative Dockerfile
.
make docker-build-testing-image
will build image from relativeDockerfile
using a testing tag.make docker-remove-testing-image
will remove the testing image (must be build first).make docker-build-no-cache
will build image fromDockerfile
with no caching.make structure-test
will run the container-structure-test tool against the built image (must be build first) using the relativecontainer_structure.yml
file.make docker-build-structure-test
will build the image and test the container structure.make docker-build-no-cache-structure-test
will build the image and test the container structure.make docker-full-structure-testing`
will build the image with the testing tag and remove after structure test.make docker-tag-latest
will add latest tag before pushing the latest version.make docker-run
will run the built image as a container (must be built first).make docker-build-and-run
will build image fromDockerfile
and run as container.make docker-build-no-cache-and-run
will build image fromDockerfile
with no caching and run as container.make push-description
will push the relativeREADME.md
file as full description to docker hub, requires username and password arguments.make verify-environment-file
will verify the existence of the required environment variables file and its content.
Chat¶
Feel free to join the project’s public Slack Channel. GitHub, Codacy Docker Hub and Snyk are integrated with the channel and keeping its members updated.
Best Practices¶
This project tries to follow the CII Best Practices guidelines. That’s not an easy task and I’m not sure achieving 100% is even possible for this specific project. At the time writing this, the project has achieved 42%. (The writing of this file was actually according one to those guidelines).
Any contribution bumping up this percentage will be gladly embraced.