Managing requirement files
Yesterday evening, I got side-tracked with another "small" side project which turned out to be a bit bigger.
This includes indirect dependencies to make sure builds are reproducible. For example, the requirement file used to install tox in the build images looks like this:
pluggy==0.3.1 py==1.4.31 tox==2.3.1 virtualenv==15.0.2
Now for larger requirement files, this gets quite cumbersome. While version updates are easy thanks to requires.io, it's easy to miss a new dependency (which then isn't explicitly pinned, but silently works) or a removed one.
It'd be preferrable to have a file with "loose" dependencies (so just tox in this case), and then generate the above file from that.
One existing solution for this problem is pip-compile, part of pip-tools. That's what I initially tried using, but I noticed various issues:
- It didn't work with the local paths (requirements inside the qutebrowser repo) I need for pylint.
- It required some options for git requirements I didn't really agree with.
- It doesn't support "blacklisting" a requirement which should be excluded from the compiled file.
- It doesn't support adding comments to the generated file, which I'd need for requires.io.
$ virtualenv venv $ ./venv/bin/pip install -r requirements.txt-raw $ ./venv/bin/pip freeze > requirements.txt
I then wrote a small script which does that for all requirement files. The output looked like what I expected, except for some small differences:
- The commit ID was pinned for git requirements, which I didn't want in this case (as I have an environment to try the newest pylint/astroid).
- The local file path (./scripts/dev/pylint_checkers) got replaced by its package name (qute-pylint).
- I still needed a way to ignore some dependencies and to add comment to the output.
After a bit of thinking, I came up with four special comments (starting with #@) you could put in a requirements.raw-txt:
- "#@ comment: mypkg blah blub" to add a comment for mypkg in the output
- "#@ filter: mypkg != 1.0.0" as shorthand to add a # rq.filter: ... comment for requires.io
- "#@ ignore: mypkg, otherpkg" to not add the given packages to the output
- "#@ replace: foo bar" to replace a line in the output
This worked out really well. I could now have an input like:
beautifulsoup4 CherryPy coverage Flask==0.10.1 httpbin ... #@ filter: Flask < 0.11.0 #@ ignore: Jinja2, MarkupSafe
And get this as output:
# This file is automatically generated by recompile_requirements.py beautifulsoup4==4.4.1 CherryPy==6.0.1 coverage==4.1 decorator==4.0.10 Flask==0.10.1 # rq.filter: < 0.11.0 glob2==0.4.1 httpbin==0.4.1 hypothesis==3.4.0 itsdangerous==0.24 # Jinja2==2.8 Mako==1.0.4 # MarkupSafe==0.23 parse==1.6.6 ...
This was a bit more work than I initially thought (start using pip-compile and be done with it), but was interesting and worth it!
Merging more PRs
Today I was able to get in three more PRs:
- #1264 adding a :history-clear command.
- #1350 improving the page history and adding titles to the completion.
- #1562 making toggling with :set use lower-case true/false.
I also fixed some small issues I encountered (or was reminded of) while developing:
- Using the private browsing mode didn't share cookies between tabs, which it now does.
- Using :restart crashed when there were broken symlinks around.
The todo list for tomorrow roughly looks like this:
- Fix a test added today which fails on Windows
- Decide what to do with redirected URLs in the history
- Merge the trivial doc PR for the Debian packages
- Package and release v0.7.0
So from how things are looking like now, I'll be able to start refactoring things for QtWebEngine on Friday.