Managing requirement files

Yesterday evening, I got side-tracked with another "small" side project which turned out to be a bit bigger.

qutebrowser comes with requirement files which pin all dependencies for the automated tests and other tasks (like linters).

This includes indirect dependencies to make sure builds are reproducible. For example, the requirement file used to install tox in the build images looks like this:

pluggy==0.3.1
py==1.4.31
tox==2.3.1
virtualenv==15.0.2

Now for larger requirement files, this gets quite cumbersome. While version updates are easy thanks to requires.io, it's easy to miss a new dependency (which then isn't explicitly pinned, but silently works) or a removed one.

It'd be preferrable to have a file with "loose" dependencies (so just tox in this case), and then generate the above file from that.

One existing solution for this problem is pip-compile, part of pip-tools. That's what I initially tried using, but I noticed various issues:

• It didn't work with the local paths (requirements inside the qutebrowser repo) I need for pylint.
• It required some options for git requirements I didn't really agree with.
• It doesn't support "blacklisting" a requirement which should be excluded from the compiled file.
• It doesn't support adding comments to the generated file, which I'd need for requires.io.

Inspired by a blog post by Kenneth Reitz, I noticed generating such a "compiled" requirements file could be much easier:

$ virtualenv venv
$ ./venv/bin/pip install -r requirements.txt-raw
$ ./venv/bin/pip freeze > requirements.txt

I then wrote a small script which does that for all requirement files. The output looked like what I expected, except for some small differences:

• The commit ID was pinned for git requirements, which I didn't want in this case (as I have an environment to try the newest pylint/astroid).
• The local file path (./scripts/dev/pylint_checkers) got replaced by its package name (qute-pylint).
• I still needed a way to ignore some dependencies and to add comment to the output.

After a bit of thinking, I came up with four special comments (starting with #@) you could put in a requirements.raw-txt:

• "#@ comment: mypkg blah blub" to add a comment for mypkg in the output
• "#@ filter: mypkg != 1.0.0" as shorthand to add a # rq.filter: ... comment for requires.io
• "#@ ignore: mypkg, otherpkg" to not add the given packages to the output
• "#@ replace: foo bar" to replace a line in the output

This worked out really well. I could now have an input like:

beautifulsoup4
CherryPy
coverage
Flask==0.10.1
httpbin
...

#@ filter: Flask < 0.11.0
#@ ignore: Jinja2, MarkupSafe

And get this as output:

# This file is automatically generated by recompile_requirements.py

beautifulsoup4==4.4.1
CherryPy==6.0.1
coverage==4.1
decorator==4.0.10
Flask==0.10.1  # rq.filter: < 0.11.0
glob2==0.4.1
httpbin==0.4.1
hypothesis==3.4.0
itsdangerous==0.24
# Jinja2==2.8
Mako==1.0.4
# MarkupSafe==0.23
parse==1.6.6
...

This was a bit more work than I initially thought (start using pip-compile and be done with it), but was interesting and worth it!

Merging more PRs

Today I was able to get in three more PRs:

• #1264 adding a :history-clear command.
• #1350 improving the page history and adding titles to the completion.
• #1562 making toggling with :set use lower-case true/false.

I also fixed some small issues I encountered (or was reminded of) while developing:

• Using the private browsing mode didn't share cookies between tabs, which it now does.
• Using :restart crashed when there were broken symlinks around.

Outlook

The todo list for tomorrow roughly looks like this:

• Fix a test added today which fails on Windows
• Decide what to do with redirected URLs in the history
• Merge the trivial doc PR for the Debian packages
• Package and release v0.7.0

So from how things are looking like now, I'll be able to start refactoring things for QtWebEngine on Friday.