Refactoring more things, a working YAML config, and more!

It's day 12 of 20, and I did a lot more work on making everything (including tests) use the new config. I also implemented initial support for reading from/writing to YAML files.

This is how things look now:

154 files changed, 6128 insertions(+), 7000 deletions(-)

This is with all end2end tests also adjusted for the new config. I haven't looked at the unit tests yet, but things are getting better!

1001 passed, 42 skipped, 10 xfailed in 374.00 seconds

I was also finally able to remove almost all the legacy config code, so it's only the new code around now.

Setting values and saving/restoring them already works, with either :set or qute://settings. As I already was working on qute://settings, I also made it work with QtWebEngine, which was something which was requested a lot.

config types

There's a configtypes.py module in qutebrowser which defines all the different types (such as a string, or a list of regexes) in the configuration.

A setting type also can convert a value (of its type) to either a string (for :set), or to whatever the Python code using the config expects (such as a Python list of compiled regex patterns).

When implementing the new config, I first thought that with the new config, the value which will be stored internally would be the converted Python value.

Originally, a type was looking like this:

class BaseType:

    def from_py(self, value):
        # Convert the value from a config.py or YAML file to a
        # value used by the rest of the code (and stored as
        # internally).

    def from_str(self, value):
        # Convert the value from plaintext to a value used like
        # explained above.

However, I found out that that's actually a problem - for some kind of values (like a QColor object, or a proxy object), they can't be converted back to a string losslessly.

The API also was quite confusing - a lost a bit of time because I once assumed from_py would be what gets the value from qutebrowser's code.

I ended up instead storing the value as it's saved in config.py or the YAML file, so as a string/list/dict/bool/int/float.

Then, a type looks like this:

class BaseType:

    def from_str(self, value):
        # Convert the value from plaintext to a YAML-like value.
        # (which is also what's stored internally)

    def from_py(self, value):
        # Convert the value from how it's stored internally
        # to what is used by the rest of the code.

This worked out pretty well, and made it easy to implement :set and reading from YAML.

Mutables

Another big issue I was facing is how to deal with mutables. If a config.py would do this:

headers = conf.content.custom_headers
headers['X-Foo'] = 'bar'

the config wouldn't get updated, and that's probably quite unexpected.

First I tried solving that by returning custom list/dict objects from the config, which notifies the config when they've been mutated:

class ConfigNotifierMixin:

    def __init__(self, data, manager, option, origin):
        self._inited = False
        self._manager = manager
        self._option = option
        self._origin = origin
        super().__init__(data)
        self._inited = True

    def __getitem__(self, name):
        item = super().__getitem__(name)
        return wrap_value(item, manager=self._manager,
                          name=self._option,
                          origin=self._origin)

    def __setitem__(self, name, value):
        value = wrap_value(value, manager=self._manager,
                           name=self._option,
                           origin=self._origin)
        super().__setitem__(name, value)
        if not self._inited:
            return
        self._manager.set(self._option, self._origin)

    def __delitem__(self, name):
        super().__delitem__(name)
        if not self._inited:
            return
        self._manager.set(self._option, self._origin)


class ConfigDict(ConfigNotifierMixin, collections.UserDict):

    pass


class ConfigList(ConfigNotifierMixin, collections.UserList):

    pass


def wrap_value(val, *, manager, name, origin):
    if isinstance(val, (list, ConfigList)):
        return ConfigList(val, manager, name, origin)
    elif isinstance(val, (dict, ConfigDict)):
        return ConfigDict(val, manager, name, origin)
    else:
        return val

But I wasn't been able to make that work properly. Either some sub-values weren't wrapped properly, or I got some funny infinite recursion in Jinja (the templating engine qutebrowser uses).

In the end, I went for a simpler solution - for every value qutebrowser returns, it saves a reference to it as well as a (deep) copy of it. Then, after e.g. a config.py has been executed, it checks the saved values for changes, and calls config.set as appropriate.

YAML saving/restoring

With all the converting already done in configtypes, implementing this was a breeze. Now whatever has been changed via :set or qute://settings is saved to a YAML file (and loaded from there), and it was really easy to implement that. The new config code being much cleaner and more modular already payed off, even when it was a long way there!

As an example, when doing :set tabs.position left and quitting, this ends up in ~/.config/qutebrowser/autoconfig.yml:

# DO NOT edit this file by hand, qutebrowser will overwrite it.
# Instead, create a config.py - see :help for details.

global:
  tabs.position: left

Keybindings

Currently, the bindings are just something like:

bindings.commands = {
    'normal': {
        'gg': 'scroll-perc 0',
        ...
    },
    ...
}

in the config - i.e., just a value like any other, set to a dict of dicts.

However, that turned out to not be very practical - if a single binding is changed, the whole bindings.commands value is treated as modified, and thus saved in the config.

This definitely needs some re-thinking and probably some special-casing.

SQL completion

The pull request for the new sqlite completion (and related completion refactoring) also has seen some progress (mostly thanks to @rcorre).

Today, I also fixed a data corruption issue which showed up in the SQL branch since somewhen recently - turned out it was an issue with how SQL columns were inserted into the database, which caused things to get mixed up.

I also decided to give reviewable.io another try, as reviewing stuff of this size is really cumbersome with GitHub... I did an initial review there, and now I'm wondering whether that'll work out better.

Dropping legacy support

With v1.0 at the horizon, it was also time to think about (finally) throwing out some legacy support.

I opened an issue to discuss about what'd be possible to drop, and what needs to stay. If you use qutebrowser on something other than Archlinux/Windows/macOS, this will probably affect you - and I'd love your input!

Next few days

As mentioned in the Kickstarter already, I'm having a planned eye surgery. I'm leaving for the hospital tomorrow (Wednesday) and should be back home on Friday if everything goes well.

I can't say when I'll feel well enough to start coding again, but I hope that's already early next week somewhen.

Also, somewhen after next week I'll take a longer "break" because I need to start learning for my exams coming up in August (and there's a lot to do).

This means the rest of the work (and also, taking care of the shirts) might be postponed until September, after my exams. But it all really depends on how much days I'll be able to do next week.

Also, for the sake of transparency: I'm starting to worry whether I can really do everything I planned during the crowdfunded time, as only 8 days are left now. The initial refactoring took a lot more time than I thought it would. I hope adding the Python config API will be just as easy as adding the YAML was, but then there are still per-domain settings which probably require a bit of work. My hope is that I'll get (a prototype of) those into the new-config branch by the end, but I'm not sure whether I'll manage to get everything merged to master by then.

Still, of course I'm fully committed to making this happen (after my exams, though) even in my spare time. The biggest part (hours and hours of refactoring, essentially) is done already. It just means it might take a bit longer than expected.

Or maybe everything will work nicely and all this is unfounded. I don't know yet.

All posts

  1. Refactoring all the things!
  2. First week
  3. First 2 days
  4. Getting started again
  5. Second qutebrowser crowdfunding launched!
  6. qutebrowser v0.10.0 released
  7. Wrapping up and looking at the future
  8. Days 39/40/41: Lots of features!
  9. Days 37/38: Hints in master, tests
  10. Day 36: Hints!
  11. Days 33-35: Mouse functionality, and rewriting hints
  12. Days 31/32: More web elements
  13. Days 29/30: Web elements
  14. Days 27/28: Settings and web inspector
  15. Day 24-26: Refactoring the WebElement API
  16. Day 21-23: After Europython and releases
  17. Day 19/20: Bugs everywhere!
  18. qutebrowser v0.8.0 released
  19. Day 18: Javascript
  20. Day 17: Printing and searching
  21. Sending out qutebrowser and pytest stickers
  22. Day 15/16: Merged!
  23. Day 13/14: Almost merged!
  24. Day 12: Tests running!
  25. Day 10/11: Refactoring!
  26. Day 9: A bit of everything
  27. Day 8: More fixing and pytest sprint/training
  28. Day 7: Fixing things
  29. Day 6: Branching off
  30. qutebrowser v0.7.0 released
  31. Day 4: Playing whack-a-mole
  32. Day 3: Last pull requests and managing requirement files
  33. Day 2: More pull requests and nicer test output
  34. Day 1: Merging pull requests, and a stupid bug
  35. About and Timeline