When a needed feature is already implemented in some other software, there’re usually some things to consider whether to use that software as dependency or re-implement the feature:
Possible advantages using other software:
not reinventing the wheel
Possible disadvantages using other software:
maybe too big
maybe introduce security issues
maybe is not maintained
Because some bwauths install sbws from the git repository, it is useful to know from which git revision they install it from. We’d prefer to do not see the git revision when it is installed from a git tag or Debian package (which is usually built from a git tag or git archive release).
A first solution would be to obtain the git revision at runtime, but:
sbws is not usually running from the same directory as the git repository, as the installation might install it in other directory.
if some other git repository is the current path, sbws might be obtaining the git revision of that other repository.
So next solution was to obtain the git revision at build/install time. To achieve this, an script should be called from the installer or at runtime whenever __version__ needs to be read.
While it could be implemented by us, there’re two external tools that achieve this.
does what we want, for 19 commits after 1.1.0 tag it’d add an string like ‘1.1.1.dev19+g76ef2fe0.d20200221. We don’t need the date, but it can probably be removed and it does not hurt.
we don’t need to maintain it.
it adds the extra dependency setuptools_scm.
it does not obtain the version from a git archive, though there’s other tool that does that.
the version reported comes only from build time, so if we make a commit without running setup.py, sbws will not report the new version.
it does not add any extra dependency. The first time, versioneer needs to be installed. When run, it will generate versioneer.py and _version.py, which are created from versioneer itself. Then it can be uninstall
does what we want, for 19 commits after 1.1.0 tag it’d add an string like 1.1.0+19.g76ef2fe0. Note the difference with 1.1.0 from he 1.1.1 generated by
we don’t need to maintain it.
it is also capable to obtain the version from a git archive.
the version reported at build time and runtime is the same.
it adds extra code to sbws and it’s quite a lot
the generated code is independent from the upstream and loses the tests.
does not seem maintained.
Because setuptools_scm gives only the version at build time, we decided to use versioneer. We might need to change it in the future if starts giving problems with other git or python versions or we find a way to make setuptools_scm to detect the same version at buildtime and runtime.
See https://github.com/MartinThoma/MartinThoma.github.io/blob/1235fcdecda4d71b42fc07bfe7db327a27e7bcde/content/2018-11-13-python-package-versions.md for other comparative versioning python packages.
Changing Bandwidth file monitoring KeyValues¶
In version 1.1.0 we added KeyValues call
relay_X_count which implied to modify several parts of the code.
We only stored numbers for simpliciy, but then the value of this numbers accumulate over the time and there is no way to know to which number decrease since some of the main objects are not recreated at runtime and do not have attributes about when they were created or updated. The relations between the object do no follow usual one-to-many or many-to-many relationships either, to be able to induce some numbers from the related objects.
The only way we could think to solve this is to store list of timestamps, instead of just numbers, as an attribute in the objects that need to store some counting.
Where the values of the keys come from?¶
In the file system, there are only two types of files were these values can be
- the results files in
Because of the structure of the content in the results files, they can store
KeyValues for the relays, but not for the headers, which need to be stored in
The classes that manage these KeyValues are:
Transition from numbers to datetimes¶
The KeyValues named
_count in the results and the state will be ignored
when sbws is restarted with this change, since they will be written without
_count names in these files json .
We could add code to count this in the transition to this version, but these numbers are wrong anyway and we don’t think it’s worth the effort since they will be correct after 5 days and they have been wrong for long time.
recent_measurement_failure_count will be negative, since it’s
recent_measurement_attempt_count minus all the results.
While the total number of results in the last 5 days is correct, the number of
the attempts won’t be until 5 days have pass.
sbws generate, with 27795 measurement attempts takes 1min instead of a few
The same happens with the
RelayPrioritizer.best_priority, though so far
that seems ok since it’s a python generator in a thread and the measurements
start before it has calculated all the priorities.
The same happens with the
ResultDump that read/write the data in a thread.
All these changes required lot of effort and are not optimal. It was the way we could correct and maintain 1.1.0 version. If a 2.0 version happens, we highly recommend re-design the data structures to use a database using a well maintained ORM library, which will avoid the limitations of json files, errors in data types conversions and which is optimized for the type of counting and statistics we aim to.
Documentation about a possible version 2.0 and the steps to change the code from 1.X needs to be created.