This is a small end of summer release, but with a few important bug fixes

• Most of the poller but gitpoller where broken because of the new committer field, which was marked as non nullable in the sql schema. This is a bug that only happen for new instances of buildbot 2.4.0 (not for upgraded bots)
• The custom_templates feature is now working again, and allows people to override some of the pages of Buildbot to customize the branding.

You can find detailed release notes as usual here

Thanks to our contributors!

$ git shortlog --no-merges -ns v2.4.1...v2.4.0

    33  Povilas Kanapickas
    12  Pierre Tardy
     2  Rajdeep Bharati
     1  Mikhail Sobolev
     1  afg
     1  angeloudy
     1  matthieu

Buildbot 2.4.1 was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

This summer I participated as a Google Summer of Code student under MacPorts. You can find my project abstract here: MacPorts Custom Views Plugin for Buildbot. It has been a wonderful experience so far, and I learned a lot about developing large scale software, thanks to my mentors: Mojca Miklavec, Pierre Tardy, Povilas Kanapickas, and all other members of the Buildbot and MacPorts communities; who helped me with my doubts and provided invaluable feedback.

The Project

I primarily worked on 3 tasks related to the Buildbot CI framework, which is used by MacPorts:

1. Custom Views Plugin

Buildbot frontend is written in AngularJS 1.x, which is an old framework for building single-page applications. Nowadays, developers prefer newer technologies such as Vue or React, which are comparatively easier to use. I wrote a plugin that would let developers write custom UI components using Vue. It is available here: https://github.com/macports/buildbot-macports-custom-views.

The basic plugin is ready and can be used directly by MacPorts (using PyPI package/port) or by hacking it (if other organizations/individuals want to use it for their Buildbot).

The challenging part was having multiple views (pages) inside one app. In the default Buildbot plugins, there is usually one page (a single item in navbar) inside one plugin. In this, I created a main AngularJS module and directives for each page (which is a single group in glMenuServiceProvider).

Another issue that I faced was manipulating the URL from Vue single-file components. One of the requirements was being able to bookmark some changes by id so that it can be shared/viewed later. At first, I tried using vue-router, however, I later realized that it wouldn’t work since the URL was controlled by AngularJS behind the scenes. So, we decided to use the $location API. This was tricky to implement. The new components allow us to see changes/builds before/after a particular date.

Here’s a demo of the changes list page:

What is left?

• Right now, there’s no support for live updates in the plugin UI (see issue #22).
• The Ports Build History page is using the macports-webapp API to fetch data. But this would be removed, and the view would be standalone (without any external API). The work is in progress in pull #22.

2. MacPorts Buildmaster

MacPorts was using Buildbot 0.8, hence the buildmaster needed a revamp to port to Buildbot 2.x. MacPorts has over 20k ports, and to tackle something of that scale, we decided to use virtual builders. In the new setup, we have a portwatcher, which is a builder that triggers build for a port (or list of ports). Now, for each of those ports, there is one virtual builder that starts running a build.

A mock setup that does fake builds is available here: https://github.com/macports-gsoc/macports-buildbot

A demo Buildbot instance has been deployed here.

What is left?

• We need to tweak the master.cfg to do real builds and deploy it to Braeburn (MacPorts server).

3. Features added to Buildbot

With over 20k ports, there would be more than 200k builders, which would make the default www plugins useless if used as-is. So I added some features like tagging, pagination, searching, etc to make it usable for large setups.

The builders page would now let us see all the ports, thanks to virtual builders.

Builds for Change page

I created a webpage that displays all builds for a change (see #3926). To do so, I added an endpoint to the data API to get all builds for a given change.

Committer field added to changes

Earlier, Buildbot had an author field for a change. In version control systems like git, the author and committer may be different. This is important for MacPorts since the author may lack commit permissions, and in case of build failure, both the author and the committer need to be notified.

• My commits to Buildbot are here
• Merged pull requests are here
• Pending pull requests are here
• My commits to macports-ports are here

What is left?

There are a few issues related to console_view which need some further discussion and design decisions (see #4951).

• All builds are not displayed for a change (#3945)
• Incorrect grouping of builds when using force/periodic scheduler (#3995)

These could be fixed if we used the APIs rather than having the complex frontend code which is currently implemented. However, this would lead to the chicken and egg problem: if we use the builds for change API, then we wouldn’t be able to get the forced/nightly builds (which were not triggered by any change), and if we use the changes for build API, then we wouldn’t get all the changes. We might eventually create 2 console views to fix these issues.

MacPorts-Buildbot GSoC Report was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.

This release is bigger than usual as it contains three months worth of work instead of the usual one. The reason for this is that the web UI build system has been completely rewritten to use the standard webpack toolset instead of a custom toolset based on gulp. It took us a while to make sure everything works as before. There can still be some remaining issues. If you see any, please contribute by writing bug reports.

Another highlight of the release is many small enhancements of the web UI by Rajdeep Bharati who has been working on Buildbot as part of his Google Summer of Code project. He is improving the web UI to handle several tens of thousands of builders which is required by the MacPorts project for their continuous integration system. Not all changes have landed in this release yet, but eventually working on Buildbot installations of this size will become more convenient.

This release contains almost 30 bug fixes and enhancements total. The release notes can be found as usual in our documentation.

Thanks a lot to our contributors!

git shortlog --no-merges -ns v2.4.0...v2.3.1
   148  Povilas Kanapickas
    39  Rajdeep Bharati
    22  Pierre Tardy
    13  Bertrand Demiddelaer
    11  Philip Guyton
     6  Gary Benson
     5  Matias Kangasjärvelä
     5  Pawel Krawczyk
     4  Georgi Valkov
     3  Jan Marker
     3  Volker Braun
     2  Cyril Roelandt
     2  Gorka Eguileor
     2  Josh Bialkowski
     2  Mukund Sivaraman
     2  PhilippSelenium
     1  Aki Tuomi
     1  Alan Ray
     1  Andrew Maier
     1  Andrew Spiers
     1  Chandler Bullock
     1  Malcolm Parsons
     1  Michael Corcoran
     1  Mojca Miklavec
     1  Robin Jarry
     1  Shinnok
     1  Tao ZHOU
     1  Thomas Molines
     1  Titouan Christophe
     1  Uri
     1  Valentin Haenel
     1  abull
     1  buildbot
     1  d4ryus
     1  nde
     1  vibbo

Buildbot 2.4.0 was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.

This release has been made and signed by Povilas Kanapickas. The tag and pypi packages have been signed by its gpg key of ID C6F7AE200374452D. This key has been signed by my own key and appended to this post (and to the mailing list annoucement). Povilas has been a great contributor for Buildbot this last year, and he will be also helping as release manager.

The highlight for this release is the conversion of the UI frontend from coffeescript to javascript. More precisely to ECMAScript6 (ES6). This is a big effort to streamline the tech of our frontend. This will help to lower the bar for contributors to hack on the UI, and also help in packaging of the front end to debian distribution. The next step is to switch the UI build system from custom gulp based to more standard webpack. This work is already nearly done here.

As we are doing this transition, we do it step by step. And one of the inevitable step is to drop support for IE11 for a few releases. We know that IE11 is only 2% of the readers of docs.buildbot.net, and we though this should be fine temporarily disable. We believe that we can put it back when we fully migrate to webpack, and upgrade our version of angularJS. If you have IE11 users for some reason that cannot switch to modern browser, please skip this release.

You can see the full release notes as usual inside the documentation and in the github release page. You can even subscribe there by using the watch button, then releases.

A thanks to our contributors!

git shortlog --no-merges -ns v2.3.0...v2.2.0
   126  Povilas Kanapickas
    57  Pierre Tardy
     6  bb-nl
     3  George Shramov
     2  Jan Marker
     2  Robin Jarry
     2 uglycoyote@gmail.com
     1  Thomas Molines
     1  Volker Braun

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1

mQENBE53SRgBCADJBhoKw9dmUYjNm9njQpq0uEDXQb/yBJQ/dJiNbEke8rKF0+eN
XU1bbZzIx6mjywNWsVFiFQV+ty43pO2FoKctD6gPyHYdPM1ZrS6ZlaKv6wXU9HHt
5Wz8znnP5bifwUh+sjkgQ0OETTcv82xmslblFNZWm14EuqKmMbl5jmqAJVpmnX5o
UeWzRG+/eSmg4gV6MQFdNzBX6SaEPShcm4U/qQ+vmxsJtjwjnINHE3UB0nPEQHju
BQAE2FaztYMkm9i3FpVYRfwdz31PGacJxqaTKNPSBUKfHv00dE5nEa9CrJsZpmRa
BRK402zYunFVCI9Sjvijl99P4gfgRClot3WzABEBAAG0JVBvdmlsYXMgS2FuYXBp
Y2thcyA8cG92aWxhc0ByYWRpeC5sdD6JATsEEwECACUCGwMGCwkIBwMCBhUIAgkK
CwQWAgMBAh4BAheABQJSkk6WAhkBAAoJEMb3riADdEUtQT4H/1AFIMJMx8n1+Aco
Zq+xn09nDFmThhmRuu+fiASVNEmiBUcM81DR2UajoYm+FA1W1hP3kAiCLKigO9Xo
bkZ0BCPxzH5n6fnUCNa1Rtnz5u+a4izxvOkgev1XnGPOIKw8oSSGtl4Wclsfihvs
pSDNT9kSBv4+PJp6eIBTsaf5KZb20lj/jp9eiRF5jC5DKbj/7PsERa8E6AvWBMNI
43qIxsDckW7iBLomTzO/nSw/eAxaNwszIJ831bAM3tabA29XmTsVYXyFGWo/WEFY
zqos/7zy2+qWhOw0qI7RJnDG2Tyd0gZzas9F7d1g2bONM75f0CHBrqkud9OweBYQ
Z4uTbsCJATgEEwECACIFAlKSTpMCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheA
AAoJEMb3riADdEUtZvEH/0/GPe6/mg5/0lotGzD9v0wEMC/6w9Wt1sIGnB266sOS
O5zPgUmbsdvl4ijpWL0bcxuWSwpNn0CFKU2k20oqlh+ppUrR8YwPMDcdzfk3zXDP
TvxrIgjJHi5mg3rRtVg1H2Qy1HrlRShCuLw3N5wHrGcXcJp0Exvq/HAZvAAVNdGS
7HgcFPMWClBen5rRGzviHt9uxLuhh9BYyg4H4Lk3fJnPNa6Ry2T1bF/+mFnv9Zoe
rTe5lICxSLcY0L4N50rCE4fHtbV33EdcTCmbOWvCiEHUKD5ugQ3OsN0IKNI+1okb
pNULWHJc60JdzLBBr5bOblAjjxJmTTBvfFey/7Ce2CKJAhwEEwECAAYFAlzQmI8A
CgkQrs1Fa00lMfxvpRAAm7qGkZnLG5YV9oGcuHCudsu+687T7LGsOY33VaozP887
p+OGKbm0aIJwCZ3iOryy2NgQbnlwPmU7OCQnBmYIpKvh2HIEhkjgLmkzoq1SIDrW
TLBP4LbPXxgz5mhHceuRy7xhNJifa+a2uRyYiPwOlNFUCvwQTxKVw91MLTckhIUD
TwSSIbi66zUBXT3GZMucvPtWEY5+4AksQG+v7lVJCg2wU616B1/vQZ5KKoinE3XF
vYCHU7Dw/oxqdhqNbqYcgIXYIVbWz2lE/XOepK8eKkfE0iVjFjd6oq4kN1qnar7O
WaIjPnhZEPGhkDhUJWkxEiwtJbxytYRBf8bin9pIdW1NVAg8xD0nTtA1AnXRnij4
eTDJ22qLP5nefDjywEXoL7I7t+Venw3vek84GlQpO8I6oPuK/NKEGCxQPwchJLNR
LXxTea0VeljkIBG1Jrw649+OrTkS84TG55LWloPAcfOtHJFWSMFQsY/MoRwwCom7
GnNxK9eL92Lc0U4SfQYASBQnwOfNF1oDid8kk2bF/4qIiRVp7XI9XsZ6nq0+51IF
rD5DZZdo9tQJtKKjEo6G8XkpoqmyBLNQHFRcqFlvyLtUx65NWzCmJ+xHbuzoRWqe
GgyCDujleSG4RAg4BZlohPMyYpqAIL19Xk0+2qARFOpP1syRxO1ExWiUc3AmpHW0
MVBvdmlsYXMgS2FuYXBpY2thcyA8cG92aWxhcy5rYW5hcGlja2FzQGdtYWlsLmNv
bT6JATgEEwECACIFAlKPs9YCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJ
EMb3riADdEUtmNEIALt73RvLxEzSBjEjQ5Ir0imUhLMGWP3jk2A8/4z3smfjcGOL
atLvOLVZOpHwFa0F4+hS5G1UOv6w0nkhFqw6EZdVheETNJCO/z3L4UxBlbYS9c/j
BxBgiE5Z/800gndV1bOpqQJs6YRFbHF+T8X7KC0mRU4kjx9mMtBj7/uPbX4wtwsM
CxvhjxBhiZAridO9g3DfSYTfj/Ze4Q4ijfey1vvGkhPpzT/8TJf4MrT6UfvUq8YW
LAldk9b4lldwpQAwAHNr5iudz7/Vy9yBrp2AcIgqidVF1FKShxxW4BqDj8kF2nMe
nBQ1l0P0tPtRGOkrja6uYXd1AZvQKM2g91bsZtmJAhwEEwECAAYFAlzQmI8ACgkQ
rs1Fa00lMfx6lRAAjlQb7qu75f/R7ew5NxnPPDV2tDAILL8wnHAtSrgsYxUdf5FB
alescN1qE19Zjcx81yhV1vAI4bSPKF72sf2T4867q6rpCnSw1iTBsQI8D2K34H1b
362XATGBLtZ1TY7tOHaHZTJrOFNHc7ayFG6IGgqXtsbBW/FPGDvqI7KmzKKNYUKd
sy3o2FE4zK4iEre1446ZF7RhPXfyy+AQ9CpXaQhmw7R+6oqL17Kj3WxL5r+XhL/T
Zjmn2nJgxJdF4jdUz/MGpG5AOwHC6eUfdHULjMdu8OCQtf9WfZGGONc/YWnVEv3/
FX8UW9g0bbZX4DgKiUGXAZkkVqbOpw8CUL3NW8tCAtnvbIokl00+LS4aVxgw1yjJ
tv8iAXwWH2Gns/b0yeQcBIrPZsitQahMMz89jeEpuuargUt7rt70YyAv1esKvc7G
57OSrMu1YZi9XD/uqWQUfcSK1tJgN6Zfr9wwvIpKnFH2WL+245Qwp10mDFWvDu5F
6KjXlR7qFBpJBZdOflLc1paY7oCRyDEy3aSFToKbGK+9BZzYO/IDAfmH8wNPmQmg
s6OSLhW3XCFf6+xZbtFzGSxC290gZRbFS3vXZQeY567vLxcejnUO8lKGfbRxsiqe
Qkj4WGuLN+26QILCoGwEn3GP3aF2VXFkrPY4VvNWbmwyR3retuNKL+SkoSK0KVBv
dmlsYXMgS2FuYXBpY2thcyA8cG92aWxhc2thbkBnbWFpbC5jb20+iQE4BBMBAgAi
BQJSj7PDAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDG964gA3RFLeSd
CACXbnk61ttGl5E1EZWs+Bn0SU1yShsz8WDoXdqMFir2Qnzxd77AcSpYmJzLkxVz
Mo4fczXyDE7B5H7F77kABxMHg6mFPdy3aWyML+xK/EbWl6KLh0CDLeyFbTGZxqvz
zJyRpBfTG77u5LjKaWdLLOHZ3xqSEyEq8noQke+0KmA3/DCIPyUB9heZuZ9sQfn/
hOMipWf+X1LU15OKzvcf0Va5CljsNFZW4Cvt8jjIplvL3zwHwaBO+NV0UvfmYQ5+
mwokDK7S3o4s48HTiwKPt5EJtSkC1okNNwKICmeAZvaMF+9Tro8ODoF6nQCqCwpg
yqlwe5bEWXFcsdntPy3jY3UGiQIcBBMBAgAGBQJc0JiPAAoJEK7NRWtNJTH8Ge0Q
ALD6ooV98EfeIfe9wkMaEeJ5M3YBTCozB4BSp3nIUukYuBI0vkWzAQHPJz5IhUgS
BYMFNwCDv+0eXnJZ6XjR2D+nk00nEfD/04WkGFh0vjJHPjig2WYvCnSH+JNItfgv
DFSr/wpigBcTX8cI2A3YrFAPZhpi4fQH2cCNmiu+8FiDJBAmNjD1M2nWb0Si7uT1
a4YKRDbWu8pKysMUPZefGUPtN9e2kWuTikFbd07WywVI5qUGXhuBYXXg32I2CkI6
cepqxxa5zFJHeX/1LG/NouHXjBgElKm/7zQ+K9ss5CyZyQg/OkAc0Y8ZxGoOjCFd
TFtqyaXD7kQwRGTa2da2a60cgcRVF/gtKshsiJt1u4O8Gf9f5tlMfz88pC0MieST
VwA21DR1m7KpMXot/XhZqUm08ObMKGsUthlyL2iCWKHhHm/MsWNjUed/bMji0WGv
9/XbXanePXCWViaiS+U2XFCNhDLzv/svsyZkqPET2cl7YOS+urmqbT41aV/8ev8k
GdLljDmLgNJURix8TrzaxlK0J7GGas+cowSHBC7vGL360pCo7yfjymwW0wV5UVSG
cdXCn2nXX2ild5uQhVW9Qj54Jfhi5QKKUoSgbNhg7yNd/APvKW0ULN2OksauFulM
fUSyjXRo1EuilQttqVgBRkEJdkO8YgM8/KXCtSKOlhtptCdQb3ZpbGFzIEthbmFw
aWNrYXMgPHRpcjVjM0B5YWhvby5jby51az6JATgEEwECACIFAlHu/3ACGwMGCwkI
BwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEMb3riADdEUt9psIAJjJp4EQpuXxNZQg
vEsE0elSzN3//d2CY9B49S/QbZ0MciwPgla2fPBAqTJYvKOQ0yT2bm2Nj7V8bVyl
KWiqpIMWoQ3pc621n9ZidTWu9wXd+j6ecHkRMCEtIMrb+Fc4BZkqkFCXZ38RmYcV
WOkuTlMoa9OdI6NJu9tDtCR2nbqXi49zP6sECgSMjbKaw6qM17j2rvWP1+EfqjXE
vd6FgdYX2UZ3CdiIrocWVxDhhxcX77iafqWur8I06CqB6NNEV+b2+3N04q5wW+6Z
NrHgklgmwoQESIj9NBATxllGwBfz2gktRWULOm7lqMLOz8170Vwa4oVbkWWNbetU
Vo9wnQmJAhwEEwECAAYFAlzQmI8ACgkQrs1Fa00lMfxyPxAAwMqkiqI6f2ZF0Hg6
cTnbsb+B2MYrKR0Kn9iY3ZWLlk5DvhNnOqUnAmmqcGQEzdS8P/DJ7wnJ9MI07IkV
6kyPn7COnVMRjvikVbEN4C+YWVQmRkNcVXLqqiuYheasW4gb3EoUAC69Qu0RopQu
29iipcosb4sXqmQXavgLxKrLmgrVN3hax/T68BWt4JZFBfZ0Le9g2KxPfMro6FqD
gdz2DSxxxNf57TYvMRuxAumsMJMjzk9PndWXp8rtLcefSneJeQpRQ0Yedf/nTXfn
KZo6WbM0fuEHGIStaH6EQWJwxtrrdowhn1/qSnNcAvAqA8yftEFQjCr5U5nlRIuh
7T7J2vZc8FV+zKtXT4Mten9B8cxkaQz9MRzct0dr62DIxTxJnb+bDLymCccyvaqM
BxFCEH+Prw76CH+THxtJ2HPyACaOgcLiHUQyuet7QcXCv09cVu/CB5uZSsWvpR7y
i34Pm3DchW+W/AV9njQCvhryv1yrnCq6pEWFBgBMFsddSapcQRbwEuQAZggYbA7U
XOs/YOinhajasQRLWmRdAnoWHH5qqh776EfFDPNkDO2z3Mr9FLM0bDpQw0fytJgY
RmtNDk8GZyoOMlufDPhbECuiE9xynxg39LRmc5jq/r+a2bUxtIzHXwGGH6zqKsI3
esF6UuM9DOmOIdV0xkolovZXnjm0HHAxMiBwMTIgPHRpcjVjM0B5YWhvby5jby51
az6JATgEEwECACIFAk53SRgCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJ
EMb3riADdEUtntEH/3Xhdq34gYM9+2dy7vB6N/Otmssz0Z3/DTC9yz1fmfXZF1Am
i53LXntgxC/xcSWp4bZVwCxBxlm8JyvM688jwGaQtZRUmkXZcIeeoXX4mkVXwlK1
eIuxarfR1zuv+tTY8kRSzCAKULIHj7qq+cP7Kv6epBbpQtB0CevMMqnPytylYFqo
iiFr4qfD16Wx8gnSSF/IktlzWrSDNmuhq89mrUKLM9GXeWggshxleAKXHHvekLXQ
fJtjKF0QXl5hGtmux9Oz063nwsYhFLy2gvLnxCiwUmAQd8DKQA/6vfF4ZiyUC0e9
fu0NL4J4BouGVaRtU79v/imAyuGlIO4b/aMQEoeJAhwEEwECAAYFAlzQmI8ACgkQ
rs1Fa00lMfw0mg//aKLZwcKgPhxjbq/WGmzFxMU2yQg2KC+Bh5150CGeW5Uq2vGE
2RvfC2BGkpwFFl8j4oI6RCKNsSSe800SGK1xgdGe9QpHqF+Zk21VhzpjiId6qwf+
xB0az5iqT0R/Eljjj6wdXidsJL7lPiMV1K+2LyiThUpBpOkx2LKsjvT4l5oM7nTD
9de7O0ZJAqNW6lSl/muyGYpZwJn05iCHgUKUJ91nPKaOIoHfZAY2mvNQ0KJ47JWe
TZz9hKa6GnBoMzkjwm1RrUiyS+Ioc/NVihB3xWyq0sconiWI/jjQl1pRnfLqSQsh
ama+OhOgshlCHch2MNM7qUYQCAzrWIqstgGGobhQ92vbw64mymOe7oeYCKiUPtB4
5MIKnWLTjZV0iceEFCzbMDYFkGaK/6AAiFi2pP4AxlyrR6odNBYJySs26mBwIeTa
OP79TG7a/jryG9eNzGePwBgYwGAqJrxtcN3Kb9SS1m7uwbVltQ7NK/9qoNgGuO51
G+zQd0QIaUfWwF4Q/TQLxIKSlg7d2OFUM2tVOEJ87AuDM6yB8SFxbnlof0zlaPlO
gJh9wY/J0lRy0rEEbXPd74g2hiOmFbA8FWmIpW3C4bi1Nz6UbCOErGWFZOuLlNeY
nTNUFKpSEVJKzDMCSMSaLOkEa7FJf41V0NDTcS+w6TiL9X7JvJD+QKBduei0IFBv
dmlsYXMgS2FuYXBpY2thcyA8bWVAcG92a2EubHQ+iQEfBDABAgAJBQJSkk/NAh0g
AAoJEMb3riADdEUtq6cH/R34SOqH0acjnwBQE9UIwPZa7WkehpvKPcHcBdx31j/T
YWucr3oVF3z5UvBNWzQM5sdjMzqYVcdZyIKzuxFPlmRayJHmpK72S/6vpr6QA8nZ
2DSXbTXhaIYNgA2PV/zbjdEp4P7AY6KXoA/+SuqdD1PZom5BU9ykz7+Y/j2rGb2t
rSofJrMbgbDiz0Q/nWRYeWm1k4z4aieVroWB2zfGlTHUO1htubgXJENbpXDtXABD
8MsFT61JxqcJSQd35sNQRh9Jw4mYARMCQEoPbSMRULjYqqSKwj9QCb3Sak5wLurS
VJzwsNLpMm+8uynzJtt+oCeQSqab7SVfaog8Ptg1QCeJATgEEwECACICGwMGCwkI
BwMCBhUIAgkKCwQWAgMBAh4BAheABQJSkQ3ZAAoJEMb3riADdEUt2VMH/RmpMRON
16j5WvV1NdcELc8ctId/W6RlU7G2uKuALJ7wXmB9sFWUjz6piWhktc1pqYpZWOhf
GrSkGeUMI0CYTqfaDVutBAVKfSBuW1rGEqQZdpQVPTtPOIKGRH9D20cdZ6Vntosu
VASTVPqohERK4P6PSUIZPgruuMnH+adlnIvVts+dVSZTaCxV09qvrVd4cSMS9gmd
N9e3+ekfGKOlKWSJqial8GqsvZCnU0TOg6drRFpL+1P6VJFFDqMy9aevSdWbB50n
Tu+j1Yc4NskZR/TOAhc1EdLmEoS8+E3dyjjTxf0Y4iF1N/QZyy88sBm7RTXAahmu
Wu6XkbDNez/0hmmJATgEEwECACIFAlKPs28CGwMGCwkIBwMCBhUIAgkKCwQWAgMB
Ah4BAheAAAoJEMb3riADdEUttjgH/RBO+lg2DWSYeJPBzGCiOKAK7T6GFx7NRbtk
QDXi0Dfs7H40GtgYr13QrTxXWkzeGM8QIQM66d6wG2RtmIUzKjbYL54f+fszOuLh
qeJ93oW7LbYKPDbZ6HyJ+RdKsOKNp8rcGdrFsLHX4EmiRLv6mcDMT6f0o9IyAtwd
Ffl/O0859XohmSs1SQbW92vcf79Vrk6KoFGwd87Tkj1LWfFkrV6tm5lPec3Qw6qE
2PYR0+rrpW1Ou+4hIGnUg2gWBJmEI+It9gac0WF3SmV231E4K59gWooszhGIAwej
95H1n1GmTMgbQQMlhhyMjJPXu2wIEwu7tuR8JnWk3t1lQYB2LUCJATsEEwECACUC
GwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheABQJSj7PHAhkBAAoJEMb3riADdEUt
fEQH/1412xZzjpBOwuirhXRLnIz88krhQAftojc0zTOtoDJuSlpIJEN2Lyj2DczV
YDNstCJ4ANASdxnUsYoknSpmZb2DWR8tLFf89U8GmnBsYjJWereFMIK8t7zftUua
9mSYKl0VeM4YTIseQ1k8Ihe53qwMbJGkbMu7NpWKIP3sPRY6epgOR/qGHgYihXJe
wgM0nRWF1X5rv2Oy2kTcnusgdnCzl7ZpCF/9Uo/UdRurQOQM1H4f9ppGc6qY8sTV
m8KrFgEXJS6MBzUhYf1yuedSIPbbPbMh+CAt4glCTySKyHfmvwgCRQ9GtrsOxuP2
Va8XE2h9tsRoXd30Ms11cC2HQza0IFBvdmlsYXMgS2FuYXBpY2thcyA8cGtAcG92
a2EubHQ+iQEfBDABAgAJBQJSkk/SAh0gAAoJEMb3riADdEUtPDIIALE/uQ1m3f3E
rx/OQtiQTipTNSt66QqIARIVfZFxixn+G4txG8GVxggCJK+Zh3JUUsJ2swYsQwi/
JB8D1DDtLqg1s70/uzvJSr4hDZGlC1VXOgP8mcE/4zA0TVgottS4lJ+7Z4sEPFjs
RwcDGLMU6pLlaF+ToLOLFNMq+TQHTPhKaucdgjX22bXh8qnBd/4kHxAscyn0XodH
lOFqzMVqenvTHV0c/N+MI3no4yTfX8NuiJ2k/6bVXKjBHMbr7J6wstDG0++FjOaw
AmeVsYXUevUGgjpDUkZMS58466YLPV9NHntjvMNJx0vp0ZXOWYJnfzEo8SQ4+0x1
Cspj+gZkWV+JATgEEwECACIFAlKP9vICGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4B
AheAAAoJEMb3riADdEUtjnsH/i7O3Ufu5JREaywshjViWI+cslN3twKa7VFqaqYi
r0tCI9cRx6d0iHhPS9S/iDMagUt+gIpinftd+wuFmfm0KSWKYjNOpQridb61N9TO
9T59wsyYcv5ARdH3Q2B1RXhxZQq9zTBtKjIOsJYETu+omUVaFHlQUGkutgQCZ6FJ
Agv2XdBY4TKTZvzjIoLhr+4JT2lqgxEDF5J4Lr4rNnJCeKtE9q2VNP5HFpmfyz6P
5/8dbMDtvYlesgEpiNKDiOjvTIqCDnOqAhXTaSzSVZ6+dDdXyD47cNhk5YudXsY1
m3mOCeytITi9oBEb0WQko+xVwvORch0tf/83/79VYy4kTYeJATsEEwECACUCGwMG
CwkIBwMCBhUIAgkKCwQWAgMBAh4BAheABQJSkQ3eAhkBAAoJEMb3riADdEUtyoYH
/1NyPMWBfrLz1WEYFyilRn4mw8grGYBPNgo3jkFB6DsKoH8km7ZfeAom1gUeaqLP
eHDtJE+7ib0Avln10oNsj49O4XGce4iXHIAPG2OeQtE/8t7NEjSlgx7t2WSr6ErG
6QXwdLMq4sp3EWZXLo1QxmRvwIyA4aEoGpt3DFMtVS419gIUoe0IVsX55ROpG4Wp
j8u0Zripx91tXmuERq+KH6yqMnm2GntK1Ss5qGyaAdsrPD6osux+G1S64Zj8jh3J
qFFtWgV8MZ2cbCv81SKSB+mBKCoaus7roHuIhE/5f/y0MdL555NC8kUuXJsuqGUW
r8X65BDeNTDpwIFmkUIiqd25AQ0ETndJGAEIAJQxvyDbsQQJg3TKH4QqwFWQCMC3
0aNWotb5kUmOfFXOalsPxQbQ3OC8RocIb5LTPhRCvnyyuG7waPk4EMjxKPawkig/
YzV6TOY5lQmOIWG/S7T/Lbl47arpPgCR/jWnpp6VpUwnV+nC8PzThwgaHd9LtEPN
n0AhRX/tUDw5PE640066XOaYclshjVF4pEb4/v5Zbn1cWaMJ6AsZ84DfNywj5UaT
szmRi33mXDL5hv7qW4UhykSIl5QdGB6ripueTH8X9IVrJA2TJ4pBSrxBoXEcFK5T
PC7fwvKv3H6LXM3n2rQcw+JRzQSxsGVoUsnsgWB7TSaVpAGmay3L9tYgXqkAEQEA
AYkBHwQYAQIACQUCTndJGAIbDAAKCRDG964gA3RFLVWOB/9gdb6I1pcoVtegYPiG
mokwWkA5MNgFMY37k71gVARu4kTcJ8w5tyaNTCP2khEyTDUlj2pY5VFsWpsXNVPf
0NmAkXQV/j9qta9uTlt4f88x9yvITc5pMnF98cUDY09QJPeeV2B5cF3E5JBPfs4b
mtsVBwJi/6isvHd96/vkc8HGlpWJfZJic+K9SNo6jxWy0MiCGcEAqs0WXrGdLY7L
vaf8TI/GBuWIbFkfU/c6Oz90A8HFOMXMG2ffDmtD7hQZIrVCLdhg7hXaq7eXl4Ml
ZGjgKOOZHLTpOrHRbBqT9FJdOjVocUZKA7KD1+5AQvEh5elDZGKIcyWtgIoiUd1S
jKXR
=hKrI
-----END PGP PUBLIC KEY BLOCK-----

Buildbot 2.3.0 was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.

This month’s highlight is the possibility to encrypt worker to master communications thanks to TLS. This works came thanks to Stefan Lundström who stepped in to build upon the work of Georges Racinet. You can find out how to set that up in the worker documentation.

Another big news for this month is the inclusion of Buildbot as a sub org of Python for this year’s Google Summer of Code. We got nice projects this year and very talented mentors, don’t miss that out!

As usual you can find the details of the changes for that release in the releases notes.

A big thanks to the contributors for this release!

git shortlog --no-merges -ns v2.1.0...v2.0.1
    58  Povilas Kanapickas
    13  Robin Jarry
     9  Pierre Tardy
     8  Stefan Lundström
     4  Alexandre Isoard
     3  Georges Racinet
     3  Prakash Acharya
     3  vibbo
     2  Monika Kairaityte
     2  Craig Rodrigues
     1  Frédéric Chapoton
     1  Ari Maniatis

Buildbot 2.1.0 was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.

[edit] 2.0.0 was wrongly marked in pypi as supporting python 2, and was removed from pypi. 2.0.1 is replacing it, now built only for python 3.

This release is a bit special as there has been a security vulnerability reported by mik317 and forwarded by mariadb folks. This vulnerability is of medium criticity as it could only allow to attack services hosted on the same domain (sub-domain cookie injection). Buildbot 1.8.1 is just Buildbot 1.8.0 with a patch for that bug.

Buildbot 2.0.0 is another story. 9039 deletion and 4098 additions over 711 files. Buildbot 2 is just the same as Buildbot 1, but cleaned up from a lot of legacy.

• Python 2 support removal allowed us to remove a lot of unneeded boilerplate (#4569, #4470).
• We removed the support for slave deprecation code (#4547), and other deprecated APIs.
• Then we used the 2.0.0 opportunity to clarify an old implicit policy of supporting only APIs that are documented. If an API is not in the doc, then you can make a PR to improve it in a non backward compatible way. If it is documented, then you need to provide compatibility stubs.
• The cleanup work is not yet finished, but those ones are already huge improvements to the codebase.

Apart from that, there is the usual set of bug fixes, details are as usual in the release notes

Thanks to our contributors!

git shortlog --no-merges -ns v2.0.0...v1.8.0
    54  Povilas Kanapickas
    26  Monika Kairaityte
    25  Craig Rodrigues
    17  Pierre Tardy
     6  Sylvain Joyeux
     5  Robin Jarry
     4  Josh Bialkowski
     2  Chih-Hsuan Yen
     2  Curt J. Sampson
     1  Felix Kastner
     1  Joao Maia

Buildbot 2.0.0 and 1.8.1 was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.

The biggest news is that this is the last one to support python 2.7. We are not planning to make it an LTS release unless users with good reasons to stay on python 2.7 step up to help and maintain a stable branch. Next release will be 2.0 and we will drop support for python 2.7 and 3.4 in that release as other big python projects do. This major release will also remove some deprecated API (e.g. slave vs worker)

As the end of year came, we got sad news about Hyper.sh shutting down. This is sad as we think hyper.sh is good tech and we hope the folks behind it will find way to continue to grow their ideas. This is also sad for us because hyper.sh was hosting and sponsoring the CPU resources which supported our metabuildbot CI. We thank them a lot for that as this really eased the task of maintaining buildbot. We implemented pre-commit CI with this with a lot of coverage and a large configuration matrix. This allowed us to release once a month with lots of confidence on the limited human resources we have. Before that, we used free hosting services like travis, circle-ci, and appveyor (we still use them by the way), but those are limiting the number of parallel build you can make, and as our pipeline grew, the feedback time grew. Maintaining our own latent workers on hyper did help a lot to make all the tests in parallel to try and tend to achieve the 10–15min feedback time.

So we had to rebuild our latent worker environment. In 2019 the obvious choice is Kubernetes. Kubernetes provides a container based orchestration framework that is able to autoscale at various level. Kubernetes is open-source and all the industry is moving toward using or providing this API. Not all the hosting actors have the same level of cost efficiency in term of autoscaling, but I am sure this will progressively smooth out, and choosing Kubernetes is a guarantee of cloud agnosticity.

So we built for 1.8.0 a KubeLatentWorker, and we now run our Buildbot infra on GKE. This support is still experimental and we have some known bugs with fixes that were not able to make it in the release. This kubernetes support was initially contributed by Julien Girardin. Thanks a lot for working on it, this was very useful to have a working starting point.

Our strategy for this deployment is to use node autoscaling with big nodes (6 vCPU). This allows to limit the time downloading build container image. It is only downloaded once and then reused for other builds. The GKE autoscaling starts quite fast (3min cold start including image download), and then takes 10min to scale down (the timer is unfortunately not configurable). Compared to hyper, we have seen our build times decrease a little bit, but as we are still working with a trial account we can only scale to one node, and this is impacting our queue time a lot (the build time is increased because the latent worker’s pods are waiting for available CPU and memory)

You can find the complete release notes as usual in our documentation

Thanks a lot to our contributors for this release!

git shortlog --no-merges -ns v1.8.0...v1.7.0
    20  Craig Rodrigues
    18  Pierre Tardy
    17  Povilas Kanapickas
     1  13costermann
     1  Robin Jarry
     1  ciarancourtney
     1  Pavol Misik
     1  Chih-Hsuan Yen
     1  Curt J. Sampson
     1  Julien Girardin

Buildbot 1.8.0 was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.

This release contains the usual set of little improvements and bug fixes that we see each month. The release note is available here.

The most exiting part of what happened this month is a new step towards more Python3. We decided to deprecate Python2, so that we can focus the development on new features of python3. The best improvement for us is with python 3.5 and the adding of async/await into the language.

Python2 and Twisted have inlineCallbacks and we have been using them extensively for a while. However these do not come without issues. inlineCallbacks requires a bunch of boilerplate code to run between each function call for its magic to work. This comes with performance impacts, and also with difficulties with maintaining the stack traces. With async/await, all those hacks are unneeded, and we will be able to focus more on the logic.

This is why we are starting the work on Buildbot 2.0, which will add an additional level of cleanups including:

• removal of deprecated APIs (slaves APIs are the most important ones)
• removal of support for python 2.7 for the master (worker package will still support python 2.7)

This 2.0 version will not be as painful as the Buildbot nine migration was, as the impacts are much smaller. We plan to release 2.0 as soon as February. Don’t wait to upgrade to python3!

After 2.0 is released we can work on the cleaning of the code (removal of usage of future library, etc).

Then we will think on how we can transition to async/await. This is not as easy as said, as we would like to have this transition be as smooth as possible, and avoid to break API. This means that if a method change to async from inlineCallbacks, callers expecting an inlineCallback should still work, and subclasses overriding this method should also work unchanged.

Thanks to our contributors for this month!

1 % git shortlog --no-merges -ns v1.7.0...v1.6.0
    78  Monika Kairaityte
    45  Povilas Kanapickas
    13  Pierre Tardy
     5  Craig Rodrigues
     2  Pedro Algarvio

Buildbot 1.7.0 was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.

This month was quite busy with Povilas Kanapickas doing lots of improvements in both maintainability and modernisation of the code, helped by Monika Kairaityte. That means improvements in the unit tests, stability of the test suite, so that we are less annoyed by fake positives in our CI, etc.

On top of that we made a few improvements to the UI:

• A chart displaying the evolution on successful build times, that would have helped us to identify a huge performance regression of our test suite.

• A new pending buildrequest which allows us to have a global view of what part of our build workers cannot keep-up with the demand.

This release also contain a few bug fixes and enhancements which are detailed as usual in the release notes.

Thanks to all our contributors!

% git shortlog --no-merges -ns v1.5.0...v1.6.0
    69  Povilas Kanapickas
    39  Monika Kairaityte
    14  Pierre Tardy
     8  Mikhail Sobolev
     4  Andreas Sommer
     4  Chih-Hsuan Yen
     3  Mark Wong
     3  Iblis Lin
     1  Aakash Jain
     1  Marius Stanca
     1  churlinprg
     1  Alex

Buildbot 1.6.0 was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.

No huge feature of bug fix for this release, but some quality of life change whose details are logged as usual in our Release Notes

Thanks to our contributors for this month!

git shortlog --no-merges -ns v1.4.0...v1.5.0
    35  Craig Rodrigues
     8  Zachary Ware
     5  Mikhail Sobolev
     5  Pierre Tardy
     1  Aakash Jain
     1  Pedro Algarvio
     1  Bas Nieuwenhuizen
     1  Richard Purdie
     1  Emmanuel Bouaziz
     1  Maciek Dems

Buildbot 1.5.0 was originally published in Buildbot on Medium, where people are continuing the conversation by highlighting and responding to this story.