ReleaseEngineering/How To/Fix Build4h not updating
From MozillaWiki
< ReleaseEngineering | How To
- Nagios alert
<nagios-releng> Fri 02:57:18 PDT [4051] builddata.pub.build.mozilla.org:http file age - /buildjson/builds-4hr.js.gz is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - Last modified 0:11:57 ago - 2896 bytes in 0.003 second response time (http://m.mozilla.org/http+file+age+-+/buildjson/builds-4hr.js.gz)
- Causes
- one possible reason for this is invalid data being imported on one or more buildbot-masters
- Fix
- check 'statusdb' for invalid entries:
mysql> select * from builds, build_properties, properties where builds.id = build_properties.build_id and build_properties.property_id = properties.id and name = 'request_times' and value like '"%'; build_properties.property_id = properties.id and name = 'request_times' and value like '"%'; +-----------+-------------+------------+----------+-----------+---------------------+---------------------+--------+------------------------------------------------+-----------+------+-------------+-----------+-----------+---------------+------------+--------------------------------------------------------------------------------------------------------------------------------------+ | id | buildnumber | builder_id | slave_id | master_id | starttime | endtime | result | reason | source_id | lost | property_id | build_id | id | name | source | value | +-----------+-------------+------------+----------+-----------+---------------------+---------------------+--------+------------------------------------------------+-----------+------+-------------+-----------+-----------+---------------+------------+--------------------------------------------------------------------------------------------------------------------------------------+ | 110001696 | 705 | 615690 | 13071 | 213 | 2016-10-20 09:08:06 | 2016-10-20 10:32:59 | 2 | scheduler | 16077227 | 0 | 403588549 | 110001696 | 403588549 | request_times | postrun.py | "{'128606532': 1476951285L}" | | 110002178 | 707 | 615690 | 12017 | 213 | 2016-10-20 09:57:12 | 2016-10-20 11:21:48 | 2 | scheduler | 16077868 | 0 | 403590537 | 110002178 | 403590537 | request_times | postrun.py | "{'128608528': 1476952425L}" |
- in this particular case the issue was that we had a number of rows with invalid request times, all coming from a single buildbot master.
- to fix it, someone had to decode the values as json, eval()'ing as python expressions, re-encoding as json, and writing back to the database.
- See also
Bug 1311964