ReleaseEngineering/BuildAPI

From MozillaWiki
Jump to: navigation, search

Overview

The BuildAPI is a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs.

Project Requirements

To run an instance of the buildAPI locally, you will need the following:

It is highly recommended to set up and work within a python virtualenv.

If you are looking to setup a local virtual environment to run BuildAPI, then follow this wiki.

Setup

Mana documentation:

Puppet setup:

Getting Started

The source for the buildapi is available here: buildapi source

Before you get started with that, you should setup your MySQL database instances. To do that all you need to do is download, extract and load the sql dumps provided. So, create the databases first 

$ mysql -u <user> -p
<at prompt>
mysql>create database schedulerdb;
mysql>create database statusdb;
mysql>exit

Download the sql dumps, then load them into the db with: 

mysql schedulerdb -u <user> -p < schedulerdb.sql
mysql statusdb -u <user> -p < statusdb.sql

NOTE: The files unzipped can account for more than 10GB. Watch out! :) NOTE2: The import can take a lot of time.

Now, to get started with the pylons project, simply run: 

 python setup.py install

In the buildapi/ directory. (And inside your virtualenv, if you're using one). which should handle grabbing pylon project dependencies. You will also need to grab and install the google python visualization library from here.

Next you will need to generate a config file for your project, do this by running: 

 paster make-config buildapi config.ini

Now you will need to edit that config.ini to use the correct host(localhost should be fine), and database URLs. Note: MySQL databases take the format `mysql://[username][:password][@hostname]/database_name`. For example: 

sqlalchemy.scheduler_db.url = mysql://username:password@localhost/schedulerdb
sqlalchemy.status_db.url = mysql://username:password@localhost/statusdb

You should now be all set up now! Running the project locally can be done through: 

 paster serve --reload --daemon config.ini

Which starts running the buildapi on your local machine. To view it, open up http://localhost:5000.

Building a simple controller

The buildAPI is built on top of the pylons framework, which enforces a strict MVC stack. To add a new feature to the buildapi you will typically want to create a new controller and associate it with one or more models and views.

To create a simple 'hello world' controller, first use paster to create a template to work with. 

 # in buildapi/
 paster controller hello_world

This automatically creates a workable template and a functional test case for your new controller. You can now hack the newly created file under buildapi/controllers to your heart's content. Models associated with your controller can be created under buildapi/model. Views are created using Mako python templates. Create a template file under buildapi/templates for your new controller.

To associate with a model, simply add an import to your controller; for example: 

 from buildapi.model.<model_name> import <functions to import>

Lastly, once you're done editing your controller and have a resultset to publish, you probably want to render a page. To do this you can call render('/<template-name>.mako') from your controller body to render a view with your results. To access results from your controller, it is best to store the results in a pylons.tmpl_context object, which will make them available to your mako template.

Simple code sample

Now, to write a simple controller that prints "Hello World" using an M-V-C stack, we need three files:

  • controllers/hello_world.py
  • model/hello_world.py
  • templates/hello_world.mako

Our controller lives in: controllers/hello_world.py 

import logging

from pylons import request, response, session, tmpl_context as c, url
from pylons.controllers.util import abort, redirect

from buildapi.lib.base import BaseController, render
from buildapi.model.hello_world import GetMessage

log = logging.getLogger(__name__)

class HelloController(BaseController):

    def index(self):
        # Return a rendered template
        c.message = GetMessage()
        return render('/hello_world.mako')

Our model lives in: model/hello_world.py 

def GetMessage():
    return "Hello world!"

Our view lives in: templates/hello_world.mako Note that the tmpl_context object is available in the template. 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>Hello World!</title>
</head>
<body>
<h1>${c.message}</h1>
</body>
</html>

To access this page, we need to set up a route to it. To do this we add to config/routing.py: 

map.connect('/hello_world', controller='hello_world', action='index')

Now restart buildapi, and point your browser to http://localhost:5000/hello_world.

Maintenance

Updating code

BuildAPI has two components: BuildAPI and Selfserve Agent. They're deployed differently

BuildAPI

BuildAPI updates are handled like any other webapp, through pushes. See also ReleaseEngineering/How_To/Update_BuildAPI

The steps are roughly as follows, from the base directory of buildapi:

  • Push new version of BuildAPI to HG
    • Edit setup.py to show the updated version (eg version='0.3.0' becomes version='0.3.1')
  • Build a new sdist tarball of BuildAPI, using normal python techniques, and upload it to the our pypi repository on relengwebadm.private.scl3.mozilla.com. 
 python setup.py sdist
 VERSION=0.3.1
 scp dist/buildapi-${VERSION}.tar.gz relengwebadm.private.scl3.mozilla.com:

 ssh relengwebadm.private.scl3.mozilla.com
 VERSION=0.3.1
 sudo mv buildapi-${VERSION}.tar.gz /mnt/netapp/relengweb/pypi/pub/
 sudo chmod 664 /mnt/netapp/relengweb/pypi/pub/buildapi-${VERSION}.tar.gz
  • Deploy to the webheads, reusing the relengwebadm ssh session from above
 sudo su -
 cd /data/releng/src/buildapi   # or for staging /data/releng-stage/src/buildapi
 ./update 0.3.1

Selfserve Agent

Deploying new functionality may require deploying a new version of the scripts/selfserve_agent.py file on build masters. This process is handled by puppet, but orthogonal to buildapi itself. The steps are roughly:

  • bump the version of the buildapi repository in "setup.py" and commit (following semantic versioning guidelines, please)
  • add a new tag to the buildapi repository for that version, commit & push
  • create a python sdist tarball of that version:
    • make sure you have a clean checkout
    • locally modify (but do not commit) setup.cfg and comment out "tag_build"
    • run "python setup.py "sdist"
    • updload the file from "dist/"
  • upload to the internal puppetagain python repo.
  • update the manifests for the new version in puppet
  • wait for puppet to deploy the new agent.

Adding branches

buildapi and self-serve pull from [2] to determine which branches are active and caches the result for a few minutes.

Kicking

If things are broken, figure out if it's

report crontask

There's a crontask on relengwebadm that runs every minute to generate the builds-4hr.js file. If this crontask gets "hung", it will prevent updates from occuring. This can happen due to DB connection issues, or bad data in the DB. In these cases, killing the hung crontask is the appropriate fix. There may be a stale lockfile that need removing under /var/lock/buildapi too.

However, the crontask takes about 15 minutes to run on a cold cache. So if its cache (memcached) has gone cold, then killing it before 15 minutes have elapsed is only delaying the failure.

From bug 1005342:

Dustin J. Mitchell [:dustin] 2014-05-02 16:18:14 PDT

I looked at 4:01 pacific, and saw that builds-4hr.js.gz had last been generated at 2:47. This is generated by a crontab that runs every minute, and makes a lot of heavy queries against the DB. On a good day with a cold cache, it's about 15 minutes. With a hot cache it can finish comfortably in one minute. It would seem that Matt's heavy query caused the builds-4hr.js.gz queries to go slowly, and thus caused the file's generation to fall behind.

I found the crontask, appropriately dated 2:47, and killed it. Cron started a new one up on the minute, and that completed in something like 45s.

I don't know why the original job didn't complete soon after mpressman cancelled his query.

Staging

You can visit https://secure-pub-build.allizom.org/buildapi

From Dustin: Buildapi itself doesn't write anything but jobrequests. For the staging instance, those go to a different DB than for production. Like the production version, it reads from the production status and scheduler db's.

The bit that it's missing is a running selfserve-agent. If you trigger some action on the staging instance, it pushes a message to the staging rabbitmq virtualhost and waits for a response, but since there's no agent that response never comes.

Development environment

Running your own instance of BuildAPI is relatively simple. You should:

  • Create a virtualenv (details)
  • Install buildapi with its setup.py script
  • Create a config. You may base your config on the production one, with at least the following changes:
    • Change DEFAULT.email_to to your own email address
    • Change server:main.port to 0.0.0.0
    • Change server:main.port to something unused
    • Comment out app:main.carrot*
    • Set app:main.buildapi.cache to nothing

Troubleshooting Tips

Host

NOTE: buildapi01 is largely unused now and the hosts are managed by webops

No MySQLdb

If after installing you run: 

 paster serve --reload --daemon config.ini

and it does not start the server, check paster.log and if you see "ImportError: No module named MySQLdb" then you need to easy_install MySQL-python into the site-packages of the python it's looking for MySQLdb in.

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/BuildAPI&oldid=1162328"