ReleaseEngineering/BuildAPI/Setup Local Virtualenv for BuildAPI

The purpose of this wiki is to show you how to setup a virtual environment for buildapi and run buildapi inside it.
Note: These instructions are currently only for Mac OSX

Installing/Creating a VirtualEnv

1. Install pip¹ and virtualenv by running:

   sudo easy_install pip
   sudo pip install virtualenv
   

2. Create and activate your own virtualenv, with your desired name (ie <foo>):

   virtualenv <foo>
   cd <foo>
   source bin/activate
   

Installing Dependencies for your Virtual Environment

1. Install MySQL:
   • Download the most recent .dmg for your version of Mac (for me that was 'Mac OS X ver. 10.7 (x86, 64-bit), DMG Archive')
   • Make a free account, then Oracle will ask what you're doing with MySQL (just tell them, it's no secret) and, finally download it and install the .dmg file
2. Create a symbolic link with libmysqlclient.18.dylib inside your virtualenv by running:

   ln -s /usr/local/mysql-5.6.12-osx10.7-x86_64/lib/libmysqlclient.18.dylib /Users/<user>/buildapi-test/lib/python2.7/site-packages/libmysqlclient.18.dylib

3. Add the following lines to <foo>/bin/activate:

   export PATH=$PATH:/usr/local/mysql/bin
   export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/Users/jzeller/buildapi-test/lib/python2.7/site-packages/

4. Install XCode:
   • Go , launch in the App Store and either sign up for a free account or use your existing credentials (It's free, so no stress).
   • Then open XCode, and go to Preferences>Downloads and install 'Command Line Tools'
     • Note 1: If you are sketched out about having your credit card info attached then follow these instructions.
     • Note 2: This should take care of the issues with not having 'clang' or 'gcc' on your machine
5. Install Mercurial, by running:

   pip install mercurial

6. Install the Google Python Visualization Library (version 1.7.1)
   • Note: Based on 'pip freeze' from buildapi01 supplied by catlee
7. Install Redis
8. Create a pip install file by copy/paste'ing the following snippet into a file called 'requirements.txt'
   • Note: This snippet based on 'pip freeze' from buildapi01 supplied by catlee

   Beaker==1.5.4
   FormEncode==1.2.4
   Mako==0.4.1
   MarkupSafe==0.12
   MySQL-python==1.2.3
   Paste==1.7.5.1
   PasteDeploy==1.5.0
   PasteScript==1.7.3
   Pygments==1.4
   Pylons==1.0
   Routes==1.12.3
   SQLAlchemy==0.7.1
   Tempita==0.5.1
   WebError==0.10.3
   WebHelpers==1.3
   WebOb==1.0.8
   WebTest==1.2.3
   amqplib==0.6.1
   anyjson==0.3.1
   carrot==0.10.7
   decorator==3.3.1
   distribute==0.6.14
   nose==1.0.0
   pytz==2011h
   redis==2.4.5
   simplejson==2.1.6
   wsgiref==0.1.2

9. If you haven't already, activate your virtual environment and move into the <foo> directory by running:

   source <foo>/bin/activate
   cd <foo>
   

   • This step ensures that you are installing dependencies ONLY within the virtual environment
10. Install the requirements by running:

    pip install -r requirements.txt

    • Note: If you run across the error "ImportError: No module named pkg_resources", then type this to fix it²:

      curl http://python-distribute.org/distribute_setup.py | python

11. Clone BuildAPI to your local machine:

    hg clone http://hg.mozilla.org/build/buildapi -r dec9d2ddedb715255db9cfce1a890d6a22bc7cd9

Setup Databases

MySQL (from production) - recommended

These directions will get you real (although old) production databases to use in your local BuildAPI instance. Currently zeller has 2 mysql dumps stored on a public folder in his people account that you can use, here: http://people.mozilla.com:jozeller/public. If they are not there, or you want newer data, you'll need to file a bug in 'Server Operations:Database' to request a current dump from schedulerdb and statusdb. Once you have one of these, you can now follow these directions.

Create your own up-to-date database dumps

If you're comfortable working with the database yourself, there are some scripts you can use to get recent db dumps:

• https://hg.mozilla.org/build/braindump/file/de8fa70e0d73/buildapi-related/dump_recent_scheduler_db.sh
• https://hg.mozilla.org/build/braindump/file/de8fa70e0d73/buildapi-related/dump_recent_status_db.sh

You'll then need to import each table, starting with the overall schema:

sh dump_recent_scheduler_db.sh
mysql -uroot -p < ./buildbot_schedulers_schema.sql
pushd scheduler_tables
for f in `ls -1 *.sql`; do
  mysql -uroot -p < ${f};
done
popd

sh dump_recent_status_db.sh
mysql -uroot -p < ./buildbot_status_schema.sql
pushd status_tables
for f in `ls -1 *.sql`; do
  mysql -uroot -p < ${f};
done
popd

Use existing database dumps

1. Setup empty MySQL databases for schedulerdb and statusdb:
   1. Make sure you have started your MySQL server, which is in System Preferences
   2. Enter MySQL:

      mysql -uroot -p

   3. Create schedulerdb and statusdb:

      create database schedulerdb;
      create database statusdb;

   4. Exit MySQL:

      \q

2. Now load the mysql dumps into schedulerdb and statusdb

1. mysql --verbose -uroot -p schedulerdb < schedulerdb_dump.sql

2. mysql --verbose -uroot -p statusdb < statusdb_dump.sql

   • WARNING: Schedulerdb will take roughly 1.5 hours and statusdb will take almost 24 hours. That is assuming that these are full database dumps like the ones that the database team do dumps for once per day.

SQlite

These directions will get you empty sqlite databases. If you'd like to run from real production data, I suggest setting up mysql with production dumps from schedulerdb and statusdb.

1. Run the following commands from your <foo>/buildapi/ directory:

   cd buildapi
   sqlite3 schedulerdb
   .databases
   .exit
   sqlite statusdb
   .databases
   .exit
   

Setting up RabbitMQ

buildapi relies on a messaging queue to process self-serve requests. If you're only working with other parts of buildapi, you may be able to get by without setting up a messaging queue.

RabbitMQ is probably the easiest to setup. Follow the links for your platform on their download page.

Once RabbitMQ is setup, you need to add a buildapi user and give it permissions to access the messages it needs. For a staging env, the easiest way to do this is with blanket permissions. There are 2 ways of going about this:

1. catlee gave me the following script he uses on linux to setup RabbitMQ for use with buildapi:

    #!/bin/sh
    set -e
    sudo invoke-rc.d rabbitmq-server restart
    sudo rabbitmqctl stop_app
    sudo rabbitmqctl reset
    sudo invoke-rc.d rabbitmq-server restart
    sudo rabbitmqctl add_user buildapi buildapi
    sudo rabbitmqctl set_permissions buildapi '.*' '.*' '.*'
    sudo invoke-rc.d rabbitmq-server restart 
   

   I was able to use a version of this on Mac (using the rabbitmq-server.wrapper instead) to get things working locally.
2. Here is the step by step I used to get RabbitMQ up and running and working with buildapi on Mac OSX.
   1. If MacPorts is not already installed, then go here.
   2. Once you've ensured that MacPorts is installed you can install RabbitMQ: sudo port install rabbitmq-server
      • The instructions for this can be found here
   3. Once RabbitMQ is installed, you need to add buildapi as a user. Enable the rabbitmq_management plugin: rabbitmq-plugins enable rabbitmq_management
      • The instructions for this can be found here
   4. Then restart RabbitMQ: sudo /opt/local/etc/LaunchDaemons/org.macports.rabbitmq-server/rabbitmq-server.wrapper restart
   5. Now go to http://localhost:15672/ and use the username/password combo of guest/guest
   6. Once in, go to 'Admin'
   7. Select the 'Add a user' option and enter the following
      • Username: buildapi
      • Password: buildapi
      • Tags: administrator
   8. Now submit the new user by selecting 'Add user'
   9. Once you have added 'buildapi' as a new user, you will see it listed under the 'All users' section above
   10. Select 'buildapi' and a window for permissions will come up
   11. Make sure that the permissions are set to the following
       • Virtual Host: /
       • Configure regexp: .*
       • Write regexp: .*
       • Read regexp: .*
   12. Now submit these permissions by selecting 'Set Permission'
   13. Once you have done this, the only thing left is to adjust the config.ini file at the root of buildapi to include the following lines
       • carrot.hostname = localhost
       • carrot.vhost = /
       • carrot.userid = buildapi
       • carrot.password = buildapi
       • carrot.exchange = buildapi.control
       • carrot.consumer.queue = buildapi-web
   14. Once you have made sure that the previous lines were added to your config.ini file in buildapi, then start up buildapi
   15. Go to http://localhost:15672/#/connections and a connection with the username 'buildapi' should be listed and the state should be 'running'

Setting up Selfserve-Agent.py

In order to have builds that are added to the mq (ie RabbitMQ) picked up by another process to finally be executed, you must setup selfserve-agent.py. In order to do this, do the following

1. Add a new user to RabbitMQ
   • Make sure that RabbitMQ is running and that the user buildapi has already been added
   • Now go to http://localhost:15672/ and use the username/password combo of guest/guest
   • Once in, go to 'Admin'
   • Select the 'Add a user' option and enter the following
     • Username: selfserveagent
     • Password: selfserveagent
     • Tags: administrator
   • Now submit the new user by selecting 'Add user'
   • Once you have added 'selfserveagent' as a new user, you will see it listed under the 'All users' section above
   • Select 'selfserveagent' and a window for permissions will come up
   • Make sure that the permissions are set to the following
     • Virtual Host: /
     • Configure regexp: .*
     • Write regexp: .*
     • Read regexp: .*
   • Now submit these permissions by selecting 'Set Permission'
2. Create a new config file called selfserve-agent.ini, in the same directory as config.ini. The contents of this config should be:

   [db]
   url = mysql://root:@localhost/schedulerdb
   
   [carrot]
   hostname = localhost
   vhost = /
   userid = selfserveagent
   password = selfserveagent
   exchange = buildapi.control
   queue = buildapi-web
   
   [masters]
   masters-url = /Users/<username>/buildapi-test/buildapi/production-masters.json
   buildbot = <%= scope.lookupvar('::config::selfserve_agent_root') %>/bin/buildbot
   sendchange-master = <%= scope.lookupvar('::config::selfserve_agent_sendchange_master') %>
   
   [branches]
   url = /Users/<username>/buildapi-test/buildapi/production-branches.json
   
   [clobberer]
   url = <%=scope.lookupvar('::config::selfserve_agent_clobberer_url')%>
   

3. Once this is all configured, make sure that buildapi is running and then to start selfserve-agent.py, open a new Terminal tab and use hte following command to start the standalone process: python buildapi/scripts/selfserve-agent.py -w -v

Setup/Running local master scheduler on laptop - not dev-master

From the wiki on How To Setup Personal Development Master

1. Download fresh_bb_setup to ~/
2. ./fresh_bb_setup Buildbot
3. cd Buildbot; source bin/activate
4. cd buildbot-configs; ./test-masters.sh
5. cd ~
6. cd BASEDIR/build-master/
7. Modify line 2 of Makefile from BUILDBOT=$(PWD)/bin/buildbot to be BUILDBOT=$(PWD)/../bin/buildbot
8. Modify line 11 of master_config.json so that 'release_branches' contains an empty list. It did contain ["mozilla-1.9.2", "mozilla-beta"] and they are unnecessary for a local setup.
9. make checkconfig
   • Should say: Config file is good!
10. Link up buildbot to your MySQL schedulerdb
    • Modify BASEDIR/build-master/master.cfg from c['db_url'] = "sqlite:///state.sqlite" to c['db_url'] = "mysql://buildbot:buildbot@localhost/temp_schedulerdb"
      • NOTE: The regex does not like a mysql user that has no password. Originally, I had specified root: , ie "mysql://root:@localhost/temp_schedulerdb", so I made a new user named buildbot, password buildbot.
    • pip install MySQL-python==1.2.3
    • Modify bin/activate by adding export PATH=$PATH:/usr/local/mysql/bin
11. make start
    • It will appear as if start has failed, but fear not! You'll see the message below, but this is simply because the buildbot process is launched as a dameon and buildmaster waits to give the all configuration is complete signal. However, Mozilla's configuration files are way to complicated to be dealt with in under 10 seconds, so it times out and you see the message below:

      (Buildbot)localhost:build-master jzeller$ make start
      cd master && /Users/jzeller/Buildbot/build-master/../bin/buildbot  start $PWD
      Following twistd.log until startup finished..
      2014-01-22 14:47:30-0800 [-] Log opened.
      2014-01-22 14:47:30-0800 [-] twistd 12.0.0 (/Users/jzeller/Buildbot/bin/python 2.6.7) starting up.
      2014-01-22 14:47:30-0800 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
      2014-01-22 14:47:30-0800 [-] monkeypatch_twisted_cbLogin applied
      2014-01-22 14:47:30-0800 [-] Creating BuildMaster — buildbot.version: 0.8.2
      2014-01-22 14:47:30-0800 [-] loading configuration from /Users/jzeller/Buildbot/build-master/master.cfg
      2014-01-22 14:47:30-0800 [-] unable to import dnotify, so Maildir will use polling instead
      2014-01-22 14:47:30-0800 [-] nextAWSSlave: start
      2014-01-22 14:47:30-0800 [-] nextAWSSlave: start
      2014-01-22 14:47:37-0800 [-] nextAWSSlave: start
      2014-01-22 14:47:37-0800 [-] nextAWSSlave: start
      
      The buildmaster took more than 10 seconds to start, so we were unable to
      confirm that it started correctly. Please 'tail twistd.log' and look for a
      line that says 'configuration update complete' to verify correct startup.
      
      make: *** [start] Error 1

12. To doublecheck that the configuration was successful, just follow the directions and type: less twistd.log | grep "configuration update complete". Make sure that the most recent timestamp matches the time with which you tried a make start, but as long as you see it pop up then you're good!
13. Go to http://localhost:8501/
14. If you see a page with "Welcome to the Buildbot for the Firefox project!" then you have been successful!
15. You can now check out what buildbot has pending by going to http://localhost:8501/waterfall

Configuring BuildAPI

1. To get started with the pylons project, make sure you are in the <foo>/buildapi/ directory and inside your virtualenv, then simply run:

   python setup.py develop

   • This handles grabbing pylon project dependencies. You will also need to grab and install the google python visualization library from here.
2. Now generate a config file for your project, by running:

    paster make-config buildapi config.ini
   

3. Now edit that config.ini
   • If you used the MySQL configuration with a production database
     • Edit config.ini to use the correct database URLs.

       # SQLAlchemy database URLs
       # Buildbot's scheduler database
       sqlalchemy.scheduler_db.url = mysql://root:@localhost/schedulerdb
       # Mozilla's status databasea
       sqlalchemy.status_db.url = mysql://root:@localhost/statusdb
       # buildapi specific database
       sqlalchemy.buildapi_db.url = sqlite:///production.db

   • If you used SQLite configuration without a production database
     • Edit config.ini to use the correct database URLs. SQLite databases take the format
       `sqlite:////absolute/path/to/database_name`.
       For example:

       # SQLAlchemy database URLs
       # Buildbot's scheduler database
       sqlalchemy.scheduler_db.url = sqlite:////Users/<username>/<foo>/buildapi/schedulerdb
       # Mozilla's status databasea
       sqlalchemy.status_db.url = sqlite:////Users/<username>/<foo>/buildapi/statusdb
       # buildapi specific database
       sqlalchemy.buildapi_db.url = sqlite:///production.db
       

   • Add the following line below "branches = b1, b2":

     branches_url = http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json

   • Change the port listed under [server:main] to 7000, or some other port that is available, like so:

     port = 7000

     • Note: If you run into the error "socket.error: [Errno 48] Address already in use", then try a new port. This is likely because the port you've chosen really is busy, and is not some wacky error in this case.

Starting BuildAPI

1. Open a separate Terminal tab and run the following to start the redis server:

   redis-server

2. You should now be all set up now! Running the project locally can be done through:

   paster serve --reload -v config.ini

   Which starts running the buildapi on your local machine. To view it, open up http://localhost:7000. Or, replace 7000 with whichever port you changed to.

ProTips

1. Install SequelPro to view and get to know the underlying database contents that buildapi uses, if you are on Mac OS X

Troubleshooting

1. If you run into the error "socket.error: [Errno 48] Address already in use", then try a new port. This is likely because the port you've chosen really is busy, and is not some wacky error in this case.
   1. Alternatively, you can simply kill all processes currently listening to that work with the following (this example uses port 5000): lsof -P | grep ':5000' | awk '{print $2}' | xargs kill -9

References

1. http://www.tlswebsolutions.com/mac-os-x-lion-setting-up-django-pip-virtualenv-and-homebrew/
2. http://stackoverflow.com/questions/7446187/no-module-named-pkg-resources