EngineeringProductivity/Projects/Everything
From MozillaWiki
Contents
- 1 Project Table
- 2 Motivation
- 3 The List
- 3.1 ActiveData
- 3.2 AlertManager
- 3.3 Autoland
- 3.4 Autophone
- 3.5 Bugzilla / BMO
- 3.6 BugzFeed
- 3.7 Bugzilla/ES Cluster
- 3.8 Buildbot
- 3.9 C++ Code Coverage
- 3.10 charts.mozilla.org
- 3.11 CodeCoverage
- 3.12 Datazilla (retired)
- 3.13 DevelopmentMetrics
- 3.14 DevTools Harness
- 3.15 dzAlerts (retired)
- 3.16 Eideticker (retired)
- 3.17 Janitor
- 3.18 m21s
- 3.19 Marionette
- 3.20 Mozharness
- 3.21 MozReview (Review Board)
- 3.22 OrangeFactor
- 3.23 Ouija
- 3.24 Perfherder
- 3.25 Pulse
- 3.26 PulseGuardian
- 3.27 PulseTranslator
- 3.28 Structured Logging
- 3.29 Talos
- 3.30 TBPL (retired)
- 3.31 Test Informant
- 3.32 Treeherder
- 3.33 web-platform-tests
- 4 Example
Project Table
There is a project table that is a short list of the most active, projects: https://wiki.mozilla.org/EngineeringProductivity/Projects
Motivation
This page exists to provide a low-maintenance list of all our projects. I do not beleive anyone knows how they all interact, but at least this list ensures here are no hidden moving parts.
Template can be found below
The List
ActiveData
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/ActiveData
- Code - https://github.com/klahnakoski/ActiveData
- Contact - klahnakoski@mozilla.com
- Inputs
- Buildbot JSON logs
- hg.mozilla.org
- OrangeFactor
- Perfherder
- Pulse exchange/build/normalized
- Structured Logs from Tests
- Talos
- Text logs (for buildbot and mozharness steps only)
- Outputs/Services
- Query endpoint: http://activedata.allizom.org/query
- Query Tool: http://activedata.allizom.org/tools/query.html
AlertManager
- Docs -https://github.com/jmaher/alert_manager/
- Code - https://github.com/jmaher/alert_manager/
- Contact' - jmaher@mozilla.com
- Inputs
- mozilla.dev.tree-management alerts from graph server
- Outputs/Services
- webUI for organizing and taking action on these alerts.
Autoland
- Docs - http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview/autoland.html
- Code - http://hg.mozilla.org/hgcustom/version-control-tools/file/tip/autoland
- Contact - [:mcote], [:glob]
- Inputs
- MozReview UI & MozReview repo
- Outputs
- Commits land on the repository of record (or try)
- Status reported to MozReview
Autophone
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/AutoPhone
- Code - https://github.com/mozilla/autophone/
- Contact -
- Inputs
- Outputs/Services
- http://phonedash.mozilla.org/ (used by who?)
Bugzilla / BMO
- Docs - https://bmo.readthedocs.io
- Code - https://github.com/mozilla-bteam/bmo
- Contact - Mark Cote, David Lawrence, Dylan Hardison
- Inputs
- Web UI
- WebServices API (REST, XMLRPC, JSONRPC)
- Outputs/Services
- https://bugzilla.mozilla.org (bug reporters, developers, etc)
- Pulse, Bugzfeed (via the [[1]])
- BMO Push service (custom notifications for specific applications)
BugzFeed
- Docs - BMO/ChangeNotificationSystem
- Code - https://github.com/mozilla/bugzfeed
- Contact - mcote@mozilla.com
- Inputs
- BMO via Pulse
- Outputs/Services
- WebSocket servers:
- bugzfeed.mozilla.org
- bugzfeed-dev.allizom.org
- WebSocket servers:
Bugzilla/ES Cluster
- Docs - https://wiki.mozilla.org/BMO/ElasticSearch
- Code - https://github.com/klahnakoski/Bugzilla-ETL
- Contact' - klahnakoski@mozilla.com
- Inputs -
- Bugzilla (direct database access)
- Outputs/Services
Buildbot
- Docs -
- Code -
- Contact -
- Inputs
- hg.mozilla.org
- Outputs/Services
- builder slaves
- tester slaves
- https://wiki.mozilla.org/ReleaseEngineering/BuildAPI ??
- JSON Logs: http://builddata.pub.build.mozilla.org/builddata/buildjson/
C++ Code Coverage
- Docs - https://bugzilla.mozilla.org/show_bug.cgi?id=890116
- Contact - [:ahal]
- Used to be possible to trigger code coverage builds with 'try: -b o -p linux64-cc -u all -t none', but I think this is broken now. Probably isn't terribly hard to revive this. Needs further investigation.
charts.mozilla.org
- Docs -
- Code - https://github.com/mozilla/charts
- Contact' - klahnakoski@mozilla.com
- Inputs -
- BMO/ES Cluster (https://esfrontline.bugzilla.mozilla.org:443/public_bugs/bug_version)
- Outputs/Services
- http://charts.mozilla.org (for management)
CodeCoverage
- Docs - https://wiki.mozilla.org/EngineeringProductivity/Projects/CodeCoverage
- Code
- Contact - jmaher@mozilla.com, klahnakoski@mozilla.com, chmanchester@mozilla.com
- Inputs
- Various tests have been configured with a code coverage option
- Outputs/Services
Datazilla (retired)
- Docs - https://datazilla.readthedocs.org/en/latest/
- Code - https://datazilla.readthedocs.org/en/latest/
- Contact - See Treeherder
- Inputs
- data pushed from Talos (see Graph Server)
- Outputs/Services
- charts at https://datazilla.mozilla.org
- raw results at https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob
DevelopmentMetrics
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/DevelopmentMetrics
- Code - https://github.com/klahnakoski/MoDevMetrics
- Contact - klahnakoski@mozilla.com
- Inputs
- Bugzilla/ES Cluster
- Outputs/Services
- various charts and dashboards
DevTools Harness
- Docs - Needs project page!
- Contact - [:ted.mielczarek]
dzAlerts (retired)
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/Alerts
- Code - https://github.com/klahnakoski/datazilla-alerts
- Contact - klahnakoski@mozilla.com
- Inputs
- Datazilla web service (https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob)
- Eideticket web service (http://eideticker.mozilla.org)
- Outputs/Services
- fxos-perf-alerts@mozilla.org (b2g sheriffs?)
Eideticker (retired)
- Docs - https://wiki.mozilla.org/Project_Eideticker#Documentation
- Code - https://github.com/mozilla/eideticker
- Contact - William Lachance, Dave Hunt
- Inputs
- Nightly/inbound b2g and android builds.
- Outputs/Services
- Eideticker dashboard (http://eideticker.mozilla.org)
Janitor
- Docs - https://mana.mozilla.org/wiki/display/ateam/Janitor
- Code - https://github.com/jankeromnes/janitor, https://github.com/jankeromnes/dockerfiles
- Contact - Jan Keromnes [:janx]
- Inputs
- (none yet)
- Outputs/Services
- https://janitor.technology
- BMO: new patches uploaded by users
- Treeherder: new pushes from users
- Docker Hub: image updates https://hub.docker.com/u/janx/
m21s
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/m21s
- Code - see project page
- Contacts - [:chmanchester][:ahal][:whimboo]
- Inputs
- Mozmill release tests
- Outputs
- 'Firefox Greenlight Tests' in Marionette, reporting to Treeherder
Marionette
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/Marionette
- Code -
- Contact - [:AutomatedTester][:ato]
- What - Mozilla's native WebDriver implementation
Mozharness
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/Mozharness
- Code - https://hg.mozilla.org/build/mozharness/
- Contact - [:armenzg]
- Inputs
- command-line invocation via buildbot or locally
- Outputs/Services
- build/test logs consumed by buildbot, Treeherder, TBPL
MozReview (Review Board)
- Docs - http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview.html
- Code - various pieces of https://hg.mozilla.org/hgcustom/version-control-tools; see docs for more.
- Contact - mcote, glob, smacleod
- Inputs
- BMO
- reviewboard-hg.mozilla.org
- Outputs/Services
- BMO
- Various repositories via Autoland
OrangeFactor
- Docs - Auto-tools/Projects/OrangeFactor
- Code - https://hg.mozilla.org/automation/orangefactor/
- Contact - :jgriffin, :mcote
- Inputs
- buildbot messages (using pulse)
- build logs
- comments from TBPL
- bugzilla bug data
- Outputs/Services
- https://brasstacks.mozilla.com/orangefactor/
- email status [War on Orange]
Ouija
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/Ouija
- Code - https://github.com/dminor/ouija.git
- Contact - dminor, jmaher
- Inputs
- https://hg.mozilla.org/
- tbpl.mozilla.org
- Outputs/Services
Perfherder
- Docs - https://wiki.mozilla.org/EngineeringProductivity/Projects/Perfherder
- Code - https://github.com/mozilla/treeherder/tree/master/treeherder/perf
- Contact - :wlach
- Inputs
- Outputs/Services
Pulse
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/Pulse
- Code - development library: https://hg.mozilla.org/automation/mozillapulse
- Contact - mcote
- Inputs
- a multitude of other systems
- Outputs/Services
- a multitude of other systems
- https://pulse.mozilla.org/
PulseGuardian
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/Pulse/PulseGuardian
- Code - https://github.com/mozilla/pulseguardian/
- Contact - mcote
- Inputs
- Persona (for auth) (soon to be Okta or TaskCluster auth)
- Pulse
- Web UI
- Outputs/Services
- Pulse
- Email notifications
PulseTranslator
- Docs - https://github.com/mozilla/pulsetranslator/blob/master/README.md
- Code - https://github.com/mozilla/pulsetranslator
- Contact - jgriffin@mozilla.com
- Inputs
- Pulse exchange/build
- Outputs/Services
- Pulse exchange/build/normalized
Structured Logging
- Docs - https://mozbase.readthedocs.org/en/latest/loggingreporting.html, https://wiki.mozilla.org/Auto-tools/Projects/Structured_Logging
- Contacts - [:chmanchester][:ahal][:jgraham]
Talos
- Docs - https://wiki.mozilla.org/Buildbot/Talos
- Code - https://hg.mozilla.org/build/talos
- Contact - jmaher@mozilla.com
- Inputs
- build for desktop or android
- commandline (done via mozharness now) to specify which test(s) to run
- Outputs/Services
- log file with raw data
- uploaded summarization of data to graph server
- uploaded raw data to datazilla
Can be referred to as what is run by buildbot or by tbpl since this is the only performance test run on our per revision CI system. Other tools are run at a different cadence.
TBPL (retired)
- Docs - Sheriffing/TBPL
- Code - https://hg.mozilla.org/webtools/tbpl/
- Contact - Sheriffing/TBPL#Developer_Contacts
- Inputs
- Outputs/Services
Test Informant
- Docs - https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant
- Code - See project page
- Contact - [:ahal]
- Inputs
- build notifications via pulse
- test manifests in tests.zip
- Outputs
- reports at e.g., http://brasstacks.mozilla.com/testreports/weekly
Treeherder
- Docs - Auto-tools/Projects/Treeherder
- Code - Auto-tools/Projects/Treeherder#Source_and_Docs
- Contact - [:camd][:mdoglio][:edmorley]
- Inputs
- Pushlog from hg.mozilla.org via json-pushes
- Build/tests from Buildbot via buildapi
- Direct pushes via ``treeherder-client``
- Build/tests from Taskcluster (under development)
- Outputs/Services
web-platform-tests
- Docs - https://wptrunner.readthedocs.org/en/latest/, Needs project page?
- Code
- Contact - [:jgraham]
Example
Use the EMPTY TEMPLATE to add more entries. Include ones you know about, even if you can not fill them.
EXAMPLE
- Docs - some wiki, or read the docs to learn more
- Code - link to code, if it makes sense
- Contact - for ekyle to contact if he has questions
- Inputs -
- automated resources
- services consumed
- Outputs/Services
- automated services provided
- dashboards (and teams that consume them)
EMPTY TEMPLATE
- Docs -
- Code -
- Contact -
- Inputs
- Outputs/Services