Raptor¶
The following documents all testing we have for Raptor.
Benchmarks¶
Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI.
Desktop¶
Tests for page-load performance. The links direct to the actual websites that are being tested. (WX: WebExtension, BT: Browsertime, FF: Firefox, CH: Chrome, CU: Chromium)
amazon (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.amazon.com/s?k=laptop&ref=nb_sb_noss_1
type: pageload
unit: ms
use live sites: false
bing-search (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
type: pageload
unit: ms
use live sites: false
buzzfeed (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.buzzfeed.com/
type: pageload
unit: ms
use live sites: false
cnn (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.cnn.com/2021/03/22/weather/climate-change-warm-waters-lake-michigan/index.html
type: pageload
unit: ms
use live sites: false
ebay (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.ebay.com/
type: pageload
unit: ms
use live sites: false
espn (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
type: pageload
unit: ms
use live sites: false
expedia (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
type: pageload
unit: ms
use live sites: false
facebook (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.facebook.com
type: pageload
unit: ms
use live sites: false
fandom (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase
type: pageload
unit: ms
use live sites: false
google-docs (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://docs.google.com/document/d/1US-07msg12slQtI_xchzYxcKlTs6Fp7WqIc6W5GK5M8/edit?usp=sharing
type: pageload
unit: ms
use live sites: false
google-mail (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://mail.google.com/
type: pageload
unit: ms
use live sites: false
google-search (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.google.com/search?hl=en&q=barack+obama&cad=h
type: pageload
unit: ms
use live sites: false
google-slides (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://docs.google.com/presentation/d/1Ici0ceWwpFvmIb3EmKeWSq_vAQdmmdFcWqaiLqUkJng/edit?usp=sharing
type: pageload
unit: ms
use live sites: false
imdb (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.imdb.com/title/tt0084967/?ref_=nv_sr_2
type: pageload
unit: ms
use live sites: false
imgur (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://imgur.com/gallery/m5tYJL6
type: pageload
unit: ms
use live sites: false
instagram (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.instagram.com/
type: pageload
unit: ms
use live sites: false
linkedin (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.linkedin.com/in/thommy-harris-hk-385723106/
type: pageload
unit: ms
use live sites: false
microsoft (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.microsoft.com/en-us/
type: pageload
unit: ms
use live sites: false
netflix (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.netflix.com/title/80117263
type: pageload
unit: ms
use live sites: false
nytimes (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.nytimes.com/2020/02/19/opinion/surprise-medical-bill.html
type: pageload
unit: ms
use live sites: false
office (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-live-office.manifest
playback recordings: mitm5-linux-firefox-live-office.mp
playback version: 5.1.1
test url: https://office.live.com/start/Word.aspx?omkt=en-US
type: pageload
unit: ms
use live sites: false
outlook (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-live.manifest
playback recordings: mitm5-linux-firefox-live.mp
playback version: 5.1.1
test url: https://outlook.live.com/mail/inbox
type: pageload
unit: ms
use live sites: false
paypal (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
type: pageload
unit: ms
use live sites: false
pinterest (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://pinterest.com/
type: pageload
unit: ms
use live sites: false
reddit (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.reddit.com/r/technology/comments/9sqwyh/we_posed_as_100_senators_to_run_ads_on_facebook/
type: pageload
unit: ms
use live sites: false
tumblr (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.tumblr.com/dashboard
type: pageload
unit: ms
use live sites: false
twitch (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.twitch.tv/videos/326804629
type: pageload
unit: ms
use live sites: false
twitter (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://twitter.com/BarackObama
type: pageload
unit: ms
use live sites: false
wikia (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
type: pageload
unit: ms
use live sites: false
wikipedia (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
type: pageload
unit: ms
use live sites: false
yahoo-mail (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://mail.yahoo.com/
type: pageload
unit: ms
use live sites: false
youtube (BT, FF, CH, CU)
alert on: fcp, loadtime
alert threshold: 2.0
apps: firefox, chrome, chromium
browser cycles: 25
expected: pass
gecko profile entries: 14000000
gecko profile interval: 1
lower is better: true
measure: fnbpaint, fcp, dcf, loadtime
page cycles: 25
page timeout: 60000
playback: mitmproxy
playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest
playback recordings: mitm5-linux-firefox-{subtest}.mp
playback version: 5.1.1
test url: https://www.youtube.com
type: pageload
unit: ms
use live sites: false
Live¶
A set of test pages that are run as live sites instead of recorded versions. These tests are available on all browsers, on all platforms.
Mobile¶
Page-load performance test suite on Android. The links direct to the actual websites that are being tested. (WX: WebExtension, BT: Browsertime, GV: Geckoview, RB: Refbrow, FE: Fenix, CH-M: Chrome mobile)
allrecipes (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://www.allrecipes.com/
type: pageload
unit: ms
use live sites: false
amazon (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://www.amazon.com
type: pageload
unit: ms
use live sites: false
amazon-search (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
type: pageload
unit: ms
use live sites: false
bbc (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
type: pageload
unit: ms
use live sites: false
bing (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://www.bing.com/
type: pageload
unit: ms
use live sites: false
bing-search-restaurants (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
type: pageload
unit: ms
use live sites: false
booking (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://www.booking.com/
type: pageload
unit: ms
use live sites: false
cnn (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://cnn.com
type: pageload
unit: ms
use live sites: false
cnn-ampstories (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://cnn.com/ampstories/us/why-hurricane-michael-is-a-monster-unlike-any-other
type: pageload
unit: ms
use live sites: false
ebay-kleinanzeigen (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://m.ebay-kleinanzeigen.de
type: pageload
unit: ms
use live sites: false
ebay-kleinanzeigen-search (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://m.ebay-kleinanzeigen.de/s-anzeigen/auf-zeit-wg-berlin/zimmer/c199-l3331
type: pageload
unit: ms
use live sites: false
espn (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
type: pageload
unit: ms
use live sites: false
facebook (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://m.facebook.com
type: pageload
unit: ms
use live sites: false
facebook-cristiano (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://m.facebook.com/Cristiano
type: pageload
unit: ms
use live sites: false
google (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://www.google.com
type: pageload
unit: ms
use live sites: false
google-maps (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm5-motog5-gve-{subtest}.manifest
playback recordings: mitm5-motog5-gve-{subtest}.mp
playback version: 5.1.1
test url: https://www.google.com/maps?force=pwa
type: pageload
unit: ms
use live sites: false
google-search-restaurants (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://www.google.com/search?q=restaurants+near+me
type: pageload
unit: ms
use live sites: false
imdb (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://m.imdb.com/
type: pageload
unit: ms
use live sites: false
instagram (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://www.instagram.com
type: pageload
unit: ms
use live sites: false
jianshu (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://www.jianshu.com/
type: pageload
unit: ms
use live sites: false
microsoft-support (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://support.microsoft.com/en-us
type: pageload
unit: ms
use live sites: false
reddit (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://www.reddit.com
type: pageload
unit: ms
use live sites: false
stackoverflow (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://stackoverflow.com/
type: pageload
unit: ms
use live sites: false
web-de (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://web.de/magazine/politik/politologe-glaubt-grossen-koalition-herbst-knallen-33563566
type: pageload
unit: ms
use live sites: false
wikipedia (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
type: pageload
unit: ms
use live sites: false
youtube (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
test url: https://m.youtube.com
type: pageload
unit: ms
use live sites: false
youtube-watch (BT, GV, FE, RB, CH-M)
alert on: fcp, loadtime
alert threshold: 2.0
apps: geckoview, fenix, refbrow, chrome-m
browser cycles: 15
expected: pass
lower is better: true
page cycles: 25
page timeout: 60000
playback: mitmproxy-android
playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest
playback recordings: mitm4-pixel2-fennec-{subtest}.mp
type: pageload
unit: ms
use live sites: false
Scenario¶
Tests that perform a specific action (a scenario), i.e. idle application, idle application in background, etc.
Unittests¶
These tests aren’t used in standard testing, they are only used in the Raptor unit tests (they are similar to raptor-tp6 tests though).
The methods for calling the tests can be found in the Raptor wiki page.