Raptor

The following documents all testing we have for Raptor.

Benchmarks

Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI.

Desktop

Tests for page-load performance. The links direct to the actual websites that are being tested. (WX: WebExtension, BT: Browsertime, FF: Firefox, CH: Chrome, CU: Chromium)

amazon (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.amazon.com/s?k=laptop&ref=nb_sb_noss_1

  • type: pageload

  • unit: ms

  • use live sites: false

bing-search (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.bing.com/search?q=barack+obama

  • type: pageload

  • unit: ms

  • use live sites: false

buzzfeed (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.buzzfeed.com/

  • type: pageload

  • unit: ms

  • use live sites: false

cnn (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.cnn.com/2021/03/22/weather/climate-change-warm-waters-lake-michigan/index.html

  • type: pageload

  • unit: ms

  • use live sites: false

ebay (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.ebay.com/

  • type: pageload

  • unit: ms

  • use live sites: false

espn (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: http://www.espn.com/nba/story/_/page/allstarweekend25788027/the-comparison-lebron-james-michael-jordan-their-own-words

  • type: pageload

  • unit: ms

  • use live sites: false

expedia (BT, FF, CH, CU)
facebook (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.facebook.com

  • type: pageload

  • unit: ms

  • use live sites: false

fandom (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase

  • type: pageload

  • unit: ms

  • use live sites: false

google-docs (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://docs.google.com/document/d/1US-07msg12slQtI_xchzYxcKlTs6Fp7WqIc6W5GK5M8/edit?usp=sharing

  • type: pageload

  • unit: ms

  • use live sites: false

google-mail (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://mail.google.com/

  • type: pageload

  • unit: ms

  • use live sites: false

google-search (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.google.com/search?hl=en&q=barack+obama&cad=h

  • type: pageload

  • unit: ms

  • use live sites: false

google-slides (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://docs.google.com/presentation/d/1Ici0ceWwpFvmIb3EmKeWSq_vAQdmmdFcWqaiLqUkJng/edit?usp=sharing

  • type: pageload

  • unit: ms

  • use live sites: false

imdb (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.imdb.com/title/tt0084967/?ref_=nv_sr_2

  • type: pageload

  • unit: ms

  • use live sites: false

imgur (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://imgur.com/gallery/m5tYJL6

  • type: pageload

  • unit: ms

  • use live sites: false

instagram (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.instagram.com/

  • type: pageload

  • unit: ms

  • use live sites: false

linkedin (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.linkedin.com/in/thommy-harris-hk-385723106/

  • type: pageload

  • unit: ms

  • use live sites: false

microsoft (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.microsoft.com/en-us/

  • type: pageload

  • unit: ms

  • use live sites: false

netflix (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.netflix.com/title/80117263

  • type: pageload

  • unit: ms

  • use live sites: false

nytimes (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.nytimes.com/2020/02/19/opinion/surprise-medical-bill.html

  • type: pageload

  • unit: ms

  • use live sites: false

office (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-live-office.manifest

  • playback recordings: mitm5-linux-firefox-live-office.mp

  • playback version: 5.1.1

  • test url: https://office.live.com/start/Word.aspx?omkt=en-US

  • type: pageload

  • unit: ms

  • use live sites: false

outlook (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-live.manifest

  • playback recordings: mitm5-linux-firefox-live.mp

  • playback version: 5.1.1

  • test url: https://outlook.live.com/mail/inbox

  • type: pageload

  • unit: ms

  • use live sites: false

paypal (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.paypal.com/myaccount/summary/

  • type: pageload

  • unit: ms

  • use live sites: false

pinterest (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://pinterest.com/

  • type: pageload

  • unit: ms

  • use live sites: false

reddit (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.reddit.com/r/technology/comments/9sqwyh/we_posed_as_100_senators_to_run_ads_on_facebook/

  • type: pageload

  • unit: ms

  • use live sites: false

tumblr (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.tumblr.com/dashboard

  • type: pageload

  • unit: ms

  • use live sites: false

twitch (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.twitch.tv/videos/326804629

  • type: pageload

  • unit: ms

  • use live sites: false

twitter (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://twitter.com/BarackObama

  • type: pageload

  • unit: ms

  • use live sites: false

wikia (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://marvel.fandom.com/wiki/Black_Panther

  • type: pageload

  • unit: ms

  • use live sites: false

wikipedia (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://en.wikipedia.org/wiki/Barack_Obama

  • type: pageload

  • unit: ms

  • use live sites: false

yahoo-mail (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://mail.yahoo.com/

  • type: pageload

  • unit: ms

  • use live sites: false

youtube (BT, FF, CH, CU)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: firefox, chrome, chromium

  • browser cycles: 25

  • expected: pass

  • gecko profile entries: 14000000

  • gecko profile interval: 1

  • lower is better: true

  • measure: fnbpaint, fcp, dcf, loadtime

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy

  • playback pageset manifest: mitm5-linux-firefox-{subtest}.manifest

  • playback recordings: mitm5-linux-firefox-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.youtube.com

  • type: pageload

  • unit: ms

  • use live sites: false

Live

A set of test pages that are run as live sites instead of recorded versions. These tests are available on all browsers, on all platforms.

Mobile

Page-load performance test suite on Android. The links direct to the actual websites that are being tested. (WX: WebExtension, BT: Browsertime, GV: Geckoview, RB: Refbrow, FE: Fenix, CH-M: Chrome mobile)

allrecipes (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.allrecipes.com/

  • type: pageload

  • unit: ms

  • use live sites: false

amazon (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.amazon.com

  • type: pageload

  • unit: ms

  • use live sites: false

amazon-search (BT, GV, FE, RB, CH-M)
bbc (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.bbc.com/news/business-47245877

  • type: pageload

  • unit: ms

  • use live sites: false

bing (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.bing.com/

  • type: pageload

  • unit: ms

  • use live sites: false

bing-search-restaurants (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.bing.com/search?q=restaurants

  • type: pageload

  • unit: ms

  • use live sites: false

booking (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.booking.com/

  • type: pageload

  • unit: ms

  • use live sites: false

cnn (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://cnn.com

  • type: pageload

  • unit: ms

  • use live sites: false

cnn-ampstories (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://cnn.com/ampstories/us/why-hurricane-michael-is-a-monster-unlike-any-other

  • type: pageload

  • unit: ms

  • use live sites: false

ebay-kleinanzeigen (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://m.ebay-kleinanzeigen.de

  • type: pageload

  • unit: ms

  • use live sites: false

ebay-kleinanzeigen-search (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://m.ebay-kleinanzeigen.de/s-anzeigen/auf-zeit-wg-berlin/zimmer/c199-l3331

  • type: pageload

  • unit: ms

  • use live sites: false

espn (BT, GV, FE, RB, CH-M)
facebook (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://m.facebook.com

  • type: pageload

  • unit: ms

  • use live sites: false

facebook-cristiano (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://m.facebook.com/Cristiano

  • type: pageload

  • unit: ms

  • use live sites: false

google (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.google.com

  • type: pageload

  • unit: ms

  • use live sites: false

google-maps (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm5-motog5-gve-{subtest}.manifest

  • playback recordings: mitm5-motog5-gve-{subtest}.mp

  • playback version: 5.1.1

  • test url: https://www.google.com/maps?force=pwa

  • type: pageload

  • unit: ms

  • use live sites: false

google-search-restaurants (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.google.com/search?q=restaurants+near+me

  • type: pageload

  • unit: ms

  • use live sites: false

imdb (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://m.imdb.com/

  • type: pageload

  • unit: ms

  • use live sites: false

instagram (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.instagram.com

  • type: pageload

  • unit: ms

  • use live sites: false

jianshu (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.jianshu.com/

  • type: pageload

  • unit: ms

  • use live sites: false

microsoft-support (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://support.microsoft.com/en-us

  • type: pageload

  • unit: ms

  • use live sites: false

reddit (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.reddit.com

  • type: pageload

  • unit: ms

  • use live sites: false

stackoverflow (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://stackoverflow.com/

  • type: pageload

  • unit: ms

  • use live sites: false

web-de (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://web.de/magazine/politik/politologe-glaubt-grossen-koalition-herbst-knallen-33563566

  • type: pageload

  • unit: ms

  • use live sites: false

wikipedia (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://en.m.wikipedia.org/wiki/Main_Page

  • type: pageload

  • unit: ms

  • use live sites: false

youtube (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://m.youtube.com

  • type: pageload

  • unit: ms

  • use live sites: false

youtube-watch (BT, GV, FE, RB, CH-M)

  • alert on: fcp, loadtime

  • alert threshold: 2.0

  • apps: geckoview, fenix, refbrow, chrome-m

  • browser cycles: 15

  • expected: pass

  • lower is better: true

  • page cycles: 25

  • page timeout: 60000

  • playback: mitmproxy-android

  • playback pageset manifest: mitm4-pixel2-fennec-{subtest}.manifest

  • playback recordings: mitm4-pixel2-fennec-{subtest}.mp

  • test url: https://www.youtube.com/watch?v=COU5T-Wafa4

  • type: pageload

  • unit: ms

  • use live sites: false

Scenario

Tests that perform a specific action (a scenario), i.e. idle application, idle application in background, etc.

Unittests

These tests aren’t used in standard testing, they are only used in the Raptor unit tests (they are similar to raptor-tp6 tests though).

The methods for calling the tests can be found in the Raptor wiki page.