nick, great mail. Thanks.
It sounds like you're asking if we care more about network performance of
firefox or network performance of standalone necko.
While both are interesting, I feel strongly that's a no brainer - I care
about firefox a lot more than the necko library. You cite a bunch of
reasons (and I can cite more) of where the interaction matters a lot.
Efforts to replicate that interaction are, as you mention, likely going to
be fragile and inaccurate and in the end don't really save anything.. all
we end up doing is trading the testing of something that we care about
(firefox content/layout) for something we don't (stoneridge docshell
hopefully 792438 illustrates strongly how much the interaction between
content and necko matters to overall performance. I don't tihnk there is
any way of getting around this.
I acknowledge the obvious downside is that a regression (or improvement) in
this test is not necessarily a necko regression - and that creates bugs
that are harder to chase (even when they are necko bugs).. Some thought on
how to make diagnosing these things easier from people better schooled in
testing would be welcome (maybe the answer to subsequently add a layer of
necko only tests that can be used as diagnostics but not tracked as
metrics), but I'm pretty confident what we need to be optimizing for are
real use cases on realistic networks and right now firefox doesn't have any
of that at all. So that's what I would want to prioritize.
Even tp5 numbers, using existing talos drivers, would be a huge step
forward if we actually measured them on a variety of networks.
Let's use 792438 as an example. This work was basically blocked on
stoneridge until it became undeniable we had to do something sooner. From
anecdotal tests I am more than convinced that doing that work is the right
thing to do - but how do we use stoneridge to pursue that in a more
disciplined manner? What are the missing pieces to do that meaningfully
compared to what we have in stoneridge now? I don't think it is a big list,
but all of them are critical to having a metric that is worth optimizing
for on that particular use case:
1. Firefox's loading pattern.. including the speculative stylesheet
2. a better metric than "page loaded" (i.e. page usable)
3. a network model with configurable IW sizes
4. a network that models queue sizes for induced latency and loss rather
than just stochastic jitter and loss.
And of course without push-to-stoneridge there is no way to do that kind
metric driven development.
I'm very concerned about boiling the ocean and ending up with nothing. But
in many cases (pinterest, 136, cnn, etc..) those things all matter greatly
to perceived performance.