Add index.md documentation page for PC level test framework
Bug: webrtc:12675 Change-Id: I779bde07683c33a7cc0dc38033235718e95b12b8 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/214981 Commit-Queue: Artem Titov <titovartem@webrtc.org> Reviewed-by: Mirko Bonadei <mbonadei@webrtc.org> Cr-Commit-Position: refs/heads/master@{#33703}
This commit is contained in:
Artem Titov
Commit Bot
parent
696cea0843
commit
a168bb9032
@ -19,7 +19,7 @@
|
||||
* Stats
|
||||
* Testing
|
||||
* Media Quality and performance
|
||||
* PeerConnection Framework
|
||||
* [PeerConnection Framework](/test/pc/e2e/g3doc/index.md)
|
||||
* [Video analyzer](/test/pc/e2e/g3doc/default_video_quality_analyzer.md)
|
||||
* Call framework
|
||||
* Video codecs test framework
|
||||
|
||||
BIN
test/pc/e2e/g3doc/in_test_psnr_plot.png
Normal file
BIN
test/pc/e2e/g3doc/in_test_psnr_plot.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
223
test/pc/e2e/g3doc/index.md
Normal file
223
test/pc/e2e/g3doc/index.md
Normal file
@ -0,0 +1,223 @@
|
||||
<?% config.freshness.reviewed = '2021-04-12' %?>
|
||||
|
||||
# PeerConnection Level Framework
|
||||
|
||||
## API
|
||||
|
||||
* [Fixture][1]
|
||||
* [Fixture factory function][2]
|
||||
|
||||
## Documentation
|
||||
|
||||
The PeerConnection level framework is designed for end-to-end media quality
|
||||
testing through the PeerConnection level public API. The framework uses the
|
||||
*Unified plan* API to generate offers/answers during the signaling phase. The
|
||||
framework also wraps the video encoder/decoder and inject it into
|
||||
*`webrtc::PeerConnection`* to measure video quality, performing 1:1 frames
|
||||
matching between captured and rendered frames without any extra requirements to
|
||||
input video. For audio quality evaluation the standard `GetStats()` API from
|
||||
PeerConnection is used.
|
||||
|
||||
The framework API is located in the namespace *`webrtc::webrtc_pc_e2e`*.
|
||||
|
||||
### Supported features
|
||||
|
||||
* Single or bidirectional media in the call
|
||||
* RTC Event log dump per peer
|
||||
* AEC dump per peer
|
||||
* Compatible with *`webrtc::TimeController`* for both real and simulated time
|
||||
* Media
|
||||
* AV sync
|
||||
* Video
|
||||
* Any amount of video tracks both from caller and callee sides
|
||||
* Input video from
|
||||
* Video generator
|
||||
* Specified file
|
||||
* Any instance of *`webrtc::test::FrameGeneratorInterface`*
|
||||
* Dumping of captured/rendered video into file
|
||||
* Screen sharing
|
||||
* Vp8 simulcast from caller side
|
||||
* Vp9 SVC from caller side
|
||||
* Choosing of video codec (name and parameters), having multiple codecs
|
||||
negotiated to support codec-switching testing.
|
||||
* FEC (ULP or Flex)
|
||||
* Forced codec overshooting (for encoder overshoot emulation on some
|
||||
mobile devices, when hardware encoder can overshoot target bitrate)
|
||||
* Audio
|
||||
* Up to 1 audio track both from caller and callee sides
|
||||
* Generated audio
|
||||
* Audio from specified file
|
||||
* Dumping of captured/rendered audio into file
|
||||
* Parameterizing of `cricket::AudioOptions`
|
||||
* Echo emulation
|
||||
* Injection of various WebRTC components into underlying
|
||||
*`webrtc::PeerConnection`* or *`webrtc::PeerConnectionFactory`*. You can see
|
||||
the full list [here][11]
|
||||
* Scheduling of events, that can happen during the test, for example:
|
||||
* Changes in network configuration
|
||||
* User statistics measurements
|
||||
* Custom defined actions
|
||||
* User defined statistics reporting via
|
||||
*`webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTestFixture::QualityMetricsReporter`*
|
||||
interface
|
||||
|
||||
## Exported metrics
|
||||
|
||||
### General
|
||||
|
||||
* *`<peer_name>_connected`* - peer successfully established connection to
|
||||
remote side
|
||||
* *`cpu_usage`* - CPU usage excluding video analyzer
|
||||
* *`audio_ahead_ms`* - Used to estimate how much audio and video is out of
|
||||
sync when the two tracks were from the same source. Stats are polled
|
||||
periodically during a call. The metric represents how much earlier was audio
|
||||
played out on average over the call. If, during a stats poll, video is
|
||||
ahead, then audio_ahead_ms will be equal to 0 for this poll.
|
||||
* *`video_ahead_ms`* - Used to estimate how much audio and video is out of
|
||||
sync when the two tracks were from the same source. Stats are polled
|
||||
periodically during a call. The metric represents how much earlier was video
|
||||
played out on average over the call. If, during a stats poll, audio is
|
||||
ahead, then video_ahead_ms will be equal to 0 for this poll.
|
||||
|
||||
### Video
|
||||
|
||||
See documentation for
|
||||
[*`DefaultVideoQualityAnalyzer`*](default_video_quality_analyzer.md#exported-metrics)
|
||||
|
||||
### Audio
|
||||
|
||||
* *`accelerate_rate`* - when playout is sped up, this counter is increased by
|
||||
the difference between the number of samples received and the number of
|
||||
samples played out. If speedup is achieved by removing samples, this will be
|
||||
the count of samples removed. Rate is calculated as difference between
|
||||
nearby samples divided on sample interval.
|
||||
* *`expand_rate`* - the total number of samples that are concealed samples
|
||||
over time. A concealed sample is a sample that was replaced with synthesized
|
||||
samples generated locally before being played out. Examples of samples that
|
||||
have to be concealed are samples from lost packets or samples from packets
|
||||
that arrive too late to be played out
|
||||
* *`speech_expand_rate`* - the total number of samples that are concealed
|
||||
samples minus the total number of concealed samples inserted that are
|
||||
"silent" over time. Playing out silent samples results in silence or comfort
|
||||
noise.
|
||||
* *`preemptive_rate`* - when playout is slowed down, this counter is increased
|
||||
by the difference between the number of samples received and the number of
|
||||
samples played out. If playout is slowed down by inserting samples, this
|
||||
will be the number of inserted samples. Rate is calculated as difference
|
||||
between nearby samples divided on sample interval.
|
||||
* *`average_jitter_buffer_delay_ms`* - average size of NetEQ jitter buffer.
|
||||
* *`preferred_buffer_size_ms`* - preferred size of NetEQ jitter buffer.
|
||||
* *`visqol_mos`* - proxy for audio quality itself.
|
||||
* *`asdm_samples`* - measure of how much acceleration/deceleration was in the
|
||||
signal.
|
||||
* *`word_error_rate`* - measure of how intelligible the audio was (percent of
|
||||
words that could not be recognized in output audio).
|
||||
|
||||
### Network
|
||||
|
||||
* *`bytes_sent`* - represents the total number of payload bytes sent on this
|
||||
PeerConnection, i.e., not including headers or padding
|
||||
* *`packets_sent`* - represents the total number of packets sent over this
|
||||
PeerConnection’s transports.
|
||||
* *`average_send_rate`* - average send rate calculated on bytes_sent divided
|
||||
by test duration.
|
||||
* *`payload_bytes_sent`* - total number of bytes sent for all SSRC plus total
|
||||
number of RTP header and padding bytes sent for all SSRC. This does not
|
||||
include the size of transport layer headers such as IP or UDP.
|
||||
* *`sent_packets_loss`* - packets_sent minus corresponding packets_received.
|
||||
* *`bytes_received`* - represents the total number of bytes received on this
|
||||
PeerConnection, i.e., not including headers or padding.
|
||||
* *`packets_received`* - represents the total number of packets received on
|
||||
this PeerConnection’s transports.
|
||||
* *`average_receive_rate`* - average receive rate calculated on bytes_received
|
||||
divided by test duration.
|
||||
* *`payload_bytes_received`* - total number of bytes received for all SSRC
|
||||
plus total number of RTP header and padding bytes received for all SSRC.
|
||||
This does not include the size of transport layer headers such as IP or UDP.
|
||||
|
||||
### Framework stability
|
||||
|
||||
* *`frames_in_flight`* - amount of frames that were captured but wasn't seen
|
||||
on receiver in the way that also all frames after also weren't seen on
|
||||
receiver.
|
||||
* *`bytes_discarded_no_receiver`* - total number of bytes that were received
|
||||
on network interfaces related to the peer, but destination port was closed.
|
||||
* *`packets_discarded_no_receiver`* - total number of packets that were
|
||||
received on network interfaces related to the peer, but destination port was
|
||||
closed.
|
||||
|
||||
## Examples
|
||||
|
||||
Examples can be found in
|
||||
|
||||
* [peer_connection_e2e_smoke_test.cc][3]
|
||||
* [pc_full_stack_tests.cc][4]
|
||||
|
||||
## Stats plotting
|
||||
|
||||
### Description
|
||||
|
||||
Stats plotting provides ability to plot statistic collected during the test.
|
||||
Right now it is used in PeerConnection level framework and give ability to see
|
||||
how video quality metrics changed during test execution.
|
||||
|
||||
### Usage
|
||||
|
||||
To make any metrics plottable you need:
|
||||
|
||||
1. Collect metric data with [SamplesStatsCounter][5] which internally will
|
||||
store all intermediate points and timestamps when these points were added.
|
||||
2. Then you need to report collected data with
|
||||
[`webrtc::test::PrintResult(...)`][6]. By using these method you will also
|
||||
specify name of the plottable metric.
|
||||
|
||||
After these steps it will be possible to export your metric for plotting. There
|
||||
are several options how you can do this:
|
||||
|
||||
1. Use [`webrtc::TestMain::Create()`][7] as `main` function implementation, for
|
||||
example use [`test/test_main.cc`][8] as `main` function for your test.
|
||||
|
||||
In such case your binary will have flag `--plot`, where you can provide a
|
||||
list of metrics, that you want to plot or specify `all` to plot all
|
||||
available metrics.
|
||||
|
||||
If `--plot` is specified, the binary will output metrics data into `stdout`.
|
||||
Then you need to pipe this `stdout` into python plotter script
|
||||
[`rtc_tools/metrics_plotter.py`][9], which will plot data.
|
||||
|
||||
Examples:
|
||||
|
||||
```shell
|
||||
$ ./out/Default/test_support_unittests \
|
||||
--gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \
|
||||
--nologs \
|
||||
--plot=all \
|
||||
| python rtc_tools/metrics_plotter.py
|
||||
```
|
||||
|
||||
```shell
|
||||
$ ./out/Default/test_support_unittests \
|
||||
--gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \
|
||||
--nologs \
|
||||
--plot=psnr,ssim \
|
||||
| python rtc_tools/metrics_plotter.py
|
||||
```
|
||||
|
||||
Example chart: 
|
||||
|
||||
2. Use API from [`test/testsupport/perf_test.h`][10] directly by invoking
|
||||
`webrtc::test::PrintPlottableResults(const std::vector<std::string>&
|
||||
desired_graphs)` to print plottable metrics to stdout. Then as in previous
|
||||
option you need to pipe result into plotter script.
|
||||
|
||||
[1]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
|
||||
[2]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/create_peerconnection_quality_test_fixture.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
|
||||
[3]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/peer_connection_e2e_smoke_test.cc;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
|
||||
[4]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/video/pc_full_stack_tests.cc;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
|
||||
[5]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/numerics/samples_stats_counter.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39
|
||||
[6]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/testsupport/perf_test.h;l=86;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7
|
||||
[7]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/test_main_lib.h;l=23;drc=bcb42f1e4be136c390986a40d9d5cb3ad0de260b
|
||||
[8]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/test_main.cc;drc=bcb42f1e4be136c390986a40d9d5cb3ad0de260b
|
||||
[9]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/rtc_tools/metrics_plotter.py;drc=8cc6695652307929edfc877cd64b75cd9ec2d615
|
||||
[10]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/testsupport/perf_test.h;l=105;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7
|
||||
[11]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=272;drc=484acf27231d931dbc99aedce85bc27e06486b96
|
||||
Loading…
x
Reference in New Issue
Block a user