From a168bb903278b38fb1b0ea87b72df95b0d08ae3f Mon Sep 17 00:00:00 2001 From: Artem Titov Date: Tue, 13 Apr 2021 11:23:59 +0200 Subject: [PATCH] Add index.md documentation page for PC level test framework Bug: webrtc:12675 Change-Id: I779bde07683c33a7cc0dc38033235718e95b12b8 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/214981 Commit-Queue: Artem Titov Reviewed-by: Mirko Bonadei Cr-Commit-Position: refs/heads/master@{#33703} --- g3doc/sitemap.md | 2 +- test/pc/e2e/g3doc/in_test_psnr_plot.png | Bin 0 -> 39236 bytes test/pc/e2e/g3doc/index.md | 223 ++++++++++++++++++++++++ 3 files changed, 224 insertions(+), 1 deletion(-) create mode 100644 test/pc/e2e/g3doc/in_test_psnr_plot.png create mode 100644 test/pc/e2e/g3doc/index.md diff --git a/g3doc/sitemap.md b/g3doc/sitemap.md index 5767269c31..98511dca2a 100644 --- a/g3doc/sitemap.md +++ b/g3doc/sitemap.md @@ -19,7 +19,7 @@ * Stats * Testing * Media Quality and performance - * PeerConnection Framework + * [PeerConnection Framework](/test/pc/e2e/g3doc/index.md) * [Video analyzer](/test/pc/e2e/g3doc/default_video_quality_analyzer.md) * Call framework * Video codecs test framework diff --git a/test/pc/e2e/g3doc/in_test_psnr_plot.png b/test/pc/e2e/g3doc/in_test_psnr_plot.png new file mode 100644 index 0000000000000000000000000000000000000000..3f36725727045773718e1c1da4bdbb018c00ea73 GIT binary patch literal 39236 zcmdqJWmMJO*EYI^4Q!N@?oEiGG}0|8-6AM0-3^i(=@z6rq(kWjkuK?OkS>+3cj5m$ z=RD&)?|8qQZ)cBj554#P#aeUCHRrspc`ZYf6{RsBl0Jk$AegUYBvc^~Bn0@69)=3O z>Gs;)2Y)~vU%pa@fsZH5I2imtiIb$3lbWrGldFNldx)uxt@V3mM+%2c5qm99D0tyOv)IV4GZf;*c(Z$jIVS=Cnce$Ogy9kpHX2k9R13f=;Y+Uzkgp$ zTe)bJYHCTa2mkmX@z(j`eWKm3Bw_&<@;#Tv(cLHD`fQj(z4lkHE5*HqMk052&FgUY!!8nBnj}r$3#HRJ%yQcPFYYfdV=21Cypd~E=v69Tfzi|Y|&`K}goGk@C;pFt65d~k-AcurG zem^|xFsz+{u)+RaMfwP0^9TH~&%eFT)OPF$53jTLtO0Zi*g+3GEeP1@v)`!rM5@cR7f!Y+9X>{Xwo3P784;1 z0fGOO)`8=ETslO9Oz(4pFIq()z|7n!@K730VJgT2P0KAyWIz4$8Z5#v6q8Cua$-ok zKE&H+xj+oT5I8v+Gx4W?F6>?Tn6ua8^5L+w52uDh8wr%h-guWDoDwP@mweW1-YZSI zXIq~9;=+12g;6=fR4#_^f8^oSea&OnDAcWWH=>t;x3H;oH)+JQ9i5_41CMj0SD$j`BSXCHr)*WGeg;2hO$zze zCtGso^88!nzs*+z|Gq~s5@k2>^2g(HmP1HOoWwA_Sa_%8(NjOWaC^fXc(y} z^CQUpicN2DxQjP}s}tq3ss&G@d>%l&od-2|O|N-j2w$G1JDiO2cTvfgsJV1cR`-tc zSNh~wKK|U^t@Uap^KPXwQly9+I^@wU$M*SfEOP!kMZu1L)vogir~Tr`V;HKxknSdW z`9%Me7F$xqdJe()(rmfPI~}y3g~)akWV{<~>2H`s zsjfC#Tw7hcV`*11QT8(fBn5@V-BT&GaO%a7`}sO{MIrm*31f#0A9h!kw?t0p`S`o# zi?;+agt}eq=bfAEx{UL1x9|R;7UDwLOt47)p6RM(EVKUUXtW(!v#{Tle6j@3*CS-%^Wl)E$Gg4WYyEmVSBe9Dz<2Xa`OUj9r76!4q0UaX^8$E32bztxW+R7OD&0sn~9gx4$#X}BRpY7?OHMwAR- zfaT5p_Jz>r!?p;qrWQm6zwxH|+j@u6$|KV)i(RsH%iR1g`d4K_Y$<~bq};@mQP=`5 zhwVCtK2S)abNhv~=kUqj3rbdor>d11i@$es7H^k2eIjQ_fvxNQocTz=x4O&A>K}rk zE8Xr&CJvYgRv*%6+C~_v{E?1UabQzp6Gw}C?;N@K7 z(F9M*@sv+%cyfcsTV8h>Z}*9f^R=b)js}G#Lold1w;kiSxYuB-=sbWpmZwGZityqs z9nM893I65{mFxfN-R}}k#r3k~^!5`+)A3@^(bY|z!6`waTi5o{6*5D4$@kYwR!9)0 zhO?=j;r%G_39h>B#if2S>aJEq zvT{qv54{>5yWTn|MoqalStiNTC-S3M!)!y+x42kJdFjyn5_H)xUk_hWOW-mvvL@7q zJV8HJ1T*^(9sdcVw%gx-BS`W5GMJ{}gjDvSgJXr+?Oprs$;zz2ffO94t?Z^#MmGHL z^h{RO5Q*|;aKY|_V4eBxAr1!WS4$su`f!SfrNNZ2xOKX!hS{Gx4h9#d#Yc0WQt6X|m&J&1**y!|%XSnOb#D3kZl!C~2n zSNV-v*z?Urzmjk#43?hm0mj(qPl-RDRJ#0{R>A4;^If0kR22X4LE!hdD}#kEwj=YS z4@D*H&Feig#-s___g#$iZLmgFKdCHDe_S2q2&dAZTQVKI#Dw@#4f9fMW;n*bwRFp@ zl5ZK_#};(^aVgjLJUVRxt@HhcnUzgfVz!#y`G>)Cc3NE6d50y5-=}X5ag;@S6Z}!G zrKL+=Q9(p^;w`q`iavs#!xT?FBhIMy+#-pTgs}Jb_aVOCNhB+Kn%#f+Os~y@U8h%E zYF)2R(4mN@O-e4BR2OSwQUvXXf4*Bij*i2L3jAX7qP9wZ{3vVTR+sPU@7IbqgB5Sm zWMt(<)f^us>uBq9pljWnpf;Vfcbi$*Y=4XF`a(~BYde*)Id@TyV?HwKbAt&v%GvcH zXnEpt96M&v-K6(6NaS{(@7&A#JOHxCP3G-5H@Kpjv{)K&@8$%DPRk=pivdw0cSwzA zL^)O2NS2d6{+K#7E4=}&Vu&%ur72t@G!=XU4ri;+T{HVQlDJ~*_*TKNn-Z$qKP(IM zOV6CVl!7bgR!-fHE*ee}*|*8*x&#{EO5;GU&AQZKK^lH1yRqmh_+LTfpnF26uWvKN zPO1{=t)nzG6KnaZHe-wH;M$pnzs#}9lK$xvmzm4Gpy>5r^kdemB)t=3`fJCldRN9OgS;zavUKVL zT$w>YJ;WdYCCDovnARsg>Y7)CN9We8Bi?$2c?#Lg%nAvoFW&jK;of_8TNe;sPx9Q3 z3R~CO8-KMM5-LYei8x8B{ODQfV7rVnfcT2C1#{`yiYm4;!H_EKcd^3B+@DDt_gq%o z8+r865ZaHQcWAz;-d-Q7ht}R%jx&mXUC&VJKf^@zw`-hJIwvK<>WOLn<^Q?ZejmCuYAkI^+G3G!(M$2#cDC~H^H0_?+Hz`?JP_5U^!OgNX9f=nxKN(i zf4KC#Xbki)I<s5D4nJ@mdvbzRLxF@dV<+(e_7k)HO*SLC{fmt5De0=AfkF zQVXx+o>|G`NtY?CPhX0wwe5>4k9w{)zUMO@0+`m9fsUFAeXRS zyWzN-nRQ^rPrq!+Bg`fVL3&0X6bS1 ziS5H8(VX=q@yR3?!lV)ffzxJd(@0WQ<05!E(5lZNohcQ)?Ll+$N>;vN3O&mO%G zOXIuYSADY;(YNls0l!a6=!kFoEB4wfV&4`d3tx(Pp~Wd4J_y7xUC$m$gVMDYC=2B}1% zUJCoIsi$~Eis|=-y7Q6^rgIh+g#$1Xl|n}7&Md!YwLPb5Y^>ddB$2*MJLH16D!t^7 z+tkj!B{>YrcGfV{v%iF$Q9kEel&sCWx<~A_V+RYeR*);NrNe=x9^Kn0@{D-Yy7@wv z@0zs!=5jT#n5A2X$}_y2+or3>Kpa#;+iqSnXur>fSMYbSlNL;5W7xWZd%?ci;6W(FY?>JU-m~dD3+KTW+`gnqR_{~%=$HLUYtY%d*Jx}PtVZaI<9#Z>=&s=fM>;p!gy zsciqm0}jg)>wzqvctjF=E-W~^Y-%^VM?~F{BRj11tdx(fM4nwboe4{S2p_7wxM`vE zDJS3OE}nRpQfj2{#768pNLa+>T(3=g3MyoZ#Se$g%5b+4NepNR!*Ag?Hw35~n+E#2 zLm`Lq5)$myo);hHN$Inm-bo2Vjry z;R(JaG$c4U7~{So0Uy-PQiFR)AN=1_*#8527*G-SSD|)_M*n*#@&7djWU+tm zH2nYj%?y3Bv$Ju!NVp>^_ZC^uuF!t~6>Ix7m}0-3mq5eFh~C@Vdpeoak+1Ib)+Sj(YdVbm}r$`5<&g16^u3=9mSFJB_j%P0Ce?@l_q z5A2pTVxgm>PuEzJeNjjW1c+HaiQ~za;gsNlMqu!mgPZeikuCsek0y0&4cZYXS`AK& zUz9SiGKPazR}BYJ`S7??Fo#9%1cfiRXa#@w&<00E1x@YlJEY}jc#~#$Uoxh4j=qJ-bm2H z0&$m4|-HhoG2Ot+oyxy``_|tLP@}W6vxC0t5Z~r<^Vy=f zni}3@sZJ0&9=-lzle^n~^_0hn8v>sCH#sEO&=2gI?Zx8qmf4Lpo<%ogaCa=d!j41Z zj8zwjX{z7-GC(TK$G&JW#{|pZS`kvyEPU8ZbJhQmqQ5M@MI^cN7Tt@@g1s z7oL(J2ylTTInryfO2X~P*n+-&eX?|PbeRPOu>N@Fjw`^e>24^jOK3JOFG4QYmn01-3_Q^6k@9YyTzStZJM4TiDf2)-g9 zAm|w!)bJnyp=bP2US3{LZ?B36NvHwb?bBmCyayj!Vm^KPgs+PULY1?$qa&w-qvOWX znpCIg5#6h~cCeoRqS=eCt@rP`7Vw|m$3NcxPm!zxiU=cQM!;c-JhthcI`7{wW3@%Kvtnk@iLup}2oTi35FdJCr?mpn~qThDY5AllAQ*2-32M#1Sv1pe*OAo#+-(~`UrZwspOLLt+2UUFD1#W zJ8cs?pu;5cdmn>pi@qM=bqcGl8Na5txs^H;UueP0C=#(qk{Uyhz-Co))keEjO^oNf2h z3{ON|U0u%NX3N!qxVSj7W{H+xcsK^ry9Rubi(aJ)lR?sOV)oSnJx-rn&w2OFwBOx> zgM+~PBKjQR?>$f75kT~RR|xYKvo+7@o&Dl@Kdmu==;GF^ikHH5$ecuh)LjldAdWr= zpIr;4Ir@pVl=W>U0yTlC@VKTxD1L-_PSx)!!A4rG5t};OGUyX$ z>*?2HeWKC$`E#OnYh4iqWo2D;_G{~NfMC=t*Nx~5BQlyQd#Bap#$IE)6se{pL;$&u zw;=SPDv8{@NO%90l%qRkQ`=!v+r7@Gymx^tR`Spk&n8_xc6RL+F!vt2ri=PIUPac37U1mWis82wRIx<*Ok?Qr(nUle@-45&J`Bi6_Y ziG}=CvX{6y4m5d^U6Mrsa-P(@>5|4udTni1%>;7gt2X9M%Tq0e!tkHhjfcvNVDtqZGvC|e{K3NfEORn}yd&Q;SSPbU;{q>cF z{ZxvM%HlMZ7~oA^9(W5`Gpv@(ZjUD*xATK33yPy4iZx4jejhKl`yT`UeqKqen)ZpG z=v1@zyWwNIgXjTDpXM>bOMAnJ%!XGdeqSF#^R%lEpX-3N?+*B@+Q{Q8;{8`h_q>Q0 z5}op^%_t#abls(~itFv%j%4LrP%dxr_vbXUYOd5MDi~g;?-q}Tm^?C3hQSi&6mGHE0l2U1me_QcT zPlt4EMsY732l9B2(3n0j8LM7MUY1i`DtgOJGq5%wALDRgSgOEYPmtsaTvS*M66&tEE zFZ`l9$b|8rTt%;jc2cK9bPFHd*BK#HomYkHbIFSjy)|><7Ea<(ArgZ_@N+926UXlp zy)V}Xiw^c7 zYd3ixLHiA2I)M{4x!2LXqS60(62Zz>b~VLS!Rg3S10$X=b#CX>PUiPZ<1e+9LdQtQ zki1;5&gM^l)0do=kckTq_185yFH&&_7oUs*HS8a@7>W@ztTn*7V?0iAnhN5e{Lt5!rdC`_J%Tgi54joc3lJvHZz>=AVDT zNCj|8taf0~e$v?$?1Ignu1_9?M!Yp7VqlXm3ul!qm-YQh6s{q>IjL-h_=ZWD7%5ZT zyhnZjAF_~gsq;^DMn4|jArv9Ot(jMjd(gG5XZB8F^x=e$UIiYBT4qR7-Gy^1T9 zXiXGns{n6aV}|rk_vEKTEUvX2iqD|o!c8%W>Y~}k8T1N;81^aaGgsBhRUcuZ0uue{ zDwua}sR6|9Dk6oTgx|{STU>yezyI&a3M4;$LNciaqvJkc#P7H_Y`oXW+Zx_dTUXJ3 zaK;cnI7xCGCOHqBfKhSo)w}0IqqwjJK@8~CMN}I@=za za8u~Sara{qJY+ErriAV&=d0W?2^vRjzbc{mU!1>~+GjOc5rzePhE}>;(@fRI&^+h$ zFZ_T}WcRr)p%eH-p$SFMoN5%)@@mF5PcXar=YSwQ%o8$MVF1BNkVsVKTWYf5(Deom19JQfLACU@!mE#uu9-2dRq_%p2?|t z@Uh3l?)c8oWiAkjYE*9i&=bW+xqqVYqYQcucP%Q|OJfcRsf;HK5s|&Jk+ITM)5TL{3 zmRtA62yfwfJ7I1~yaj8BJ0K^NR1wxy>=m_>(cDV#b~h5xo)_+%%%WH2w(F}9CXO*G zv4Xb^8K_c%<)t&X8d!&gM{I`=(om3yWjewO_E_r!xv8`TvzV+EiyZv)L&pM3a({Yu zE@%uuK9AQ|yhz1-A)9}+l88%}2k<}#-}rcBT2L}ItSvX|scUl2%6fR_aH607#EPH3 zc7b?puI`Mea-YnpLa2D^geT&IG|?_pK=7uu83vVyRqPV=A@n z5J(A6!-V?o8KVIg_k5rtd=bbH4~)O!NtkDC&^E2pQA#PI-7#1GW+hzT+s!0nbdW~1 z`v`&}5cou#R}M#%fouyql-JY_wd+^jvu5(nM0vzKpQq@P{G;fv_by?6hI+KIs11ps z03n3*JebcsP#>{?fPP@$d=_D0ZgH0Nb+CAPR#u%&T$zR<&15}X|BX1=;;a@bB#~#L zZbmfZFWQ(N=_I=99>fe_%_x(i}R`5j$D@nt%uJMHXjFvhp{%szWdrOz3AD zV*4wk-?QTtTo_K{=m#A7Y0I3&r|m>I><0QC5<&HrZ^Wc{X?sAydY^iXq$N3vl7;MM z2Y7Sf{H?y^nl_kcLQFVuQ9o79=Hy7}(#&e%6XqJmm_eKtY0Ft(T$)<^X4*Pl`Q4kc z={ECwF?9|A?}60Lk1{xUX)P+(*Gmne{E*w%J2(+pdJbp=5214W6KGO7Nkw`t3(V)2&`J0KszHjP_wn}pLU#7#Vp9gKlfcU-vn3O1~bSM zM#)KAFiHtENPSm+3WU6=@k)3&?(*v#8Ecw&NHWN%5#rkS>kHiwlp zv?1BL&MQwrxsV}N`U$z66cvH5g8_YYHs1nina1N+FqVz;L4P#}gc)^SUE8;@CNLBF zSp#;ei<)fp4jDFv>=#WZMe!r-pQzJzpeF};)lVS4#a){eCs+PIRcL#{;hl*F-ZTKs zLB(*`Y!{nYYRlg>62?ANlJfKvR8>`FGasd^swK-Y>1LQbC>L*J_4V$x=greoRj2A7 zDI|Ln;=!AzPHE&6E>Rd5(`UldbPH#qe{l0Ds)ajG29tJaR}g1jck$yO8kqstqdEuA zbg=%Vek-C!%CU3NTwS^3;!p$`w^du^uy7b4id^cw_GhaT*^N-|Ikl7BDWi@+%({(u z9eoax)%Wb-pF!yr8zi|2QvLw_0~fiit8N)*ghn?pJ#)-A`>$oVaW&bFsuV^260dYR zt}Ndkq5Mq|@;0|-hb$wqy;VwwCmQhcF&S6*5(XZsG8OKou|L$-(P95^Y9)9!?ZSW;Ghnr)NlW*>MyB{HV$AD62s8S6L1YkgUWL+wC z;$!7QPQ%}cQi8U1JH(vOC!>>gk~sojX39TZ{$w3*dLcbPS=lI0|G+;l=hrhG0$l~C zruwh=ls-5eS5FM)eaG>9{MuicVsYZ+YK=O>&u1;m4FEHtRb@7Un=#kwBLdL}WpJlt1ul-H_{zjJI<`HukZKC)dh7Y668LU&YIH%#Rkce6-XUvb@#vG zz}vUe8U!94*KKD5>ba6HBkXCIpOM9GnxmlpG`hM(7LV&qzC$9zw*x&*psyYPG$&YCafK}lWdp@b>BGaF58 z+Kw%4`n;K3fnj-@CKs}RiXh{$k#KhA>WUyk3+qGzbS5BV$(4jo@b865-(Gfyr#bX7 zzH6~LKU(2+J6dKh7XXaI3ofqSyU+3QQYY>|dkf7lP!KbWU*|~U_=7O3x#pd&O>i@f zBK4Taq}D~1--8lG6<*gU>!Z`b8YJCyZTzsE^!p4U`Up@(8Jy+vgZ#|UC zN6c+Sd{Dnhi9ybbGcz+YU2XX~;c9QOnT+VIT|0ZZW$oOId^YvQifDz8#U7Ga&U@%P zkWvM+tdB}^jqa>u{PgXut&rNkxKNSKc|yj?V6v6au-&(2XQ=DnkV>C}B;EHe%jPB{ ztv7~!;fXcoRDF|~%T+YKsfL@RG4y{z^Zwc;=c#vY%YJD06h=6u5GiZRIZeym?T4n5 zNgY|Hq-;?vsG1>f7t4lf8^WPv zqpQ{huI^uJ8wh9}K7H0zm1z3yj4pwDpY%}!qegS^g40rRF;kL* zwhFspfvDX?k%siV%RMCk`alskh73DJI`c-&zwd}CFCm8yW&2pmCo|m{fzR1s zO-TKTRPr+KM?3jOW)7#co8|O|0et$6i!3>apt?Rd8q5a;<#My6njhf25~k9F%}q7L z5W5KyclMk5u`q-@cet_@sACqMJ&KC#RgBzcTZxR73|FDmV#bUWZlA2^w0fynXa1BB zllC`;7=bM&a@)_AEYgMrei*bij@wY~4W5+=UYTH8PPKkmZ?K24`)2+`;Wo}&7Z^+j2V_w|NOM~J=V z#Rn(XYL`j`CJXE0+++UN=cZPVx2v08JcT|mv%gX~{0%BW4hW!8bX29-;yZF5tZ{{@ zr!A`k%;(W;hK6{LX$=d_@(Gj2H2xB398eQf{r2g37f+UBfYPAi_h~_w##7DI38sX~ z3dQJq#(z8Bsp;(iCqi>=02c!)sb2fh5*o!RuT3){@>#;&vhGg^jW+L^i`fz#WVRF! zT+#HKYnUJsKb`I6*B_$MPI4h@8xW%j*krW_4lI-neM|F~!Xjyzb(qCXEIzErOGk@_ zE8Og4U}f0#bxu_aMK9KK7Lp0Q?F}>Cd*B)G2k%5e78}j`;*#OH_ojpsLGiBR5*wO# z*%l5JVETpEnzk-`qC_b|nmtWS?TTSZhZ9FC1-L%5Z(_?&F5a$?FObLP=OV&?QlWM_ zV?Y@%L50ooYNoBqO=rM|ouu%M!~yYvlouSQ$mdf-m|XU`oZ@p3f&!Az*;1e{YA)7G zT}l}W9asJ=a6zW&9y6lca|-umg%?=-gt#JXDE|;C_sdEd4Q&3vCtuBo89!S3gW$PPWcVF(@e0(~8VK?I1T&WlG3O zM`Y@q*{*7=7>q6?Q8UDlNoC{1LbIpy-bg$TJl&tBoD6lbh1hzkq55tOC7}%^1WPWr5djwc)X%qyiyZ$!lnC*k?x?&z%J9gT_1;QoPUXJF*y=wjGE-iv zuC~f;d>-J*pW~Fki<%6);&dZzvAe_i^?u5Lo_7i-~`z=@1Pm8=;rt3?Z>#Ji7O}E?FQ{xLevHuOf>%;R>t$1D%`pL6obk{ zI~3qVyoHKK5KaXC`+HOO5hPPH*!SdzZG9+k>YlVaQ#y>U3AEG^GCPsDJz*T_A?XMr zi>Aa*>4es@29KhinEL5rMTOU!flb#rhp!uPi{DcTow#cjodva}Q}-@*A1!ZO5pEby z8!U;toQnBC89tSdM$!)GL2_R5@RyoL`gNyltO`05CBV$>URCzvZ$X^xucZXq8%}Dy zna(YIFu405;QB0O9!IjxhMh6KknKB@n7PihIVbIi&GyKY3$T}GM~_&93Di%~8SxAo zO4L+~yeNtTDrGkQ1h9_&P7OWnATGX8ekM8#dR|WXIY!f2(@FWGOkzSAJWRHao_3y` zJI6m<$r)kG?i^u%T&So#C_U&^w~w<3LM9j-nvYiQUk9h>0(|3p>yLsR{*bA>0IszV zgDi$ZDoH2R1JexJOgYIl()vV{_XYuQrG-xOO7~A)5Ggk*p}X&N=kE?&AZ-eGXoeDs z-@ZQ|WsQ9meW%agmh+1t6zrG-8H$dX-aZtBA~F=w41ALWC$H3?@E6;JIbhScvZo?Z z%q`t~A|B}y3}wEputEdRC!T-$pjLsT1PrVk?y1`U4@v(0H!MIJu+eVCH3mA9!?;g8 z12ZLxd_tY1l>6ttLw>UIVTITep8IF-prZRU|+-1k`2@* zltR?{<{MDI^*f>(%Ig+=f6fvAe?!JY>ueb#Fe@5ojI&o&9VRLVb2^8EG}SEF*J8K` zoXhtUWXLr0_te|QHwZIUY&-_KDO7HEZYA_PuBXuP)rNq|*lK&*fZM$NEso@rp#%c% z^dj8uLt^$vVE$0j9pndXS%3USH59U}^ra9KnvREyQmDDFDxtK09+hQ}VEQP31jM#x zw!_SF>}NvTSbxhWyU^F(dI7H7pT%)}Mc#3T>U@6~Bi+M>lgn^#f0_xjXEW>=5XIsa zOK35F+Otue7~yvkv;6%E&TYo3za)}mVxp4O7;vmC&J1n^vY1!&Qs4R7BiB7C!RNzS zS2hoQe7^_;5m^w)UJ``wZEav)08tbVyknH99TRMKvk&hFC~If-J8Zw}e0Lmr*0z0N z&i=uQzu`YvP^VS^BbVuZnxIzsY=ya7{f6!_;Py9hIi6C%;P`ggNO==#=`cYM9e*Xij&@({BDmyG>v^eRSL%3ujx^4{|zK#DnGVTBO9 z2qQMJJf}l$w-d@xUz*a%X4rW~YsP`@cH1y^99H=V0vYyf!JZOqrK~PZFA(~*v%uwM zEnYbPgy}O+^zUgt`4QtpQM-|BNg&w$^J`FG)e$5K#3L48I1Dta3XzI)WTL25n%5f4Vyax+ zfeK8arCZSpJ)L+U26?*(2$tsNmdJ4i4pFoSn(j};rkzVZ+=PEviBV%AP0fWE3Ag*j zy4J<7c^Z!a6I8cKWobY7XZHbQN}kvkJzzUVms|2aV=FY=jvZD{>2?5GqQamfFvt5A z0FkygrfC6Gw?{z>Enk4Tt$R$RVqt#Xb96ViXNy-?k)qwV!y=G%dCD4|_<8(_1o61B zGaS&PePna?6^j$Qj$tR`3=V=o1kx1}Ks+KsQhV#l=|XAICF%3d|7kh7|DV7UzJn=E z?)u}EQ}J{CIciU}QQsl-X~!Vr)pU}M>?V&LW=2}A&EMHG`{Jsp-?)C}vi>(j4YVZp zh?*q(eSf^cM&Dfi^ zLFZgb{q<|3`8xaG)<8~K{JOkrwKL5A&mfP{Qj3?&tL;l;-%m=xcaZmmlPkO8o@|zw3#d%kl}?@DKm~){g#N z*`syS4;6VOxUNc6fvjhft2bcm*5*bD<$Qn$WFa5H<_x_o$}S9TDF1(uA?_E@@$ZXc z1Y!`Kgt=b>N&dhQR)FwBOiBvXrs=qgC4-9H5PJJNq4i$XBmB!5=#})HKgcjOm%CsBqQt|e8GW01=Oh`gd?@eNXsGF(AY;2$E_1+W}PwWE|La}^e^ zF*Wy#VnkSB_u;axqJY%||3qNBJ}NgA@ztYk3Bh{J4d{6%*2H?zfl0KN2A73iBTrx6Ddv{^1DsA<+&FYbG5mp7^xf@Ns@?K3 z(ry?*x5<((XKH1vG399Was#8%KSjmMxsrNNZWYe-p07tbuAHI}P!%Y~BxDQ$lt~fz z5E`X%_B3SOS2MJfquxl10+dP1Rk9>vH{;-3k70h zmYLiE|G}AoCN`uOQIw%f3&Xf{Mg@R>d0O-k>g={?=aoWqn+X$xT>f%K;YSLvEjt+Adju>S0vM(ol?I? zk@LWX#eYnZ|BW%p@UrW%DdyljRO_is%zZ&CrU1CEbrW0)XH3`BXOCxwE+PYC(r*xR zTB=cigU+(FvrKe~d>XF&g1AU3`mbCXi5S3TBB0%bOG!ZsU{FI6Zi$VH(MZseQSUDY zh$!HA0oM_^l^i_&1vs4EUg>+d4DS7_vfc&^(j4>8la^V8TIB>w>;%W5ic^IF5Z&4; z=$kJ*hn&Cz5reQ$C4@DaU`WzLEzqRJMkeo?|4Q9*1G)YeSEHBN@yBmA7JRx`KR3zS z%}k+DuW8RO4oI0iZP8ArK6;}^_9?Ub}p8%%gph#2j@2#TxN918D4B2i(m{R9*$&X-R zmY8^uJJ$2~>_MjwxtaywV89Bz`qQT|&5q$D{6>O`M68z87Z7T`J*H5_(GH?Yz~UW}=^e@fC+hi7Ae=SOSV~Ot5B$;J$0z>g5gV#CIuk%7^$yO zz;;G_R><-dAdR4r7#Eu9#RAhhN6LOnrQpPIe~U9{d)1Vwrp!Pcc{}oghE|FgY{8k# zkR*Ae-yIPeXl~==lAzx*`l32A^1;i><6K-04>RJz@OGe!PSX-Q{}_YJRf3lr3mRcB z6iO*NK1B{0?3+UK$_&7%kGLlkT(3P-E9*lV!u;Uo+5w>e%Xsq#NvcRVhvCC!33M@I zzG7maA+9V>TDjh8F9oLPE(Rytslg>FySt;m6goSF|Hz8X8U^*Cr=N0=VTQ#0Q{yry zhhRIrIQWvd<+co+Kt=SipUii{@VlN5ztr~oZ6KE%t%@WSb4Gm%nucp!2vskD6fn6T z@jMQWQ;#NjcrTP#K0$Ln<9xHQ*VTMz)zNoY`~b4{XUuM@D(hjnY%4I0Nk}#@#b$27 zAN+^;iWblrcKKXN(0>Q$%+4K|S8FLy%6m0HF^gA9eSeeL& zOhG?CnkX=7&F20`wX85PW!tC4x=3O}ICQGmVmQ$mxKY6Mfu3em57J{seBP!4Q(-dKsZ+{+SFJ-kWEYV+xe4p*?k{|?OwW;wu!9!_@weaY z7j+5QT1z^Ip@HpK4^&;H|5RO@R}G8zeyaMRn=Lcrm|bIKgdND;3;hM9BCLh+r|Q|7^gfNs@t2KWau3{Y^1rCMh>p?k26b&)+{znS z*046(UostiGfa>1CI+)c*j>WVd$Dde6mtn3(lYx5g65hZ%%A_30=KboJSEAPw`cJe zB>*TgdHetqNPSQ;4@wP$#$kOCG`;pXP%P&ss$mGH6EwhA8Qu!$eEp-$Il5-;_RDlF zn$T9wyj(3Q>XGovxepSX* zs0t32C6q|Sswvl`jnyrNVDUUPYw>qhFX0TU-A0#2L)gDR>prINH$ZBa3?mf7fexhb zU_+Gk^rCcZn+e2&umZ-mC3JNuZ5sEvL5GF0Np%N{kG5d-63A8}GL^3U;NU>*xMUw} z5_Byr&KR4Lv264pJgwhwomp5F0omIjLYcLbMwY`V0k1RYKoz{&t0d-g$o15bltg*< z?Ag(@QIr8tKf0|15FL$3Q>}o0+iSbL@psx=#HfEqxdwnUamn8IABdCp3*+||5k~!Q~(e-TivhoCfi|Aj0Uz zMt#rFM6-i+GPDrvJEi*pGlkpwJCKAbx-B&{@L#WB`?LWyrrZ1#vvw8i{*b}=<3b`N zG2EbB$3P8TTC>_jFu+V=3*wPk6Gc{NT=@m~0DH-9yhAwmUa_uB_Z z$Yb7eJBtI75>$0zuav#wWpSYHAGE?i(gCq%`WLG*Y+-Smwp4FHWPvUkv@QQoT!w=rEBg?k?{Yt*D~@t6ePgN7h&! zxr3zrrkVw4sbwHF`ZHLbI*r;uuXQZ`S5!jOO zmU90+G_QfyT$cp5P%d@izu;-+B*B=dozW{~O060Sh~3oRtVCWqPLn8wn4BHapKHew z)?_X~{CwZCegFZ=-JRd{$Y@w_5fPDRQF2eq{~SALFOMHFi811k1L=uOHBc^Zh#pm4 zS0xIXI9HP0Z`u~$;78JbI|uO|8U3!qcLx=`$nCk&H8>X0napKrceE@9^%vt!Qw*mN zAOtPy`!gufX=w|)yXD%|vbT5Oqy~b)!m%u6#Qc3?l|)g($oixm#l5S)j>e5*J)P_- zah%@8UVU9|^AaR3@o$fh^82N}@=0{4kD%5$ZL4BX@Z$r&X9maX4;pq^z$prW^YgW+ z<8q&yM|e*ae}ob|XR})top%~jC@3sc@#xKkmy)wS9ogBYVTsa9r`Bp~prV=Ri&{}O z2f4t#+_sD72`0!F%T%v30^J&m9AwAEi_WJ$fWtJ$_mKwxMz*BLa0S-ZYf$mKyYr=c z{(jZ;cU12Dq}+#M_x&eXr*W7?PQA1Z;fNNfz*S#`)l$zCmRPTt&D3YGG=r zl|s_KI{R}g@!E+JS2EY+zp8SW8h)1r!Bhj=W&kEA6<(_H;_+s&39Uwnf6WgSjV@@+ zwbCydD@uB!fmO_nCR-^SEkov@N>Pyrt-3-gu&~e3b1=S>S=1W!)ob~S3G}0&)wvwN zmv3aJX(&PEAD0tKD%P7W^xfLB@B4REcItQiD&m^ z-Ei4r&^^!v07EUvbfubXNd4w(`nV%_FqOv^^}N^F0yw^NezkNLnPSs`9wqo2CVYJ) z4z%dx_Zog(w-uDUgjWxrkJ(984Ca3ojm{6nF_#jPLh|iN<=exC9yhwWF0Wv>ph4cc zY6?e1jl6O1-4pnx~dO>g|1<3g5lou#6Q@m0)&mWbQ5Koah4K zf6sd)aMCBV#I8h)nJ_bzi!3@H)jsemSb+Bno_LHJGq#r7yuJlFN>(Js~8 zD4RHx@1m}E_>d4$aVBw}%0$MP?Uu8V6v=jc)FA5ED>=w1qnn}$$glckv;Q-`A*TJ> zC$2+qtb304r)i!2DTV#RNBZN>$yd&=M6T?#@;d#v#AEHtwyoeo*ezAJYsrTT^(}-1 zlx&1-L7~F=5DrmThlD4Cd!dCUo7N1|rU(`7G@W;n&aAX@tcSC%kn*ZSf}vik$Y%&H zKW-Blp@VWv=shYI%P`E<*-3TvX;c0HOH3Y|k z3gG!;-AR<*|KoGcHBX^MKm zpISP!W#aqU4oe!U4sBCdtyrDEruEu&-eU)>&aW?ls!X@q+Tb;!6tdW-y)w1$Z|VHB z&gSoH`x6rrzdXy&-#<@Mj*g3^*YFD;g~BFgtRrTk^k#0m^0#}GfSpga-yD6qtK}i8 z5c3zmi$6jv zoz$bRZgS0Rtbfs3`{3b`J5=SzRa}2i1L=1ka=1f7PI+HiaIp8Iuh!C^q7xA2$MMW? z%o*&snV|EoU9ptWyr(QMSV*`)+@w=^ohV|R?_eq{C>YOLCTo$2+SHv5qZ?F~ULTRM z;@{rMNi$x4&W}{9xaWpkS&LmlL=aMz2^V+WcTyu z2d%95czAefPEQ$DKyG-noc+hti!O$}!+DFT+o_QR(Jb$}R2#?}fBSN}^>baH3o&}- zs?3tkg}Cqh5@BXOr8DC1Lw+1eOHUu`+56$)bbt8*%JuT^-{r%eVV36| z58_cxM;e3b6p8-kdxJ;cQ|-^5q7yWvlaD|nhQBV~o=+4=4*U&2?`#)lcO+Sh%~;*1 zlVQ2eceCowlha>HhU`mAiw6t*m7GqdHTp~X+O$mB7H0*qo$ma7j()d#9S?Am-d+IChRjUu?R-YHPcNS&Kkwt`WZ#JNMEV0hE z%p5_g{q7T~)r<)B0DMKk2!2p5X!#QG5@U_=*{0Vq+!4;)E*_z`|}=IPgK`|}Nra-pa2P8|x8e zCK{nv%wiL=uD7e#n_p2Ja4txo@(?XZH=etgTs|{c*M&B;xH4v6SmkQP_^Q zhAV`wH9UU7D$B`HC2sXqWt=# zM*6URbo3E=c6BjSw%%+L=K3}FN3E^E!qohL>o9oKH`wtpDwcKye*D{|(| z{!$T9ON&Hzr;eCnvl13!BaBz4$q#EJfbu z!eAgfM^xAmm7gTr)t|HTKYn!JaLYJ$C-bbaEfyJ$7=$0asa+V9UJRYCtCl(QK zB(7Zj;Af!Wp7 zyI@{ppw-O?R!ck>i2}*OU9$Wgw`JBXgARz5E9+;K%vJg)H;eQ2?nQ|##IxCLW&xl1pK{s-Rr z?{Mnq{ITH>{>7lS=~SsX83Rqn>_oZIGoO1Q5+oGA-)bkt-6BCxE zGha3on`|yNiOb633i%1Hb& zTqIz{KO>U#sFq$if%HaEMe-<>Y8DzL46%Hb&?l{Y%05+d_J^~<)!n5Br%i@bXoNy(_BM_ftN$jUeEQRVg-%Hr^IH|HWv z`f~HlOm;sVIPTODn?(~OOB`s?qnR{rb(V`TU5}=Ge0QSvp>5~3>442$!aY9Ra*DfaafvQeyXfhuk zy|VH|qrFq*E_1U>-^|BHN5jtlUHz~EWVQZqO5{~+<5304;hRz0*p9w2c9$^x`KB?$ z_Hoi!Q>`NtHsbSXmNw7|d5{>RWQ=nc5C25A>4=jn+n0fY@P{c`mkOUK4X`*8>py$D zNppVimV-&mevY5w#}fvFzmTTq*7lPQ^F~MisN6x8oZ~vG&$pROwz%5gLk_V*KF`p7{DJcd8U4X1K(JZ=b*_Bw_LJqtv9yPoIQE zM5u_jgp+#<&MB^N>uK@EN~;sjgC`#1NM1fos)4WX=CC*+960`PGf_%TVrFOG2*x!d zBO0L{3*^bE3QDr_>df;;tVT}ks}ZSV7mHAQyJ7f6H)ZSWM_9^~e$fb2xOxW!Y$9oD zYdbv4k~v)7PiTL~$HzsIk_=|2*9}CR++Y#3bzc2W-5XP0;@eO$Bn z0T%A|OgBYC&z*>#2(3dy+<*T3;gYr`g|%;)_&NJdpO^2-@Md0a4(j;k=*pA)>i-V< zJ6OMviNw9VmB;s@D}ilqo_g|Ax`d3L?7he+DG^pvuHJ-#v>rJfD(srb-jGhWk0EsRqKu1uOUs$|Am>>=x}qF+ z_B##V0O7yuKovh9AKxgpmjCQqyY=5oCNWxKKf6YDlcIi5dfE+e`S7=e>~Q1IwbM$D z?asg!osVQvYb8Q=0LaCeAy$ZpcD2~T&^4S}-4he@HTkQq1u0BO&?Ro`EqZrep6A_9 zwS6-ja<<|`6GF+CY+6hU$#Hh z8h4fKc*~LIj6F@p22$`DN8E9&tlX(_z5 zICiI7>Xxyl6ycji*vt8t-qzSrhORAWFMjT$BzaH2-i}2XMX7+Mxp`J#3%YsevkUhp z+nFo0zT);hJ93JRLXVN*xVF>k-G(z4J?VQ`$;KzK@vZL#l5>;;s7c7a14w`Qvf0SU z$m(98@xAL(XRf?)^3lCQ@^A?iB9+W0;Gp@9-_|i{N;he^ZzXpWeP=u}u#eFLqgw|IY+6{Tzd-a=&bA@OcmTKnZZU5I1R*Z1u^{8~1pgwk}VJPYwdY`)wx=}~__`ikUkb9H4~@H?GcD~m=AFX4_k zmX8Gq_|__kf0SJb9(=R5+ucHco})fvtFW*%S!6R;S$GeRLYtx8@s@};w?BCMbV;+K z@y&7PpbKF5`uw5$XPV0R9-5z_mk`*$49h^a<;mb(A+k@5ZirM zhAfhh5PY~ctCm45BKo=A_8mwnvYpyB^;nbGZN$guui!Jz6LI&IjE5~lk1&&*e0Sfp zP$os_l4rRE9Qj4!hxDGm6jt3UD9hbDOkHl=*(3atfq}OyJn?IJzR7nZ5**w_Gsz+? z-=QI{`K{PMHEXuk_u_II-!EL~fMJX0^&6?_s<}Da!$ei#;5yhVGkh&hh@8^nfR9Ay z#K%3eIR!(TW*W3P}!0wGfgWDbFE2o5X z>46Vbr_Ts;k;&nsqqgv7M!VT+?A_F$FsRFyxck~nfvOV8~Xd?MLtCq0J{w&_~XrJuh0Qj2@W{{EmIuOXk!J9Vnvhgd9f%*dwOe= zDYfIKXl}z$$Lf{6iW{9D>ySC~`Eaer5R26DyAPo+8Jn582=ax3gtjf51lSyD^3#B( zYXSTW>(X?8$ElWeF`r4l%Y!l=9dUV<RCz>FDB82s=s&LsV&t zV-UFbnY z9{}%8A}RaNw5}{njE#*^0Lau?CYF?lQ&CZ+5x|Zz@!;0b>nSOAKdoh7LnXk1ubV`2O)JDeU~8AJXC=wqM!X+Fw|k%t)H#z1sT6c{=7*(;q@=cbGq zX%{!}&9cq=e0(W`okkfqIzgo#cCb+0(9q6FbLZZj@e*vbpO=373Z6Q3iV2YJ&LeZq z6X(qbW%z76`uFDrx|-kQgaPWn`M>3Eb4W6TAtJ;|DD(JE3bbtbYIc*gZ6h6r(C*!A zp`oGO&z@;II*KF-?o0ao*|pZKpEf)^JWI(>B4g(XS|%nYqVgt&k-r_ttBCRb^X<8U zuJ7XVnl#5V%lj>MlkcUPD_pPh(|DIt2+Q;chhk;4jgr`*(Bt6UYIu7~7ZeuCxb_N> zTK}xC%A9(yLNBPsXJKJMLr?!OL+ZFRwq@Mb>fnYm23v{`Fs2`1tTeoIErfsi`25>r z-#Eo0DnqQK0?mYlZ#(-bk%w#SG@`nW*SdzoYFmo)yc3(l_T=xNPLkMI+ZdFvYs=|k zSJ#GgCL2m@H`A%+Xx*Tz8~Eqg{L0@3X)z}|8-KZYFXWbcHapBAVGx`HlodBA-6wGd zi{3jiJNGhF$iLwi_eua!J^kXP95X;!oNT?WD8@gQZ5($Vv0@ira>Y?R&6 zQRDrC`uSn+^T$b|&CybF+R4iC%w*%U#5LmhJFW?e-(FOTIMxW7ft(&0Cv+u!P-f<_ zW^e1uuQWUo5mp*_V1X{pCMuDeBkk_P=fctQ zRE-3pNRPq|^d!b;B?=o*C6-9l4%EN{x68v*mu!Gx_ZSyo6N{k0XXb9ME7glCj-S}~ zZ_nHk_H-HVfSY&+V5Sv1UdmYWWk^0Y!6Sd#L>;84Q9OCx4{tE>8>1t~V++fosZSbW zlvjgqJ2Z&YofzY5+i;)c8UEg5fUF!nBz{0s^CPjd>^^gOjaNl$^fjF|?Uvq6pH(jr zD4O!`b^lv3t6gs$vW`DZzhUwuKkX@rr1Nd^lap3#M`5a{@2Tz zr0;Ky_G0w*skyAc^VM;#j;fl+pKa_7Xf0C;+@G&~mvjKz_DqL9KK0Y|40I8P@13D1 zFy#OKl(o&bGy!?y$9vBSuSg$>e0?pX+i5?YzbC-gM!f9Doq&|43%xa8rf96UTkflA zyeY@|11^vjC*yMSbJge_POcn}-()=y4y@B87T8z|zoz@qmCk#wJSPAvJ9$ThGZ)Vu z+;=6yiV|5Ya*>zXGip-ce_7YBX2X*9(fM`+Tq}wHpSci)b5#?F zj%jF~^{`k!lf)(#{z7e4lA1~2qRshrc9-YZ+Q7wZ$sG7{v$G<8YLhIvBDV1j`QCyw z-7h7Ios4=bO^R8I!+3?u;>C=u*O1lmT3m;-M!T?ZIO}{=NoJvK5yAUGIbd|p2%(lF zzjF@Dau|Py*_f+KZhW5ImM?GAs{ZW*2`9PYUH>mvjeL3gmsdB(K7Nd{z8}=~?j0|v zk1iB^O8S-5yzmhGg9bE@&wvh4;W$!t-d}^dgetcr$ zspj$`@}kG7(q&Yaih+FN__$ja~E8*5Nx*m7JFZR_&$^TDZUI6I30 zY5AzM zE6b;y%{7vLQ0{O^`skpr4`#ir+;Ce2lYmMq(Figti)QcMy;PevZN<7GyS;%#$UV^0 zkJx_#szDdf%O0TI%cMu zi3hP3es@39CS^sQ{(v;>gqtSao%!dE&@*v_cZqnc(@jCup!y-zfgW;N)x9RrzM`7({qtk^0`P-#u3m*1wq1nKp*-1oE z;Hk#z9`uMN6yKF`nyP8&^456HgsolXAZ)HL;=&FVFNZE*CNTdv>q>)?kc=u*N{ikx z_KUyQ1sH8~BXSNz7dt4oKG(Z?%#E_EK&$x9p%fYpGpggO-l6c~X8a%O+~d*;G(OmJz4dU0f z0u}vRj?Ept-j6j87pFNKH8z{i&f*n2sZ>&SqGhBbrCj&$_DOT4yOI4BF6P+^S4-Si z-1-i@1Xa-9-&Xa=2c2YRN1FqrjE@S_kQ|zJA7l-XPrXw+_&dR=jJWq2?^Aur^Q{~e z2UNEl0T_#pYW(-qoUCn@`)_j2i#Ip8H;5=l@tsn8wSiP4_KEuT?oXAp7&Y=sOQXFl zaYVXKu=&FEw0w)AN9^izgts?}jkA?XzFHV%O>&fN9nm$LUMWd>;QGFGbE)F5!L`3$ zSL8EtaPP*9o3k${XiK*^KW@g``Ul?nI|CLhTcstfY$_$>7EB~21Kn-y%)DpgvUOhc z?A7O6ew7e&!G*~!JmbBAid#FIMh7BR_wioof1XwqTbH6kt>Y3CUFKrFtxfH`@4<)B+!KMjw=JOw zQW(k%7L2lrR*baVK-%*sgzluG<>g27BaO`p{qAS<)YvfV2o!u|lZ1DtQ&7t-|JJp7Qxk=J4hxKbmlWTOd>_Ntd z3Mtpe#eUu_O|ICeHz#LBZnTQ7DJUELbh*BBM91v_baC&v6ATXFJKfJ^_pe+t?PjCX zyF)~?Zp7Ule$Nx(#Qpja?<{ZQ7VYv59#+$hEy}^1GUwGU=xy)1eGB&~>f_o#{f_cq z2<#T|ZMts1kFdFq7Bk85_ITOo8yl~!+vUE+Rj(ei#2p?}5aHWXBD&#zmPb}<;fbEB zTDNy^j6I)58AikjL4XzR;c2ozxvRRvgpAI(nJ}IG6{N}6vY$ab#djAuA=?(;Y*;%jj5xnS3l$)YK<${GKPa# zv$dc8wY&$%*cv9hlx-J_IBLjDbxZiOaa6utq*^#bZ)K@6%uUF-itlB%)(;4Z+h@UA ziP0uBOl!4@MEb<&7Po)b)Hb?YiAnPg=alj|E1j!&Q3nxgo8G!OyusBkh-6kxvmwf? z4NE&7<#&_$te5~rt|69b)YxB5z{!1He4AkDkfn;mZ;5id{3Dlxvf8)U-I0+LuiW0N zag(#`grB@A)$u!^gkp18k9?WE5cbh=skP3*vSz|ji`=}os9lwp%p?j_FBmrFusdLSu~c7lh4Ye#7^X1 z(!SfdZMCY^a7#?{W&YuunXx&?nBQ*fD%Z^H$B%u`-H0H~QC2K8!@fEV_?s%f#;!;y zZuEYwm7^Q2?ak!;#%%-xR zX2H36cI&P9Xwb(~8w-3OWYaC*{#mXYZ##{b9Njnn;ag@#9DghOu>+c%zH7tviep#l zPv`Q)lC>o9FLtrrd)^HPHCY}YN`-;$C0i9QkB zItdHZRW;UB*|f^=ttE?w6j+vitd5fl-DHFK@L5Ko6!Q+P{vnqvH>O+e4Yc5@JK8ng z?cMEe^&eMed28}nyPaKdG2io_R;$fBeQS4EYS&!6pDP%kC_@=w;tN*3D@=$HQ-_Ru zKnF<+3vXy^YxDN=iz+YQ5%hB=4nnxJrt3m9Gz<(;24Wp^Bh8|*wI0#uqInhGo&H*{ zRHnq#{tJ5p~lbW%S+M1)k>?z zmlCX%SLfsm-n_UKf0_yzhpA1`cU|fc7+g7+vvUWCh2hVu=svm?H+0i5=K!*h6r}TO z3!5@_*(AxiJkPV!_(bHJ?|7c0EOYj|Ylyfd>V-}@y^ckG+Xf$f=ghuu%< z|MLQQ&$mTLq!a89DJ(ge8B#-PbnJ{MEZpif+z=kLG|fG79$A{t@9KAfEVBOeYi4%V z?$oI(4`=K+vbCn0%AS!`4%hnIR<}}nA306zRD+wiFwn zb{wq`?{^eVA6Jav}4}4zQFzGuH)w9eg#moBSepFm+H~sY5cbU3$yX$sL+ssNQ z!=@Q;vK;SJRa04mw|Ty;NL`FFy4LC6z7@cMH1oY~SJsUDsPv52N??c18RL5IW9I_` z8x_ozc4HHPu&7KUF6K`^a8mXPebUDYQCA(R#(x)}!v)fxCD~iD{bMHkFr^&Hr zIohN!mqrhI9E{Dx=Y>*XR*yQ1Jk>4SHw)Z5BN#yZS=ckpotJQ>l%Fo6xkPd-bg`4jj> ztyW2OgGx?s`#Y`i5`4q~RxULw-$G<-wfC>A{Mmo>QbrqbN7312%#Pk2HQ}$f;93m8cL@1Y z5((i2Y;qxyl*c!V6LJqAws_<|{{Me$qQ}4A6V3%CYg;wOb+F${%ILqni(d9Q-leJY>pdV+n(PTGaD;*X{Z@RtMZs!XUzlaph8@ZbT9mJ2@ug=DLp+Q@bE z1Th>tb}akPC6Cts=J!6TGTpV>yxSI%t^f0>*$TDyEgO4#Zz`S0^A%^KrRS0+=@olk z8vpUXzcEB0exZ^0dgkjf-%m7nQ``oFLJMhgsuBk=2jX_#?7Mnoh4Jj&M?pFf|747f ziobghOM?G+J2iXt&aE#ExW1|2MkcnmSf7(%_f29PeYP21Yei;d?CV!LD0fDh1N8Y* z9idm~L(?j7AgxcI(iI*|Y_D@4C}Mm%j*B6aD!G&K9sCKOg7^OEnu-H~feHvKQCTkK zC(f~qk_>gR%iQOFt*q$qXNbUW#>BC1BNPyBbv*pa{tuytATD)vkEi8yzyYz#b`Zvx z#5nr_S>)q$+n+N-fw2DztKV7*Pknd*|Em=ea_@D%g{i;)9;Gv%!}`mQ>GQ`=+S}Vp zJ3OMlrsPKpcK?97It4*ep_P59#+?!r3rwIHT?eE{M9M^Eeeo5FNJMs?XcF;rO;tMm ziT+L1i3DW6N;Ogp;A2y=vcy9>A3f6JPwgzeaibvM+xwb3cRG*^^ZxbSN%HtzdgPNX zEA38DKK<(m1j)qt0*L#gzz?1>*kvC;#JM1v0fJIA0m2|+= zH0`yWc_Q>H6k*J)tXuuWQ_F2y9@K=c2Vo-`I6L3yEzqJH0}pZF@1OOk`u9Ql)5(%^ z9ZdV!@UTLDT&b90vU~yQDINZ3vmo@OzYrEiQm-DsJ1`r%>sj6!pO@a}R;Lg!$A=f- zab^|$seQgr>c$layI;mI3=$LynF=glKA%z2Ja|wT5(A|9KqWkeE2Knvz5|l(Vn!_! zs8{B3Bp|>3H`HM7srPA0A_-V%CXh_h{xjx*XV2z?!GlmL=e3=#VR^qj^c6%+0?tnn zW7o7axmFevIeNdGfof+@(+HS3}p2 zi;9cwAV$hkIV+3Z<*}37Ku4z)Wld3n=hw)LjEp8RTT7u)Tljz@r~M*VmCVc>hFm0` z&7i$oQonj&D}*P6tdf+67lrZ!795y))D!9P<(%1ru>|;KkZ5{XIeV?%C%avHNwiS~ zrc1lFZ!a}XPxg3$_VVb`C%~udPfrkoN_@H{rV>=t)P$e~nZD1x`IL|?iTasy1k4R9 ziinEl>u2tSObUNu$6ukQgY5Sf#68_s1v`W=`E##+{Ud7e&; z79D9ky(xxC@DTtF-i{mGqAZZpiKa7TV7Y-!c#yAfi#G$po# zX)I=`Q}LhwaEJPADT^3UK(%YvE_&WQ2GfTtIAWqJ51X5FVN}8eT#Ar=K&6DUuY#2o z!+NKur$?bcBH_b_COGr?SF454Q(2BME`MrZMBQTD7{8vSgHYSEHAZ@Er>ezH-=T1R~oCm+BceH)@U~{o~DbIq!XDT**mClMp87Rsr#*3OT8`LbOFFSh;CzCYrA=0`h@ zW?tgn7kaN3iNm(%&kb=(*$-Y4kg{qzqdQ$ALWscfG2tQtBUO3{(}UKVCNUycA~!M7 zk}lvC>hj0;zlEU?f}Jau_8AvBdHXQ-rEuy)NQD%%0Wtq?p5x!g2meRMk{s)nKGAgv z-0c=AsY9t>1yzICDlO|GtgH9FlW+2$jmMo`nO%dt89u4ojR6i0VbPwZ%45~mItB(8 zva_>AWbUudH4`>}OJ?@B7gL)AL&DG|@QVD9h%fhcenL+-CtzE@vU*|m*{qs#h>8^=bB&l&SP+pakaJT z%)5Ffb#--UwTQtkDLFZbIKp|b+$A!K*0*QV0 zM{L~Oacz;CI@;RQ@68roQab?2z%6vY*nRRP6_fx3`F}=KRERp z35MPxUDm;O?>YtsI8QzJbtBLq|FA92^UAmLf88ToQe#U(^hzm(ge9JqnU&8^T|Zxd z7e`bwQIH@=h3v_{boJBZB%voKlHEWygbMp4T>FJ2&dO0{9b2K4B`z$E;7^d=?&LcZqtwzds3HllDg>>va| zKN0FbDBBswd&h3yNV>QF^IdEbhASS*o<~HTaW>~&{i%fIUAol+4!~aCK0f4XzuC9) zGap(6eH;0(yenukO5_y?aTUhI%r7jo!Oz*bxWvG|2ZmSEtxjQSYpc;*gT(uuNWNt&N&KFl>z?f8AA0n;o9Mc|`> zWk``>a0tCBpIQhPF<<7v$~YaO=h#cZ5U;#;ljq~--|@RIF4pY=K(>Wcx}MYPQ9B7F zogxfx5<=VGKWC3ucAa<-G9)6SprizaHJJ-7Ply6ac$Y1E_Hg{ulMx!SwY}t&MnoCH z<^6aY&`$GU+EetMJ95J#BZqw7L;PS5?Q|eI+}`pSl07s1nwZH&gz#_cPhA2?%J|i= zMI+i4drAJmAH0eDS6_PX2M3h&yLr63%aD|mG{3ye`R&ImqkVRe1AvfZN~p4Kq$%k+k4e(upvYNUt*O864d}v$D{D;y8N5S#+CCJ^&_mR z%J2TBLgtWyN(WUd1X4g?r{>pwDGVt5fmQqX+?e}5347wc5_`pwFpWvQS- zK&|MLWXqoh8;T4jfhPe=WLB5HfB!x|Kc+E!KbRBqpfJ!!epMbfTJq^hCCwXZa47_{ z<>j>#wX>8sOn0oS^YTbYEvx(b$`W(KKm%nj1sM@om0(?-r0+<%e*IGL@*KkC4RFme zU;N-~iNv5_=n935j7&rD^3L5n4B&J6N{qJ>dS2wOHR11&s9t};tEi~h5VR!W>+4&# z#+9zO@%;Jou+0cluE;218v5=2x0;6w`-q|d8YZR~7yx2`6JTW<`Z-Guy!vuxu=XS? zhopH}ZS8Kt@q_&d%goHQs9F{gFxo(23*ZjR-=urFZE^8T2)Z5-6}4nRL7V9n1|`M`jHxJ8eqLNRxI6q#kB!!@imaj0N{xJGWGNfp;_SLR7y z#hq3SVG!~YdMwmZaS~5H`SGLO%OYI^Y7k5X2B|d8j*cic7KAiD`LQGVu|ZC4@AnWw zL}2Y34Q#cRo~0tp;uV8A`_2m}eHQ=5Ren%41uTbcWe&S@}uV~$s2k|tC_WDggC0hhO{SR&k z(X4<%=xYu*wfJ>_3HU)gT^wHe6cMktgs%5uc%W|!ahQpkUIb$DK>`v9$CxkBesjoT z63=`p<2$KC5YY%b?Ck8&jVpU)jm3EOHd;Oq7X%p}xYU2l2Zav4-?D-HnbcNDggFP` zB9KoZQ?m9Yflk1oi&ZC@g~i2b5FGCELluD6=+c0VHhzrE!V>WhQLo+#T(GCwxRLwZ zIh1jaJM2)lwg{=~1V2nDFo+W~?n!j7N5lHwTMpXT4V&mVB=@_yxiwCeJaISfTcRJ3 z{1#Byw2<|d^%xZ$Ep3fM7vnt)4YYTehP>5h>N!857d%#F)dq3`eEs^;u|&##*WJYU zEA+Xua%O43NI?ucD|j@p-RSUPW@4NnQP}}E!g;Db<4K-@m6hWj|7a*?%cWa z^M{u6iBpEoE-nrbw&J91AQ6ZXc<-R@&2ac&F~e&#(0UWq=MCros=>SwtO8cK_U$26 z?8o`}`LM`HN>d55QceB*!>v6%40^@4+i%dp&VUCiFKh<$7Gl=yO=vG$E%-mgk_$EWT`kXd2X$U+ph^t?Yi)OgyZ!lvatY)3_f zWC$)z@M5_^)7uLjL|Kb(mmNWTq2N7AI+(~~_K`XjPc$}QbDq5gp@-}N4UHWk&xjFV zkjB~r-IKz=C5)oB^z-}s!Rb|-*~A(!%aIXu7y}F@5yk&5>Ia!dJce0W(c_Q4CH>P9 zHXlqEgYGMM=2vW)fl<{c>>EYpV<-5FNN`S9J(M<*(*Zj#5WVT3itZ{gE+mnT-}Pu~ z^7}fE(i|qV&F@$GfNdn>HLP$;hzt=Bp!fv#F7h$QN>{JLWFmMu03=T$5eE#02nU54 ztAu!)p5%>Nm#tnV$}REk3AEh#4h2~DX9U4w*1|>(XYEkJF}YET(1XBgKr)TQc*49_f2kZ!zl4!PeYT@lq z?0Q|6Ln9_CItF6vA=o(pL4s2vjye=Mu5W9%zp1c@1VxZtF>PYZ@_BkXo;dMD$nx^# z%O?gwX?mj9a&pSqjj-Umc@ACaLL5=!_?QJ9`*>p$5?D{wJIZDOhyaOEffVWd+*jCF zf67KYy>pk-;|mQHlY!_&h;&j1!ng9_Lm`kl`y>@1*+;DM$#JR6kELOez$cAjJ#>tW zx zsn$}805*t67#=uGE}|%rFZL zZ5a6!NRYeZl<;Ha3BQ5L2g(ZElPuknB&$AL?@ zUdw>Z2&|csnR)DsiUtzDQ07m7!8ON(V)U($1}g!VyfFz+oov z1w`ttpb(IzAnq{a+&LoduduA6U7j21fH;pL@8yJiTy@j%@bEnR^900n%&cuVR}O>#V>aiUUh0`4$|x1JH_TOu zQ;($PUT^36v_KJcJPj=3v(JH#i-HKSjma_(FsuA>rv>@+HidnE*W|22e>piiIRVpy z8Yi0aU=yZTRJ-RP;6$zh^1`iBQkBpCc1H;1f!lOqtfq-*?JSOy1T~}9p1egP?`Sze z}3qRtqAQw~B2hOqqmy@YGm z_%yY&(zB_K5{nAFUA)Xn)#iAbmx)gyY-MGIcFM(c0FiP8D62=GU65wb(be6)EQ7K* zc7aW_+n)QQG>BEqKzvyZCMNmm$KJ0?fh+S{P&r&?%6>24TkcA!w%BEebemsxrHRmN z0P2i^Q1;kca+Rrs&+1Q4)Vy)^aH^o>g%CL4ejq7zARz7aZE#r=5)8V`DYZllg2WCR z8=H`nR8Ps3Sv(wH%di^n**2#pp4M+-->}w!#Ly!`0V&D5P$DMAJ4q4)AwlYe(|CUbR3kk*gL=7 zILqka3E^GRL-!i6zqni?O@4;?&p=p&=SB!afKIfxXagX;^mqC1ADhNYNHx(5sD;E5 zRu#MdAp)Kn09nGXTtELu5&OdYq%9&P01ss%?3nntJ5A3iJtG|=A+g}^*t(~_8y&&} zbzytYgTCT9_at!b@}mj;tTg$3wb$Q5ZcU1R$QDsOx-PV(e^0_2c{ZCpj*^ z6^czxnw{p0^U#9k1iViS(Hso_ON2;wy+$Pd9LiBU7ZlZz5Z$sB5dautLeni1IB!d8 z1##^S2O&H4njH(iZ(qyn4@Mp02h9j}3G5R`3or)wpkkhb^mzRgdBaR4qL;FTA55Ew zvk^%?B4h!4jb~7y#+ky-Afh)OehkT-k_PM~8p@GqFIN!*n!)oR(vsylT2=L=>R9>* zh9i%Dh)5VveM4Y~ve>Oi^b#Nn5f9OrBJT3Y4qxn9qQw|7Vwot?0_{hnjJUwh4~8Fs z3NVQn=MEC=ARSjGipw_DS)Q2lE)=5i_V#WCVO3@oN3^{l7^%t0Nnia&h#@R1?>;-5 zK%S zQhWFf&c$tUb9I^Z`E@@c^N^nLCS6YyDx3fUdK|1@h433FGzJA+F#CyO`q@|WxhJ93 zmifK)1QJY~gpz-4@{nc*QSq0L4NsI`5p(EK83l`Th|gICLngrDY!&PwSvBqL{$NjsiTQtPNHvj&R(YUlSwRUmr1Q1BHlRYR7r&tG3y9 zi^qIu;mNpM4ABZ53yj&~1Y?t!WVTcXOtpP!uQiNTSJ$RQ9(jC5SYc$2FMt)`^}Ni( z#YLm6tXw!iTe4DDG844FvxV0zu4@?EW_^8~Xt!W>KZqS3w`)iBJrl2?MoC1!XeGmr za7EE-7692EReHGaxixg?RA$aDW)@H{MT27_z_NYa8FT;v2)3%LS}g()P7+X!$Md84 zMB&Tp$B;>Zy93W#{r!cGru;Gx1mP($6^j$L#OlO}Ti~<^Wj?l+23o|G`#mr&JV-=? z`WceymX^D`Mdc@K3fq%iOjpI69|LG=|h^&nr${)sKJ|g+5JM2S4vOc$D zMrws;Ad1}yKJa?T_syV8iDC&j1$%hr@!^Sk_f$f#!%*X zMFWK?1U~=+lI{7Cq4H<~vJ({I!;edgh;%SKMzMAI6YGR0ri;y}6_rT*w*~Nuh$&=% zifDzI{%+lrj9)|)i}KX3aboSfsp{&?L`RB3M`;jnp!yCXq1}t4>eh9X1IHJ+eWGIY z>>9!pG!^9DiNV>^S&f(XQzD^6;5>K&O&nZ3DQk-%r3fP1Gx_*3CeS@|o9*M};fW`P zgX3L*O#>I(R(tYQ4N%SE+z2ls#_sO!RstXZh{fyQ;~Ga==_ji^gPAsB2{J#w)N`ry zrypH=^TLO!j&W#geL>}kPOh+@r@g7zRlflIvc%mMW<)$_Ohw%l)~J6U#V zg7ioA=BrP+->1_tl79inf+SOof0X^ttJtl z%jOR5HF6wGJS-lO^{61nVB?EV&(H3hD_k@^BUdwS)bCq?kY8BmlD|4<5pKt zSH~{-)c)*Q3*Eb=R{A|3V6MT&$cmdTRXe4?tQZ!QTr{> zqEt*wENR_vqMz*HpV z$O`fF2RU)Iv-%^P4U=1ihLe0$Moxa|USD$!-1 zj*gC*YRkB$T%Yw5sZWmn%_l6LE(fZ}MA|CRQ%AaSt$qJUhd_QK`ie$JQy+Tt_t%eY z)SEV)7N&Yq5XtxA$IANZ-n;en=>t^it#9A*U{oHOD4J(%?2z8R?6`*~(Yw*A!Fz0j zv$J!}g9lYgm(46J;%?j!yBEC7sPe)gAb=AGoRnf#cm1CzN2r|rL!lc#dTRw zQq4i8&dkV=R#c4JW|ZqH+wmR8?;34*Utq4bkr5LDDmF4z%emF)pFjQ4fK%n%@7)E{ zA!T})KKS>~IjfdXeoVg%e=&8W$&QVav#_j;7I|hZZS74U^G5xf$Id7%KsivZL{MwlBYoHmZ+G(?40|DCigkl-PaTc?X)J0>o_9iHxRwf;>UW8}w=A2-i#l*#T($v4>H?}ci>x?a& zL{nKT($Ub2Wyi-_d~%O%rENVt==WqZpcbtGmbHET)Dfa@Dg+(KV3PNRuI+bm*@pGs z&AW@;_at|QQ+mOx@2B-YovGnaNYXfeexG?p=jYGjK+?T$-emOBdo6~hw6`C)c=2Lh zVIgJf*oP~9t`|nU_T9QKN=ZpsQdRZn{d;Rmsd4;T`i^6%JMU=FY`IW`8#PC6wLbjL zJ<}{HEaKJLCgUUcmjLRoMHZ4A7ZVf52C8k$EiA&HepFgu7N1aHJa_IKArjIwHgIn# z=y5(mTlWdWD^nLzH{sD^-KDF@V{p7mP>}FZ?|Op!j=n@ndd3ad1*39^#?YP@Sm=hp=Q+q+lc=M>@}(n literal 0 HcmV?d00001 diff --git a/test/pc/e2e/g3doc/index.md b/test/pc/e2e/g3doc/index.md new file mode 100644 index 0000000000..d676476ddc --- /dev/null +++ b/test/pc/e2e/g3doc/index.md @@ -0,0 +1,223 @@ + + +# PeerConnection Level Framework + +## API + +* [Fixture][1] +* [Fixture factory function][2] + +## Documentation + +The PeerConnection level framework is designed for end-to-end media quality +testing through the PeerConnection level public API. The framework uses the +*Unified plan* API to generate offers/answers during the signaling phase. The +framework also wraps the video encoder/decoder and inject it into +*`webrtc::PeerConnection`* to measure video quality, performing 1:1 frames +matching between captured and rendered frames without any extra requirements to +input video. For audio quality evaluation the standard `GetStats()` API from +PeerConnection is used. + +The framework API is located in the namespace *`webrtc::webrtc_pc_e2e`*. + +### Supported features + +* Single or bidirectional media in the call +* RTC Event log dump per peer +* AEC dump per peer +* Compatible with *`webrtc::TimeController`* for both real and simulated time +* Media + * AV sync +* Video + * Any amount of video tracks both from caller and callee sides + * Input video from + * Video generator + * Specified file + * Any instance of *`webrtc::test::FrameGeneratorInterface`* + * Dumping of captured/rendered video into file + * Screen sharing + * Vp8 simulcast from caller side + * Vp9 SVC from caller side + * Choosing of video codec (name and parameters), having multiple codecs + negotiated to support codec-switching testing. + * FEC (ULP or Flex) + * Forced codec overshooting (for encoder overshoot emulation on some + mobile devices, when hardware encoder can overshoot target bitrate) +* Audio + * Up to 1 audio track both from caller and callee sides + * Generated audio + * Audio from specified file + * Dumping of captured/rendered audio into file + * Parameterizing of `cricket::AudioOptions` + * Echo emulation +* Injection of various WebRTC components into underlying + *`webrtc::PeerConnection`* or *`webrtc::PeerConnectionFactory`*. You can see + the full list [here][11] +* Scheduling of events, that can happen during the test, for example: + * Changes in network configuration + * User statistics measurements + * Custom defined actions +* User defined statistics reporting via + *`webrtc::webrtc_pc_e2e::PeerConnectionE2EQualityTestFixture::QualityMetricsReporter`* + interface + +## Exported metrics + +### General + +* *`_connected`* - peer successfully established connection to + remote side +* *`cpu_usage`* - CPU usage excluding video analyzer +* *`audio_ahead_ms`* - Used to estimate how much audio and video is out of + sync when the two tracks were from the same source. Stats are polled + periodically during a call. The metric represents how much earlier was audio + played out on average over the call. If, during a stats poll, video is + ahead, then audio_ahead_ms will be equal to 0 for this poll. +* *`video_ahead_ms`* - Used to estimate how much audio and video is out of + sync when the two tracks were from the same source. Stats are polled + periodically during a call. The metric represents how much earlier was video + played out on average over the call. If, during a stats poll, audio is + ahead, then video_ahead_ms will be equal to 0 for this poll. + +### Video + +See documentation for +[*`DefaultVideoQualityAnalyzer`*](default_video_quality_analyzer.md#exported-metrics) + +### Audio + +* *`accelerate_rate`* - when playout is sped up, this counter is increased by + the difference between the number of samples received and the number of + samples played out. If speedup is achieved by removing samples, this will be + the count of samples removed. Rate is calculated as difference between + nearby samples divided on sample interval. +* *`expand_rate`* - the total number of samples that are concealed samples + over time. A concealed sample is a sample that was replaced with synthesized + samples generated locally before being played out. Examples of samples that + have to be concealed are samples from lost packets or samples from packets + that arrive too late to be played out +* *`speech_expand_rate`* - the total number of samples that are concealed + samples minus the total number of concealed samples inserted that are + "silent" over time. Playing out silent samples results in silence or comfort + noise. +* *`preemptive_rate`* - when playout is slowed down, this counter is increased + by the difference between the number of samples received and the number of + samples played out. If playout is slowed down by inserting samples, this + will be the number of inserted samples. Rate is calculated as difference + between nearby samples divided on sample interval. +* *`average_jitter_buffer_delay_ms`* - average size of NetEQ jitter buffer. +* *`preferred_buffer_size_ms`* - preferred size of NetEQ jitter buffer. +* *`visqol_mos`* - proxy for audio quality itself. +* *`asdm_samples`* - measure of how much acceleration/deceleration was in the + signal. +* *`word_error_rate`* - measure of how intelligible the audio was (percent of + words that could not be recognized in output audio). + +### Network + +* *`bytes_sent`* - represents the total number of payload bytes sent on this + PeerConnection, i.e., not including headers or padding +* *`packets_sent`* - represents the total number of packets sent over this + PeerConnection’s transports. +* *`average_send_rate`* - average send rate calculated on bytes_sent divided + by test duration. +* *`payload_bytes_sent`* - total number of bytes sent for all SSRC plus total + number of RTP header and padding bytes sent for all SSRC. This does not + include the size of transport layer headers such as IP or UDP. +* *`sent_packets_loss`* - packets_sent minus corresponding packets_received. +* *`bytes_received`* - represents the total number of bytes received on this + PeerConnection, i.e., not including headers or padding. +* *`packets_received`* - represents the total number of packets received on + this PeerConnection’s transports. +* *`average_receive_rate`* - average receive rate calculated on bytes_received + divided by test duration. +* *`payload_bytes_received`* - total number of bytes received for all SSRC + plus total number of RTP header and padding bytes received for all SSRC. + This does not include the size of transport layer headers such as IP or UDP. + +### Framework stability + +* *`frames_in_flight`* - amount of frames that were captured but wasn't seen + on receiver in the way that also all frames after also weren't seen on + receiver. +* *`bytes_discarded_no_receiver`* - total number of bytes that were received + on network interfaces related to the peer, but destination port was closed. +* *`packets_discarded_no_receiver`* - total number of packets that were + received on network interfaces related to the peer, but destination port was + closed. + +## Examples + +Examples can be found in + +* [peer_connection_e2e_smoke_test.cc][3] +* [pc_full_stack_tests.cc][4] + +## Stats plotting + +### Description + +Stats plotting provides ability to plot statistic collected during the test. +Right now it is used in PeerConnection level framework and give ability to see +how video quality metrics changed during test execution. + +### Usage + +To make any metrics plottable you need: + +1. Collect metric data with [SamplesStatsCounter][5] which internally will + store all intermediate points and timestamps when these points were added. +2. Then you need to report collected data with + [`webrtc::test::PrintResult(...)`][6]. By using these method you will also + specify name of the plottable metric. + +After these steps it will be possible to export your metric for plotting. There +are several options how you can do this: + +1. Use [`webrtc::TestMain::Create()`][7] as `main` function implementation, for + example use [`test/test_main.cc`][8] as `main` function for your test. + + In such case your binary will have flag `--plot`, where you can provide a + list of metrics, that you want to plot or specify `all` to plot all + available metrics. + + If `--plot` is specified, the binary will output metrics data into `stdout`. + Then you need to pipe this `stdout` into python plotter script + [`rtc_tools/metrics_plotter.py`][9], which will plot data. + + Examples: + + ```shell + $ ./out/Default/test_support_unittests \ + --gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \ + --nologs \ + --plot=all \ + | python rtc_tools/metrics_plotter.py + ``` + + ```shell + $ ./out/Default/test_support_unittests \ + --gtest_filter=PeerConnectionE2EQualityTestSmokeTest.Svc \ + --nologs \ + --plot=psnr,ssim \ + | python rtc_tools/metrics_plotter.py + ``` + + Example chart: ![PSNR changes during the test](in_test_psnr_plot.png) + +2. Use API from [`test/testsupport/perf_test.h`][10] directly by invoking + `webrtc::test::PrintPlottableResults(const std::vector& + desired_graphs)` to print plottable metrics to stdout. Then as in previous + option you need to pipe result into plotter script. + +[1]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39 +[2]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/create_peerconnection_quality_test_fixture.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39 +[3]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/pc/e2e/peer_connection_e2e_smoke_test.cc;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39 +[4]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/video/pc_full_stack_tests.cc;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39 +[5]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/numerics/samples_stats_counter.h;drc=cbe6e8a2589a925d4c91a2ac2c69201f03de9c39 +[6]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/testsupport/perf_test.h;l=86;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7 +[7]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/test_main_lib.h;l=23;drc=bcb42f1e4be136c390986a40d9d5cb3ad0de260b +[8]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/test_main.cc;drc=bcb42f1e4be136c390986a40d9d5cb3ad0de260b +[9]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/rtc_tools/metrics_plotter.py;drc=8cc6695652307929edfc877cd64b75cd9ec2d615 +[10]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/test/testsupport/perf_test.h;l=105;drc=0710b401b1e5b500b8e84946fb657656ba1b58b7 +[11]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/test/peerconnection_quality_test_fixture.h;l=272;drc=484acf27231d931dbc99aedce85bc27e06486b96