This replaces references to the ResourceAdaptationProcessor with
references to its interface. This would make it possible to have
alternative implementations or inject fake/mock implementations for
testing.
The VideoStreamAdapter is still responsible for constructing the
ResourceAdaptationProcessor, but beyond construction it is agnostic
towards the implementation.
With this CL, I claim https://crbug.com/webrtc/11222 complete.
TBR=ilnik@webrtc.org
Bug: webrtc:11222
Change-Id: I6e7a73bf1d0b5e97bc694f66180a747b27ffb018
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174160
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31148}
This CL adds miscellaneous unit tests for the
ResourceAdaptationProcessor, the VideoSourceRestrictions comparators and
the VideoStreamInputStateProvider.
Bug: webrtc:11172
Change-Id: If95f69644aaf2b43e3b19d5729bedef0b438c77b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174101
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31147}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The ResourceListenerResponse was used to make the QualityScaler
not clear QP samples and instead increase its frequency of checking for
QP under certain circumstances, see enum description:
https://webrtc.googlesource.com/src.git/+/c70b1028d47c1aee4892545190cd66e97d09cd55/call/adaptation/resource.h#33
Because the QualityScaler depends on whether and how adaptation
happened it should listen to adaptation happening.
This CL moves the logic that was previously in VideoStreamAdapter closer
to the QualityScaler: QualityScalerResource::OnAdaptationApplied().
This would allow the VideoStreamAdapter to operate on a separate task
queue in the future, with no dependencies on any stream-specific
resources that might operate on other task queues.
Bug: webrtc:11172, webrtc:11521
Change-Id: I07971a8a5fab5715f4ccb7d2c63f1b92bd47170f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173090
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31143}
This CL breaks up the CheckQp() operation into several steps managed
by the inner helper class CheckQpTask, making responding to high or
low QP an asynchronous operation. Why? Reconfiguring the stream in
response to QP overuse will in the future be handled on a separate
task queue. See Call-Level Adaptation Processing for more details:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
Instead of "bool AdaptDown()" when high QP is reported,
synchronously returning true or false depending on the result of
adaptation, this CL introduces
void QualityScalerQpUsageHandlerInterface::OnReportQpUsageHigh(
rtc::scoped_refptr<QualityScalerQpUsageHandlerCallback>);
Where
QualityScalerQpUsageHandlerCallback::OnQpUsageHandled(
bool clear_qp_samples);
Instructs the QualityScaler whether to clear samples before
checking QP the next time or to increase the frequency of checking
(corresponding to AdaptDown's return value prior to this CL).
QualityScaler no longer using AdaptationObserverInterface, this class
is renamed and moved to overuse_frame_detector.h.
The dependency between CheckQpTasks is made explicit with
CheckQpTask::Result and variables like observed_enough_frames_,
adapt_called_ and adapt_failed_ are moved there and given more
descriptive names.
Bug: webrtc:11521
Change-Id: I7faf795aeee5ded18ce75eb1617f88226e337228
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173760
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31140}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
This gets to the heart of unblocking call-level adaptation, largely
made possible due to the previous CLs in the chain.
The parts of the code that are responsible for responding to resource
usage signals, obtaining adaptations and applying them are moved to
ResourceAdaptationProcessor in call/adaptation/.
The parts of the code that are responsible for managing
VideoStreamEncoder-specific resources stay inside the
VideoStreamEncoderResourceManager class in video/adaptation/.
After this CL lands it should soon be possible to move the Processor
over to a separate task queue and let the Manager stay on the encoder
queue if PostTasks are added for communication between the two objects.
Bug: webrtc:11172
Change-Id: Ifa212467b4afd16e7ebfb9adfe17d2dca1cb7d67
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173021
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31105}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
This CL is a pure rename CL.
The next CL split this class up into a "Manager" part and a "Processor"
part. By doing the renaming separately, reviewing of the next CL should
be easier.
Bug: webrtc:11172
Change-Id: I2d5bac049b4bb88d4f947f8c4dc61e2131164a59
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173020
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31104}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
By pushing VideoAdaptationCounters updates on VideoSourceRestrictions
changes, alongside the Resource* that triggered the adaptation, we are
able to update |active_counts_| without an explicit dependency on the
VideoStreamAdapter. This allows a future CL to split up "processor"
logic from "video stream encoder resource and active counts" logic,
which will ultimately be necessary in order to do processing on a
"processing queue" and encoder and stats logic on the "encoder queue".
If the restrictions got cleared by an API call
(ResetVideoSourceRestrictions() or SetDegradationPreference()) we pass
null as the "reason_resource". This allows is to clear the
active_counts_, and the code that invokes
OnVideoSourceRestrictionsUpdated() does not have to be aware of
active_counts_ (needed to split the processor module in two).
Bug: webrtc:11172
Change-Id: Icab6d5121c0ebd27d2a00f1bffc8191f8f05f562
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173000
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31103}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
This is a pure move CL. In the future, the Processor will live in
call/adaptation/. This prevents circular dependencies.
Bug: webrtc:11172
Change-Id: Ib72503cc20e27ab6425538e3d55930c65e0b4a90
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172931
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31100}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The VideoStreamAdapter is currently responsible for aborting and not
providing adaptations if we are bitrate constrained
(kIsBitrateConstrained). Whether or not we are bitrate constrained is
clearly a resource question and should be phrased as such. By moving
this logic to Resource::IsAdaptationUpAllowed(), the VideoStreamAdapter
can continue to be thread-agnostic when a future CL introduces a
"processing queue", and the VideoStreamAdapter can be simplified: it
returns Adaptations even if we are constrained (but we refuse to Apply
them any resource rejects it).
This CL adds new Resource classes as inner classes of
ResourceAdaptationProcessor that take on the responsibility of
kIsBitrateConstrained logic:
PreventIncreaseResolutionDueToBitrateResource and
PreventAdaptUpInBalancedResource.
A third class, PreventAdaptUpDueToActiveCounts, also allows us to move
adaptation-aborting logic. This piece of code appears to be about not
adapting up if we’re already at the highest setting, which would be
VideoStreamAdapter responsibility (covered by
Adaptation::Status::kLimitReached), but it is actually more complicated
than that: the active_counts_ care about "reason", so it is really about
"is this resource type OK with you adapting up?". We should probably
rewrite this code in the future, but for now it is moved to an inner
class of ResourceAdaptationProcessor.
Other misc changes:
- ApplyDegradationPreference is moved to video_stream_adapter.[h/cc]
and renamed "Filter".
- OnResourceOveruse/Underuse now use Resource* as the reason instead of
AdaptReason. In a future CL, the processor will be split into a
"processor" part and a "video stream encoder resource manager" part.
Only the manager needs to know about AdaptReason since this is only
used for |active_counts_| and we want to get rid of it as much as
possible as it is not future-proof.
Bug: webrtc:11172
Change-Id: I2eba9ec3d717f7024c451aeb14635fe759551318
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172930
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31099}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The "input state" of a VideoStream, needed for adaptation and
decision-making, are: source resolution and frame rate, codec type and
min pixels per frame (based on encoder scaling settings). These values
are modified on the encoder queue of the VideoStreamEncoder.
But in order to unblock call-level adaptation processing, where
adaptation and decision making happens off the encoder queue, a snapshot
of the input states need to be available at point of processing:
introducing the VideoStreamInputState.
In this CL, the VideoStreamInputStateProvider is added to provide input
state snapshots across threads based on input from VideoStreamEncoder
and VideoStreamEncoderObserver.
The input state's HasInputFrameSizeAndFramesPerSecond() can now be
DCHECKed inside the VideoStreamAdapter in favor of having less
Adaptation::Status codes. Whether input is "sufficient" for adaptation
is now the responsibility of the Processor. (Goal: adapter is purely a
Adaptation generator and apply-er.)
Somewhat tangental, this CL also deletes VideoStreamEncoder-specific
methods from ResourceAdaptationProcessorInterface making them an
implementation detail of ResourceAdaptationProcessor. In a future CL,
the "processor" will be split up into a "processor" part and a "video
stream encoder resource manager" part - more on that later.
Bug: webrtc:11172
Change-Id: Id9b158f569db0140b75360aaf0f7e2e28fb924f4
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172928
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31098}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The VideoStreamAdapter is responsible for generating adaptation
suggestions according to its DegradationPreference. Today there is one
DegradationPreference that you set, but internally the preference it
uses to make decisions is EffectiveDegradationPreference() which
reinterprets “balanced” as “maintain-resolution” if screenshare is used.
By moving the “effective” logic to the ResourceAdaptationProcessor, the
VideoStreamAdapter will not need to know about the type of track, and
the responsibility of the adapter is minimized. The “effective” logic
is non-standard and something we want to get rid of - until then, it
should be the responsibility of the processor to configure the adapter
to use the appropriate strategy, rather than for the adapter to know
about more states of the system than it needs to.
Future CLs will further minimize what the adapter needs to know, moving
"decision-making" logic to the Processor and "is adapt up allowed?"
logic to the Resources.
By removing the VideoInputMode enum the VideoStreamAdapter does not
know if we have input which has to be checked externally. Input
handling is followed-up on in the next CL.
Bug: webrtc:11172
Change-Id: I37ec9e7392f835cf8fef9829a2c945183f0e9b65
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172927
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31096}
Replaces this with 2 methods instead, adding clarity.
ClearAdaptationStats
- Resets the adaptations statistics to 0. This is done,
when the degredation is reset, for example when the preference
is changed to/from BALANCED.
UpdateAdaptationMaskingSettings
- Updates the settings for adaptation statistics reporting.
This way we don't report quality adaptations if quality scaling
is not enabled (same for resolution/fps scaling).
The adaptation counting inside the SendStatisticsProxy is
now done in a struct that counts the totals, and then masks
out these counts based on the adaptation settings. The
MaskedAdaptationSteps uses optionals to hide the values we
shoudn't report, while the AdaptationSteps always hold the real
totals.
All tests have been updated to use the Reset/Clear method as needed.
Now that AdaptationCounters and AdaptSteps use the same structure,
AdaptationCounters was moved to api/video and replaces AdaptSteps.
The AdaptReason enum is also redundant now, and will be removed
in a follow-up CL.
R=hbos@webrtc.org
Bug: webrtc:11392
Change-Id: Iaed6488581325d341a056b5bbf76a01c19d6c282
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/171685
Reviewed-by: Niels Moller <nisse@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31083}
This code was previously only exercised by
video_stream_adapter_unittest.cc and other tests, acting more like
integration tests than unit tests. Now that the VideoStreamAdapter is
in a good state, more extensive test coverage is added.
Testing includes:
- Default restrictions.
- Adapting up or down in "maintain-framerate", "maintain-resolution"
and "balanced", including...
- expecting how frame rate and/or resolution is affected,
- reaching kLimitReached,
- and reaching unrestricted.
- That "disabled" does not adapt.
- When adaptation is not possible, including...
- kInsufficientInput
- kAwaitingPreviousAdaptation
- kIsBitrateConstrained
- PeekNextRestrictions()
- "balanced" + "screenshare" = "maintain-resolution"
- Change degradation preference to/from "balanced" clears restrictions.
- That using invalidated adaptations triggers DCHECKs.
Bug: webrtc:11393
Change-Id: I28e2cf227bc1fd8871ee0d18d9570d4063449160
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/170625
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30816}
This CL introduces the Adaptation class used by VideoStreamRestrictor.
This refactors the AdaptationTarget, AdaptationTargetOrReason,
CannotAdaptReason and AdaptationAction.
What is publicly exposed is simply a Status code. If it's kValid then
we can adapt, otherwise the status code describes why we can't adapt
(just like CannotAdaptReason prior to this CL). This means
AdaptationTargetOrReason is no longer needed. Target+reason are merged.
The other classes are renamed and moved and put in the private
namespace of Adaptation: Only the VideoStreamAdapter (now a friend
class of Adaptation) and its inner class VideoSourceRestrictor needs to
know how to execute the adaptation.
Publicly, you can now tell the effects of the adaptation without
applying it with PeekNextRestrictions() - both current and next steps
are described in terms of VideoSourceRestrictions. The rest are hidden.
This would make it possible, in the future, for a Resource to accept or
reject a proposed Adaptation by examining the resulting frame rate and
resolution described by the resulting restrictions. E.g. even if we are
not overusing bandwidth at the moment, the BW resource can prevent us
from applying a restriction that would exceed the BW limit before we
apply it.
This CL also moves input to a SetInput() method, and Increase/Decrease
methods of VideoSourceRestrictor are made private in favor of
ApplyAdaptationSteps().
Bug: webrtc:11393
Change-Id: Ie5e2181836ab3713b8021c1a152694ca745aeb0d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/170111
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#30794}
This introduces AdaptationTargetOrReason and gets rid of
VideoStreamAdapter's dependency on the VideoStreamEncoderObserver.
AdaptationTargetOrReason provides information about why an adaptation
target could not be returned from GetAdaptUpTarget() and
GetAdaptDownTarget() with the enum CannotAdaptReason and the boolean
min_pixel_limit_reached.
While the enum value is not used by the caller in this CL, it makes
explicit reasons the adapter is allowed to reject a target. TODOs are
added documenting how we want to get rid of kAwaitingPreviousAdaptation
for multi-stream use cases and how kIsBitrateConstrained can be
rephrased as a resource problem in the future.
min_pixel_limit_reached() allows us to move the responsibility of stats
reporting to the module. A TODO documents how this could be replaced by
kLimitReached or similar logic in the future.
Bug: webrtc:11393
Change-Id: Iffdd8ddb01641937741fac353174ea14168477ce
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/169928
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30789}
The EffectiveDegradationPreference() exposed an "implementation detail"
of the VideoStreamAdapter - how degradation preference may be modified.
By changing the return value of ApplyAdaptationTarget() this dependency
could be removed. We still have a TODO to get rid of the
ResourceListenerResponse enum, but that is QualityScaler related work.
This CL does the following:
- Module's GetAdaptUpTarget/GetAdaptDownTarget/ApplyAdaptationTarget
methods are removed in favor if invoking the VideoStreamAdapter's
version of these methods directly.
- Removing the EffectiveDegradationPreference() usage in
OveruseFrameDetectorResourceAdaptationModule meant moving that usage
to VideoStreamAdapter.
- MinPixelsPerFrame() is moved to VideoStreamAdapter; this is "can
adapt?" logic, i.e. the adapter's responsibility.
Bug: webrtc:11393
Change-Id: I75091ce97093bfa48a6d883492de30ed4b004492
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/169859
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30714}
This CL moves GetAdaptUpTarget(), GetAdaptDownTarget() and
ApplyAdaptationTarget() - and related code - to the VideoStreamAdapter.
This includes pieces related to calculating how to adapt, including:
- DegradationPreference
- BalancedDegradationPreference
- AdaptationRequest and last_adaptation_request_
- CanAdaptUpResolution()
The VideoStreamAdapter's interface has changed: VideoSourceRestrictor
methods are now hidden in favor of methods exposing AdaptationTarget.
This CL also does some misc moves:
- GetEncoderBitrateLimits is moved and renamed to
VideoEncoder::EncoderInfo::GetEncoderBitrateLimitsForResolution.
- EncoderSettings moved to a separate file.
// For api/video_codecs/video_encoder.[cc/h] changes, which is the
// moving of a function.
TBR=sprang@webrtc.org
Bug: webrtc:11393
Change-Id: Ie6bd8ef644ce927d7eca6ab90a0a7bcace682f3c
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/169842
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30708}
This CL simply moves the VideoSourceRestrictor from being an inner class
of OveruseFrameDetectorResourceAdaptationModule to a new class,
VideoStreamAdapter.
In follow-up CLs, the responsibility of determining what the next step
for adapting up or down should also be moved to the VideoStreamAdapter.
The end-goal is that the VideoStreamAdapter takes care of "can adapt?"
and "do adapt!" type of logic so that a multi-stream aware adaptation
module can decide which stream (adapter) to adapt, and the adapter can
take care of the nitty gritty details of doing so.
In this CL the "can?"/"do!" part is realized but not the logic for
determining what the next step up or down is, and the class interface
needs improvement.
This CL also sets up the video/adaptation/ subdirectory and moves the
AdaptationCounters class here. Other adaptation-related classes (e.g.
the module and its resources) should move into this directory as well
in the future.
Bug: webrtc:11393
Change-Id: I2c12c1281eca854c62791abb65f0aca47a119726
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/169542
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Andreassson <henrika@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30705}
