Because a single Resource only has a single ResourceListener, injected
Resources only gets wired up to the stream's ResourceAdaptationProcessor
that was last to call SetResourceListener. This could potentially lead
to the relevant stream not adapting based on the injected resource
because it got wired up to the wrong stream's processor.
This CL fixes this issue by introducing BroadcastResourceListener. By
listening to 1 resource (the injected one), it can spawn N "adapter"
resources that mirror's the injected resource's usage signal, allowing
all ResourceAdaptationProcessor's to react to the signal.
This is wired up in Call, and tests are updated to verify the signal
gets through.
Bug: chromium:1101263, webrtc:11720
Change-Id: I8a37284cb9a68f08ca1bdb1ee050b7144c451297
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/178386
Reviewed-by: Stefan Holmer <stefan@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31612}
When a resource is removed from the ResourceAdaptationProcessor,
and the resource is the most limited resource, we reset the current
adaptation to the next most limited resource level. In the case that
this resource is the only adapted resource then we reset all adaptations.
Bug: webrtc:11636
Change-Id: I29acc5a3934f42f00db79b3bb2171156b1313937
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176406
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31520}
This CL adds AddAdaptationResource to Call and
AddAdaptationResource/GetAdaptationResources method to relevant
VideoSendStream and VideoStreamEncoder interfaces and implementations.
Unittests are added to ensure that resources can be added to the Call
both before and after the creation of a VideoSendStream and that the
resources always gets added to the streams.
In a follow-up CL, we will continue to plumb the resources all the way
to PeerConnectionInterface, and an integration test will then be added
to ensure that injected resources are capable of triggering adaptation.
Bug: webrtc:11525
Change-Id: I499e9c23c3e359df943414d420b2e0ce2e9b2d56
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/177002
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Stefan Holmer <stefan@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31499}
A more detailed explaination is in the bug, but this changes
the way that adaptation happens when multiple resources are
limited. Only the one that is most limited can trigger an
adaptation up. If multiple resources are most limited both
need to underuse to adapt up.
Some of the changes in this patch to make it all work:
* VideoStreamEncoder unittests that did not reflect this
new behaviour have been changed.
* PeekNextRestrictions returns the adaptation counters as
well as the restrictions.
* Adaptation statstics have changed so that when adapting
up all resources are tagged as triggering the adaptation.
Additionally the statistics for the current adaptation is
now the total number of adaptations per reason, rather then
the number of adaptations due to that reason.
* PreventAdaptUpDueToActiveCounts is removed as most limited
resource is a strong implementation of that.
Bug: webrtc:11553
Change-Id: If1545a201c8e019598edf82657a1befde8b05268
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176128
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31497}
Use the current task queue in the test instead.
R=hbos@webrtc.org
Bug: None
Change-Id: I266796df3de675a808bcf2da08901411fadc86ae
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176414
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31439}
This is a prerequisite to implementing it externally.
Bug: webrtc:11525
Change-Id: I9cb3b4418396485d3eb9f25cafa51cbff6db7817
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176401
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Stefan Holmer <stefan@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31418}
ResourceListener::OnResourceUsageStateMeasured() now takes
ResourceUsageState as argument, making Resource::UsageState()
superfluous.
With the existing "fire-and-forget" behavior of always clearing usage
state on reacting to a signal, there is no longer a need to call
ClearUsageState() so this too is removed. (We may want to have a
callback in the future to hint to the Resource that it is a good idea
to clear internal measurement samples, i.e. because the load of the
system is about to change, but we can revisit that when we need it.)
Moving the usage state to the callback has the benefit of getting rid
of the assumption that UsageState() has to return the same value every
time it is called in the same task.
This CL is also the final nail in the coffin for Resource needing to
know about the adaptation task queue: ResourceAdaptationProcessor's
ResourceListener now takes care of posting to the adaptation task
queue. To support this, the processor's SequenceChecker is replaced
by a TaskQueueBase pointer.
Bug: webrtc:11525, webrtc:11618
Change-Id: I2277e71cc3759c85b62465020935603f03792c94
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176376
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31416}
IsAdaptationUpAllowed is moved from Resource to AdaptationConstraint.
OnAdaptationApplied is moved from Resource to AdaptationListener.
In a future CL, Resource will be moved to api/, but
AdaptationConstraint and AdaptationListener will stay in call/.
The processor, encode stream and manager are updated to keep track of
both resources, constraints and listeners. Fakes and tests are updated.
After this CL, the manager's inner classes that prevent adaptation
implement AdaptationConstraint instead of Resource.
Bug: webrtc:11525
Change-Id: Ie9cd5b1ba7d8e161951e131ab8f6bd9d5cf765bf
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176368
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31409}
This CL is in preparation for moving Resource to the api/ folder. It
does not move it, but makes it such that the moving CL can be a pure
move.
In order to do this, we must stop depending on rtc_base/rtc::TaskQueue
in favor of api/webrtc::TaskQueueBase.
There are also other rtc_base/ dependencies that we do not want to
expose to the api/ folder, like critical sections and thread
annotations which are not publically exposed. To get around this, we
make Resource an abstract interface and move all of the base class
functionality into a new non-api/ class: VideoStreamEncoderResource.
The Resource now has Register/UnregisterAdaptationTaskQueue() methods.
By explicitly unregistering, we can ensure validity of the pointer even
if the Resource outlives the PeerConnection. While public interface
methods are only to be called on the adaptation task queue, posting to
the task queue happens off-queue, so a |lock_| is introduced to guard
it.
Bug: webrtc:11525
Change-Id: I50b3a30960cdec9032016c779b47001c01dad32f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176320
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31402}
This should help debugging when adaptation is or is not happening
unexpectedly. Log spam is prevented by not logging if the same
result happened to the same resource already and we haven't
adapted since then.
Bug: webrtc:11616
Change-Id: Ia6c5cc35061d252f1c66f2f2bf3b94d2485498d6
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176221
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31378}
This will help show what the restrictions are before
and after they transform into sink wants.
These will not be so spammy as adaptations do not happen
very often.
Bug: None
Change-Id: Ib72b313f68c6934d7833d8a3f14ce00df602832b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175800
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31341}
Now we only await a previous adaptation if the degradataion
preference is the same as our current degradation preference.
Without this guard we can get stuck as detailed in the attached bug.
Bug: webrtc:11562
Change-Id: I91be48546446ef8d01fe901bc6889201a5b97ba6
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174805
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31236}
This CL unblocks future Call-Level Mitigation strategies by moving the
ResourceAdaptationProcessor to a separate task queue. This signifies a
major milestone in the new resource adaptation architecture because
with this CL the threading model is in place and moving the Processor
to the Call and increasing its responsibilities is made possible.
In this CL, we still have one Processor per VideoStreamEncoder and the
VideoStreamEncoder is responsible for the creation and the destruction
of its Processor and that Processor's task queue. But the PostTasks are
in place and the decision-making is executed on a separate queue.
This CL:
- Moves ResourceAdaptationProcessor to an adaptation task queue.
It continues to be entirely single-threaded, but now operates on a
separate task queue.
- Makes Resources thread-safe: Interaction with the Processor, i.e.
OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens
on the adaptation task queue. State updates are pushed from the
encoder task queue with PostTasks.
- QualityScalerResource operates on both task queues; the QP usage
callbacks are invoked asynchronously.
- The VideoStreamEncoderResourceManager operates on the encoder task
queue with the following exceptions:
1) Its resources are accessible on any thread (using a mutex). This
is OK because resources are reference counted and thread safe.
This aids adding and removing resources to the Processor on the
adaptation task queue.
2) |active_counts_| is moved to the adaptation task queue. This makes
it possible for PreventAdaptUpDueToActiveCounts to run
IsAdaptationUpAllowed() on the adaptation task queue.
A side-effect of this is that some stats reporting now happen on
the adaptation task queue, but that is OK because
VideoStreamEncoderObserver is thread-safe.
The Manager is updated to take the new threading model into account:
- OnFrameDroppedDueToSize() posts to the adaptation task queue to
invoke the Processor.
- OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation
task queue, updates |active_counts_| synchronously but posts to the
encoder task queue to update video source restrictions (which it
only uses to calculate target frame rate).
- MaybePerformQualityRampupExperiment() posts to the adaptation task
queue to maybe reset video source restrictions on the Processor.
|quality_rampup_done_| is made std::atomic.
Bug: webrtc:11542, webrtc:11520
Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31231}
In a future CL, adaptation processing and stream encoder resource
management will happen on different task queues. When this is the case,
asynchronous tasks will be posted in both directions and some resources
will have internal states used on multiple threads.
This CL makes the Resource class reference counted in order to support
posting tasks to a different threads without risk of use-after-free
when a posted task is executed with a delay. This is preferred over
WeakPtr strategies because WeakPtrs are single-threaded and preferred
over raw pointer usage because the reference counted approach enables
more compile-time and run-time assurance. This is also "future proof";
when resources can be injected through public APIs, ownership needs to
be shared between libwebrtc and the application (e.g. Chrome).
To reduce the risk of making mistakes in the future CL, sequence
checkers and task queue DCHECKs are added as well as other DCHECKs to
make sure things have been cleaned up before destruction, e.g:
- Processor gets a sequence checker. It is entirely single-threaded.
- Processor must not have any attached listeners or resources on
destruction.
- Resources must not have any listeners on destruction.
- The Manager, EncodeUsageResource and QualityScalerResource DCHECKs
they are running on the encoder queue.
- TODOs are added illustrating where we want to add PostTasks in the
future CL.
Lastly, upon VideoStreamEncoder::Stop() we delete the
ResourceAdaptationProcessor. Because the Processor is already used in
posted tasks, some if statements are added to ensure the Processor is
not used after destruction.
Bug: webrtc:11542, webrtc:11520
Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31217}
The problem is that a resource that signals Underuse would be
able to trigger an adapt up when it was never limited in the past.
This means that an underused resource would be able to negate the
adaptations made for an overused one.
For example, consider a fast CPU on a bad link. The QP for the image
is high but the CPU is underused. Without requiring previous underuse,
everytime the QP would signal overuse and trigger an adpatation down,
the CPU would signal underuse and trigger an adaptation up.
This works today as we want by using the active counts in the
VideoStreamEncoderResourceManager. This change
makes it a normal behaviour independant of active counts.
The problem with active counts is that is only works with 2 resources.
When resources are injectable it no longer works as expected.
Bug: webrtc:11522, webrtc:11523
Change-Id: I140636ce206d74e00a6b6f8558162bb8afffda1c
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174482
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31181}
This replaces references to the ResourceAdaptationProcessor with
references to its interface. This would make it possible to have
alternative implementations or inject fake/mock implementations for
testing.
The VideoStreamAdapter is still responsible for constructing the
ResourceAdaptationProcessor, but beyond construction it is agnostic
towards the implementation.
With this CL, I claim https://crbug.com/webrtc/11222 complete.
TBR=ilnik@webrtc.org
Bug: webrtc:11222
Change-Id: I6e7a73bf1d0b5e97bc694f66180a747b27ffb018
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174160
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31148}
This CL adds miscellaneous unit tests for the
ResourceAdaptationProcessor, the VideoSourceRestrictions comparators and
the VideoStreamInputStateProvider.
Bug: webrtc:11172
Change-Id: If95f69644aaf2b43e3b19d5729bedef0b438c77b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174101
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31147}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The ResourceListenerResponse was used to make the QualityScaler
not clear QP samples and instead increase its frequency of checking for
QP under certain circumstances, see enum description:
https://webrtc.googlesource.com/src.git/+/c70b1028d47c1aee4892545190cd66e97d09cd55/call/adaptation/resource.h#33
Because the QualityScaler depends on whether and how adaptation
happened it should listen to adaptation happening.
This CL moves the logic that was previously in VideoStreamAdapter closer
to the QualityScaler: QualityScalerResource::OnAdaptationApplied().
This would allow the VideoStreamAdapter to operate on a separate task
queue in the future, with no dependencies on any stream-specific
resources that might operate on other task queues.
Bug: webrtc:11172, webrtc:11521
Change-Id: I07971a8a5fab5715f4ccb7d2c63f1b92bd47170f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173090
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31143}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The ResourceUsageState was written as: {kOveruse, kStable, kUnderuse}.
The assumption was that if a resource neither wanted to adapt up or
down it would report kStable. But with the addition of
Resource::IsAdaptationUpAllowed() (prior CL) the notion of being
"stable" was already captured outside of ResourceUsageState.
Furthermore, kStable failed to capture what IsAdaptationUpAllowed() did
not: whether we can go up depends on the resulting resolution or frame
rate (restrictions_after). Perhaps we can go up a little, but not a lot.
This CL also adds Resource::ClearUsageState(). After applying an
adaptation, all usage states become invalidated (new measurements are
needed to know if we are still over- or underusing). This was always
the case, but prior to this CL this was not accurately reflected in the
Resource::usage_state() in-between measurements.
Bug: webrtc:11172
Change-Id: I140ff3114025b7732e530564690783e168d2509b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173088
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31110}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
This gets to the heart of unblocking call-level adaptation, largely
made possible due to the previous CLs in the chain.
The parts of the code that are responsible for responding to resource
usage signals, obtaining adaptations and applying them are moved to
ResourceAdaptationProcessor in call/adaptation/.
The parts of the code that are responsible for managing
VideoStreamEncoder-specific resources stay inside the
VideoStreamEncoderResourceManager class in video/adaptation/.
After this CL lands it should soon be possible to move the Processor
over to a separate task queue and let the Manager stay on the encoder
queue if PostTasks are added for communication between the two objects.
Bug: webrtc:11172
Change-Id: Ifa212467b4afd16e7ebfb9adfe17d2dca1cb7d67
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173021
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31105}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
By pushing VideoAdaptationCounters updates on VideoSourceRestrictions
changes, alongside the Resource* that triggered the adaptation, we are
able to update |active_counts_| without an explicit dependency on the
VideoStreamAdapter. This allows a future CL to split up "processor"
logic from "video stream encoder resource and active counts" logic,
which will ultimately be necessary in order to do processing on a
"processing queue" and encoder and stats logic on the "encoder queue".
If the restrictions got cleared by an API call
(ResetVideoSourceRestrictions() or SetDegradationPreference()) we pass
null as the "reason_resource". This allows is to clear the
active_counts_, and the code that invokes
OnVideoSourceRestrictionsUpdated() does not have to be aware of
active_counts_ (needed to split the processor module in two).
Bug: webrtc:11172
Change-Id: Icab6d5121c0ebd27d2a00f1bffc8191f8f05f562
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173000
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31103}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
This is a pure move CL. In the future, the Processor will live in
call/adaptation/. This prevents circular dependencies.
Bug: webrtc:11172
Change-Id: Ib72503cc20e27ab6425538e3d55930c65e0b4a90
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172931
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31100}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The VideoStreamAdapter is currently responsible for aborting and not
providing adaptations if we are bitrate constrained
(kIsBitrateConstrained). Whether or not we are bitrate constrained is
clearly a resource question and should be phrased as such. By moving
this logic to Resource::IsAdaptationUpAllowed(), the VideoStreamAdapter
can continue to be thread-agnostic when a future CL introduces a
"processing queue", and the VideoStreamAdapter can be simplified: it
returns Adaptations even if we are constrained (but we refuse to Apply
them any resource rejects it).
This CL adds new Resource classes as inner classes of
ResourceAdaptationProcessor that take on the responsibility of
kIsBitrateConstrained logic:
PreventIncreaseResolutionDueToBitrateResource and
PreventAdaptUpInBalancedResource.
A third class, PreventAdaptUpDueToActiveCounts, also allows us to move
adaptation-aborting logic. This piece of code appears to be about not
adapting up if we’re already at the highest setting, which would be
VideoStreamAdapter responsibility (covered by
Adaptation::Status::kLimitReached), but it is actually more complicated
than that: the active_counts_ care about "reason", so it is really about
"is this resource type OK with you adapting up?". We should probably
rewrite this code in the future, but for now it is moved to an inner
class of ResourceAdaptationProcessor.
Other misc changes:
- ApplyDegradationPreference is moved to video_stream_adapter.[h/cc]
and renamed "Filter".
- OnResourceOveruse/Underuse now use Resource* as the reason instead of
AdaptReason. In a future CL, the processor will be split into a
"processor" part and a "video stream encoder resource manager" part.
Only the manager needs to know about AdaptReason since this is only
used for |active_counts_| and we want to get rid of it as much as
possible as it is not future-proof.
Bug: webrtc:11172
Change-Id: I2eba9ec3d717f7024c451aeb14635fe759551318
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172930
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31099}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The "input state" of a VideoStream, needed for adaptation and
decision-making, are: source resolution and frame rate, codec type and
min pixels per frame (based on encoder scaling settings). These values
are modified on the encoder queue of the VideoStreamEncoder.
But in order to unblock call-level adaptation processing, where
adaptation and decision making happens off the encoder queue, a snapshot
of the input states need to be available at point of processing:
introducing the VideoStreamInputState.
In this CL, the VideoStreamInputStateProvider is added to provide input
state snapshots across threads based on input from VideoStreamEncoder
and VideoStreamEncoderObserver.
The input state's HasInputFrameSizeAndFramesPerSecond() can now be
DCHECKed inside the VideoStreamAdapter in favor of having less
Adaptation::Status codes. Whether input is "sufficient" for adaptation
is now the responsibility of the Processor. (Goal: adapter is purely a
Adaptation generator and apply-er.)
Somewhat tangental, this CL also deletes VideoStreamEncoder-specific
methods from ResourceAdaptationProcessorInterface making them an
implementation detail of ResourceAdaptationProcessor. In a future CL,
the "processor" will be split up into a "processor" part and a "video
stream encoder resource manager" part - more on that later.
Bug: webrtc:11172
Change-Id: Id9b158f569db0140b75360aaf0f7e2e28fb924f4
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172928
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31098}
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing
The POC was used to demo the new design but was never used outside of
unit tests. The plan being to iteratively improve the
ResourceAdaptationProcessor rather than to replace it, we delete the POC
classes to avoid bloat and conflicts.
Bug: webrtc:11172
Change-Id: Ic49afdc471d2d774541f8ef3316b4c6d0a9b8a14
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172923
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31094}
This CL introduces the Adaptation class used by VideoStreamRestrictor.
This refactors the AdaptationTarget, AdaptationTargetOrReason,
CannotAdaptReason and AdaptationAction.
What is publicly exposed is simply a Status code. If it's kValid then
we can adapt, otherwise the status code describes why we can't adapt
(just like CannotAdaptReason prior to this CL). This means
AdaptationTargetOrReason is no longer needed. Target+reason are merged.
The other classes are renamed and moved and put in the private
namespace of Adaptation: Only the VideoStreamAdapter (now a friend
class of Adaptation) and its inner class VideoSourceRestrictor needs to
know how to execute the adaptation.
Publicly, you can now tell the effects of the adaptation without
applying it with PeekNextRestrictions() - both current and next steps
are described in terms of VideoSourceRestrictions. The rest are hidden.
This would make it possible, in the future, for a Resource to accept or
reject a proposed Adaptation by examining the resulting frame rate and
resolution described by the resulting restrictions. E.g. even if we are
not overusing bandwidth at the moment, the BW resource can prevent us
from applying a restriction that would exceed the BW limit before we
apply it.
This CL also moves input to a SetInput() method, and Increase/Decrease
methods of VideoSourceRestrictor are made private in favor of
ApplyAdaptationSteps().
Bug: webrtc:11393
Change-Id: Ie5e2181836ab3713b8021c1a152694ca745aeb0d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/170111
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#30794}
This CL moves GetAdaptUpTarget(), GetAdaptDownTarget() and
ApplyAdaptationTarget() - and related code - to the VideoStreamAdapter.
This includes pieces related to calculating how to adapt, including:
- DegradationPreference
- BalancedDegradationPreference
- AdaptationRequest and last_adaptation_request_
- CanAdaptUpResolution()
The VideoStreamAdapter's interface has changed: VideoSourceRestrictor
methods are now hidden in favor of methods exposing AdaptationTarget.
This CL also does some misc moves:
- GetEncoderBitrateLimits is moved and renamed to
VideoEncoder::EncoderInfo::GetEncoderBitrateLimitsForResolution.
- EncoderSettings moved to a separate file.
// For api/video_codecs/video_encoder.[cc/h] changes, which is the
// moving of a function.
TBR=sprang@webrtc.org
Bug: webrtc:11393
Change-Id: Ie6bd8ef644ce927d7eca6ab90a0a7bcace682f3c
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/169842
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30708}
This CL simply moves the VideoSourceRestrictor from being an inner class
of OveruseFrameDetectorResourceAdaptationModule to a new class,
VideoStreamAdapter.
In follow-up CLs, the responsibility of determining what the next step
for adapting up or down should also be moved to the VideoStreamAdapter.
The end-goal is that the VideoStreamAdapter takes care of "can adapt?"
and "do adapt!" type of logic so that a multi-stream aware adaptation
module can decide which stream (adapter) to adapt, and the adapter can
take care of the nitty gritty details of doing so.
In this CL the "can?"/"do!" part is realized but not the logic for
determining what the next step up or down is, and the class interface
needs improvement.
This CL also sets up the video/adaptation/ subdirectory and moves the
AdaptationCounters class here. Other adaptation-related classes (e.g.
the module and its resources) should move into this directory as well
in the future.
Bug: webrtc:11393
Change-Id: I2c12c1281eca854c62791abb65f0aca47a119726
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/169542
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Andreassson <henrika@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30705}
* This replaces the video stream methods for forcing adaptation
with a mock resource that triggers overuse.
* Resources can now be injected to the Module using the AddResource
function.
* Resources now have tests for adding and removing callbacks.
* Quality/EncoderUse% resources are tracked in the Resource list of
the adaptation module.
* The adaptation module ties all resources to a reason to keep stats
working as expected.
BUG=webrtc:11377
Change-Id: I1f5902f7416dc41b4915c0072e6f0da2bb3bb2b7
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168948
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30610}
Adapting up or down is currently a "Maybe Adapt" method. In the future
we will want to be able to decide which stream to adapt, and as such we
need to be able to tell if a stream is able to do so.
This takes us one step in that direction, by refactoring
OveruseFrameDetectorResourceAdaptationModule::VideoSourceRestrictor's
methods to follow a simple pattern:
- What is the next step?
GetHigherFrameRateThan, GetLowerFrameRateThan,
GetHigherResolutionThan, GetLowerResolutionThan
- Can we adapt?
CanIncreaseFrameRate, CanDecreaseFrameRate,
CanIncreaseResolution, CanDecreaseResolution
- Do adapt!
IncreaseFrameRateTo, DecreaseFrameRateTo,
IncreaseResolutionTo, DecreaseResolutionTo
Hopefully this makes the code easier to follow.
This CL changes the "Request Higher/Lower" methods to take the target
as input instead of implicitly calculating the target from the current
input resolution or frame rate.
Bug: webrtc:11222
Change-Id: If625834e921a24a872145105f5d553fb8f9f1795
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168966
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30600}
This reverts commit 93d9ae8a17f2e7b90641cbac28e740afc67d383a.
Reason for revert: Perf regression.
Original change's description:
> Remove ResourceAdaptationModule::OnMaybeEncodeFrame
>
> We can react just as well at OnEncodeVideoFrame, which is the same
> behaviour except after checking if the Encoder is paused and the frame
> dropper.
>
> For the initial frame drop, the frame dropper is irrelevant as the frame
> can not be dropped until we are accepting frames. If we didn't drop the
> frame, the encoder can't be paused as the data rate
> is over 0.
>
> For the quality rampup experiment, similar for encoder paused - we can't
> rampup if we are paused anyways since the data rate needs to be non-zero.
> If we are dropping frames we likely don't want to do quality rampup
> anyways.
>
> Bug: webrtc:11222
> Change-Id: Ie3e09d9d8d509dc17ba7a1443cf4747f61c04f6a
> Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168601
> Reviewed-by: Henrik Boström <hbos@webrtc.org>
> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
> Commit-Queue: Evan Shrubsole <eshr@google.com>
> Cr-Commit-Position: refs/heads/master@{#30539}
TBR=ilnik@webrtc.org,hbos@webrtc.org,eshr@google.com
# Not skipping CQ checks because original CL landed > 1 day ago.
No-Try: True
Bug: webrtc:11222
Change-Id: Ifb2fc74eb7572568fb0ee1b53a09e4180f87b30c
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168880
Commit-Queue: Mirko Bonadei <mbonadei@webrtc.org>
Reviewed-by: Mirko Bonadei <mbonadei@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30568}
We can react just as well at OnEncodeVideoFrame, which is the same
behaviour except after checking if the Encoder is paused and the frame
dropper.
For the initial frame drop, the frame dropper is irrelevant as the frame
can not be dropped until we are accepting frames. If we didn't drop the
frame, the encoder can't be paused as the data rate
is over 0.
For the quality rampup experiment, similar for encoder paused - we can't
rampup if we are paused anyways since the data rate needs to be non-zero.
If we are dropping frames we likely don't want to do quality rampup
anyways.
Bug: webrtc:11222
Change-Id: Ie3e09d9d8d509dc17ba7a1443cf4747f61c04f6a
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168601
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#30539}
The Resource interface (previously a skeleton not used outside of
testing) is updated to inform listeners of changes to resource
usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements,
CurrentUsage). The interface is implemented by
OveruseFrameDetectorResourceAdaptationModule's inner classes
EncodeUsageResource and QualityScalerResource.
The new ResourceUsageListener interface is implemented by
OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding
AdaptationObserverInterface::AdaptReason to the ResourceUsageListener
interface, the module figures out if the reason is "kCpu" or "kQuality"
by looking which Resource object triggered
OnResourceUsageStateMeasured(). These resources no longer need an
explicit reference to OveruseFrameDetectorResourceAdaptationModule and
could potentially be used by a different module.
In this CL, AdaptationObserverInterface::AdaptDown()'s return value is
still needed by QualityScaler. This is mirrored in the return value of
ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added
to remove it and a comment explains how the current implementation
seems to break the contract of the method (as was the case prior to
this CL).
Follow-up work include:
- Move EncodeUsageResource and QualityScalerResource to separate files.
- Make resources injectable, allowing fake resources in testing and
removing OnResourceOveruseForTesting() methods.
(Investigate adding the necessary input signals to the Resource
interface or relevant sub-interfaces so that the module does not need
to know which Resource implementation is used.)
- And more! See whiteboard :)
Bug: webrtc:11222
Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#30469}
It would be nice for this to stay in video stream encoder,
but this feature is mostly related to quality scaling. Perhaps
something easier to understand is possible in the future.
Bug: webrtc:11222
Change-Id: I71705f33ff94bbcf2fb9b5c94226c8e76dcba94c
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168051
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30446}
This defines the following methods:
- OnFrame(), replaces SetLastFramePixelCount().
- OnFrameDroppedDueToSize(), a rename of FrameDroppedDueToSize() to
match the other methods.
- OnEncodeStarted(), a rename of the incorrectly named FrameCaptured().
- OnEncodeCompleted(), a rename of the poorly named FrameSent().
In order to get rid of SetLastFramePixelCount(), the "we don't know the
frame size" use case - which was previously implicitly avoided by
invoking SetLastFramePixelCount() with a made-up value for
last_frame_info_ - is now avoided using ".value_or()" in
LastInputFrameSizeOrDefault(). This does mean that a constant 144p
resolution value is referenced in two places, but the fact that this is
a magic value is at least made explicit. This may help future
improvements.
Bug: webrtc:11222
Change-Id: I3b28daa8c5ecf57c6537957d4759f15e24bb2234
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/166961
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30352}
The poorly named SetEncoderStartBitrate() is renamed
SetEncoderTargetBitrate() and added to the abstract resource adaptation
module interface.
The so-called "start bitrate" was updated to match the target bitrate,
so this was only ever a "start bitrate" until we had any estimates. The
variable is renamed in VideoStreamEncoder as well, and usage of optional
types are introduced to avoid magical values in a few places in the
existing code.
Bug: webrtc:11222
Change-Id: Idde92f68f34616aa3c34ab77a791fdbe7ea7af26
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/166880
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30347}
