admin/webrtc_m130
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[Adaptation] Move adaptation logic to a separate task queue.

Browse Source 
This CL unblocks future Call-Level Mitigation strategies by moving the
ResourceAdaptationProcessor to a separate task queue. This signifies a
major milestone in the new resource adaptation architecture because
with this CL the threading model is in place and moving the Processor
to the Call and increasing its responsibilities is made possible.

In this CL, we still have one Processor per VideoStreamEncoder and the
VideoStreamEncoder is responsible for the creation and the destruction
of its Processor and that Processor's task queue. But the PostTasks are
in place and the decision-making is executed on a separate queue.

This CL:
- Moves ResourceAdaptationProcessor to an adaptation task queue.
  It continues to be entirely single-threaded, but now operates on a
  separate task queue.
- Makes Resources thread-safe: Interaction with the Processor, i.e.
  OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens
  on the adaptation task queue. State updates are pushed from the
  encoder task queue with PostTasks.
- QualityScalerResource operates on both task queues; the QP usage
  callbacks are invoked asynchronously.
- The VideoStreamEncoderResourceManager operates on the encoder task
  queue with the following exceptions:
  1) Its resources are accessible on any thread (using a mutex). This
     is OK because resources are reference counted and thread safe.
     This aids adding and removing resources to the Processor on the
     adaptation task queue.
  2) |active_counts_| is moved to the adaptation task queue. This makes
     it possible for PreventAdaptUpDueToActiveCounts to run
     IsAdaptationUpAllowed() on the adaptation task queue.
     A side-effect of this is that some stats reporting now happen on
     the adaptation task queue, but that is OK because
     VideoStreamEncoderObserver is thread-safe.

The Manager is updated to take the new threading model into account:
- OnFrameDroppedDueToSize() posts to the adaptation task queue to
  invoke the Processor.
- OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation
  task queue, updates |active_counts_| synchronously but posts to the
  encoder task queue to update video source restrictions (which it
  only uses to calculate target frame rate).
- MaybePerformQualityRampupExperiment() posts to the adaptation task
  queue to maybe reset video source restrictions on the Processor.
  |quality_rampup_done_| is made std::atomic.

Bug: webrtc:11542, webrtc:11520
Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31231}
This commit is contained in:
Henrik Boström 2020-05-12 18:49:07 +02:00 committed by Commit Bot
parent 1190f49aea
commit 381d10963a
15 changed files with 933 additions and 559 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
call/adaptation/BUILD.gn
View File

@ -60,10 +60,14 @@ if (rtc_include_tests) {
":resource_adaptation",
":resource_adaptation_test_utilities",
"../../api:scoped_refptr",
"../../api/task_queue:default_task_queue_factory",
"../../api/task_queue:task_queue",
"../../api/video:video_adaptation",
"../../api/video_codecs:video_codecs_api",
"../../rtc_base:checks",
"../../rtc_base:rtc_base_approved",
"../../rtc_base:rtc_task_queue",
"../../rtc_base:task_queue_for_test",
"../../test:field_trial",
"../../test:rtc_expect_death",
"../../test:test_support",

32
call/adaptation/resource.cc
View File

@ -17,14 +17,30 @@ namespace webrtc {
ResourceListener::~ResourceListener() {}
Resource::Resource() : usage_state_(absl::nullopt), listener_(nullptr) {}
Resource::Resource()
: encoder_queue_(nullptr),
resource_adaptation_queue_(nullptr),
usage_state_(absl::nullopt),
listener_(nullptr) {}
Resource::~Resource() {
RTC_DCHECK(!listener_)
<< "There is a listener depending on a Resource being destroyed.";
}
void Resource::Initialize(rtc::TaskQueue* encoder_queue,
rtc::TaskQueue* resource_adaptation_queue) {
RTC_DCHECK(!encoder_queue_);
RTC_DCHECK(encoder_queue);
RTC_DCHECK(!resource_adaptation_queue_);
RTC_DCHECK(resource_adaptation_queue);
encoder_queue_ = encoder_queue;
resource_adaptation_queue_ = resource_adaptation_queue;
}
void Resource::SetResourceListener(ResourceListener* listener) {
RTC_DCHECK(resource_adaptation_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
// If you want to change listener you need to unregister the old listener by
// setting it to null first.
RTC_DCHECK(!listener_ || !listener) << "A listener is already set";
@ -32,10 +48,14 @@ void Resource::SetResourceListener(ResourceListener* listener) {
}
absl::optional<ResourceUsageState> Resource::usage_state() const {
RTC_DCHECK(resource_adaptation_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
return usage_state_;
}
void Resource::ClearUsageState() {
RTC_DCHECK(resource_adaptation_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
usage_state_ = absl::nullopt;
}
@ -53,7 +73,17 @@ void Resource::OnAdaptationApplied(
const VideoSourceRestrictions& restrictions_after,
rtc::scoped_refptr<Resource> reason_resource) {}
rtc::TaskQueue* Resource::encoder_queue() const {
return encoder_queue_;
}
rtc::TaskQueue* Resource::resource_adaptation_queue() const {
return resource_adaptation_queue_;
}
void Resource::OnResourceUsageStateMeasured(ResourceUsageState usage_state) {
RTC_DCHECK(resource_adaptation_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
usage_state_ = usage_state;
if (!listener_)
return;

14
call/adaptation/resource.h
View File

@ -19,6 +19,7 @@
#include "call/adaptation/video_source_restrictions.h"
#include "call/adaptation/video_stream_input_state.h"
#include "rtc_base/ref_count.h"
#include "rtc_base/task_queue.h"
namespace webrtc {
@ -47,6 +48,9 @@ class Resource : public rtc::RefCountInterface {
Resource();
~Resource() override;
void Initialize(rtc::TaskQueue* encoder_queue,
rtc::TaskQueue* resource_adaptation_queue);
void SetResourceListener(ResourceListener* listener);
absl::optional<ResourceUsageState> usage_state() const;
@ -69,12 +73,18 @@ class Resource : public rtc::RefCountInterface {
virtual std::string name() const = 0;
protected:
rtc::TaskQueue* encoder_queue() const;
rtc::TaskQueue* resource_adaptation_queue() const;
// Updates the usage state and informs all registered listeners.
void OnResourceUsageStateMeasured(ResourceUsageState usage_state);
private:
absl::optional<ResourceUsageState> usage_state_;
ResourceListener* listener_;
rtc::TaskQueue* encoder_queue_;
rtc::TaskQueue* resource_adaptation_queue_;
absl::optional<ResourceUsageState> usage_state_
RTC_GUARDED_BY(resource_adaptation_queue_);
ResourceListener* listener_ RTC_GUARDED_BY(resource_adaptation_queue_);
};
} // namespace webrtc

463
call/adaptation/resource_adaptation_processor_unittest.cc
View File

@ -18,6 +18,8 @@
#include "call/adaptation/test/fake_resource.h"
#include "call/adaptation/video_source_restrictions.h"
#include "call/adaptation/video_stream_input_state_provider.h"
#include "rtc_base/event.h"
#include "rtc_base/task_queue_for_test.h"
#include "test/gtest.h"
namespace webrtc {
@ -67,22 +69,38 @@ class ResourceAdaptationProcessorListenerForTesting
class ResourceAdaptationProcessorTest : public ::testing::Test {
public:
ResourceAdaptationProcessorTest()
: frame_rate_provider_(),
: resource_adaptation_queue_("ResourceAdaptationQueue"),
encoder_queue_("EncoderQueue"),
frame_rate_provider_(),
input_state_provider_(&frame_rate_provider_),
resource_(new FakeResource("FakeResource")),
other_resource_(new FakeResource("OtherFakeResource")),
processor_(&input_state_provider_,
/*encoder_stats_observer=*/&frame_rate_provider_) {
processor_.InitializeOnResourceAdaptationQueue();
processor_.AddAdaptationListener(&processor_listener_);
processor_.AddResource(resource_);
processor_.AddResource(other_resource_);
processor_(std::make_unique<ResourceAdaptationProcessor>(
&input_state_provider_,
/*encoder_stats_observer=*/&frame_rate_provider_)) {
resource_->Initialize(&encoder_queue_, &resource_adaptation_queue_);
other_resource_->Initialize(&encoder_queue_, &resource_adaptation_queue_);
rtc::Event event;
resource_adaptation_queue_.PostTask([this, &event] {
processor_->InitializeOnResourceAdaptationQueue();
processor_->AddAdaptationListener(&processor_listener_);
processor_->AddResource(resource_);
processor_->AddResource(other_resource_);
event.Set();
});
event.Wait(rtc::Event::kForever);
}
~ResourceAdaptationProcessorTest() override {
processor_.StopResourceAdaptation();
processor_.RemoveResource(resource_);
processor_.RemoveResource(other_resource_);
processor_.RemoveAdaptationListener(&processor_listener_);
rtc::Event event;
resource_adaptation_queue_.PostTask([this, &event] {
processor_->StopResourceAdaptation();
processor_->RemoveResource(resource_);
processor_->RemoveResource(other_resource_);
processor_->RemoveAdaptationListener(&processor_listener_);
processor_.reset();
event.Set();
});
event.Wait(rtc::Event::kForever);
}
void SetInputStates(bool has_input, int fps, int frame_size) {
@ -100,42 +118,52 @@ class ResourceAdaptationProcessorTest : public ::testing::Test {
}
protected:
TaskQueueForTest resource_adaptation_queue_;
TaskQueueForTest encoder_queue_;
FakeFrameRateProvider frame_rate_provider_;
VideoStreamInputStateProvider input_state_provider_;
rtc::scoped_refptr<FakeResource> resource_;
rtc::scoped_refptr<FakeResource> other_resource_;
ResourceAdaptationProcessor processor_;
std::unique_ptr<ResourceAdaptationProcessor> processor_;
ResourceAdaptationProcessorListenerForTesting processor_listener_;
};
} // namespace
TEST_F(ResourceAdaptationProcessorTest, DisabledByDefault) {
EXPECT_EQ(DegradationPreference::DISABLED,
processor_.degradation_preference());
EXPECT_EQ(DegradationPreference::DISABLED,
processor_.effective_degradation_preference());
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
processor_.StartResourceAdaptation();
// Adaptation does not happen when disabled.
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
resource_adaptation_queue_.SendTask(
[this] {
EXPECT_EQ(DegradationPreference::DISABLED,
processor_->degradation_preference());
EXPECT_EQ(DegradationPreference::DISABLED,
processor_->effective_degradation_preference());
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
processor_->StartResourceAdaptation();
// Adaptation does not happen when disabled.
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest, InsufficientInput) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
// Adaptation does not happen if input is insufficient.
// When frame size is missing (OnFrameSizeObserved not called yet).
input_state_provider_.OnHasInputChanged(true);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
// When "has input" is missing.
SetInputStates(false, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
// Note: frame rate cannot be missing, if unset it is 0.
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
// Adaptation does not happen if input is insufficient.
// When frame size is missing (OnFrameSizeObserved not called yet).
input_state_provider_.OnHasInputChanged(true);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
// When "has input" is missing.
SetInputStates(false, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
// Note: frame rate cannot be missing, if unset it is 0.
},
RTC_FROM_HERE);
}
// These tests verify that restrictions are applied, but not exactly how much
@ -144,212 +172,273 @@ TEST_F(ResourceAdaptationProcessorTest, InsufficientInput) {
// restrictions. For that, see video_stream_adapter_unittest.cc.
TEST_F(ResourceAdaptationProcessorTest,
OveruseTriggersRestrictingResolutionInMaintainFrameRate) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
EXPECT_TRUE(
processor_listener_.restrictions().max_pixels_per_frame().has_value());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
EXPECT_TRUE(processor_listener_.restrictions()
.max_pixels_per_frame()
.has_value());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest,
OveruseTriggersRestrictingFrameRateInMaintainResolution) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_RESOLUTION);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
EXPECT_TRUE(processor_listener_.restrictions().max_frame_rate().has_value());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_RESOLUTION);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
EXPECT_TRUE(
processor_listener_.restrictions().max_frame_rate().has_value());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest,
OveruseTriggersRestrictingFrameRateAndResolutionInBalanced) {
processor_.SetDegradationPreference(DegradationPreference::BALANCED);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
// Adapting multiple times eventually resticts both frame rate and resolution.
// Exactly many times we need to adapt depends on BalancedDegradationSettings,
// VideoStreamAdapter and default input states. This test requires it to be
// achieved within 4 adaptations.
for (size_t i = 0; i < 4; ++i) {
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(i + 1, processor_listener_.restrictions_updated_count());
RestrictSource(processor_listener_.restrictions());
}
EXPECT_TRUE(
processor_listener_.restrictions().max_pixels_per_frame().has_value());
EXPECT_TRUE(processor_listener_.restrictions().max_frame_rate().has_value());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(DegradationPreference::BALANCED);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
// Adapting multiple times eventually resticts both frame rate and
// resolution. Exactly many times we need to adapt depends on
// BalancedDegradationSettings, VideoStreamAdapter and default input
// states. This test requires it to be achieved within 4 adaptations.
for (size_t i = 0; i < 4; ++i) {
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(i + 1, processor_listener_.restrictions_updated_count());
RestrictSource(processor_listener_.restrictions());
}
EXPECT_TRUE(processor_listener_.restrictions()
.max_pixels_per_frame()
.has_value());
EXPECT_TRUE(
processor_listener_.restrictions().max_frame_rate().has_value());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest, AwaitingPreviousAdaptation) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
// If we don't restrict the source then adaptation will not happen again due
// to "awaiting previous adaptation". This prevents "double-adapt".
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
// If we don't restrict the source then adaptation will not happen again
// due to "awaiting previous adaptation". This prevents "double-adapt".
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest, CannotAdaptUpWhenUnrestricted) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest, UnderuseTakesUsBackToUnrestricted) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
RestrictSource(processor_listener_.restrictions());
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(2u, processor_listener_.restrictions_updated_count());
EXPECT_EQ(VideoSourceRestrictions(), processor_listener_.restrictions());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
RestrictSource(processor_listener_.restrictions());
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(2u, processor_listener_.restrictions_updated_count());
EXPECT_EQ(VideoSourceRestrictions(),
processor_listener_.restrictions());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest, ResourcesCanPreventAdaptingUp) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
// Adapt down so that we can adapt up.
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
RestrictSource(processor_listener_.restrictions());
// Adapting up is prevented.
resource_->set_is_adaptation_up_allowed(false);
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
// Adapt down so that we can adapt up.
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
RestrictSource(processor_listener_.restrictions());
// Adapting up is prevented.
resource_->set_is_adaptation_up_allowed(false);
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest,
ResourcesCanNotAdaptUpIfNeverAdaptedDown) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
RestrictSource(processor_listener_.restrictions());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
RestrictSource(processor_listener_.restrictions());
// Other resource signals under-use
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
// Other resource signals under-use
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest,
ResourcesCanNotAdaptUpIfNotAdaptedDownAfterReset) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
processor_.ResetVideoSourceRestrictions();
EXPECT_EQ(0, processor_listener_.adaptation_counters().Total());
other_resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
processor_->ResetVideoSourceRestrictions();
EXPECT_EQ(0, processor_listener_.adaptation_counters().Total());
other_resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
// resource_ did not overuse after we reset the restrictions, so adapt up
// should be disallowed.
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
// resource_ did not overuse after we reset the restrictions, so adapt
// up should be disallowed.
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest,
MultipleResourcesCanTriggerMultipleAdaptations) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(2, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(3, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(2, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(3, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(2, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
// Does not trigger adaptation since resource has no adaptations left.
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(2, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(2, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
// Does not trigger adaptation since resource has no adaptations left.
resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(2, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(0, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
EXPECT_EQ(0, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest, AdaptingTriggersOnAdaptationApplied) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, resource_->num_adaptations_applied());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, resource_->num_adaptations_applied());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest, AdaptingClearsResourceUsageState) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
EXPECT_FALSE(resource_->usage_state().has_value());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(1u, processor_listener_.restrictions_updated_count());
EXPECT_FALSE(resource_->usage_state().has_value());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest,
FailingAdaptingAlsoClearsResourceUsageState) {
processor_.SetDegradationPreference(DegradationPreference::DISABLED);
processor_.StartResourceAdaptation();
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
EXPECT_FALSE(resource_->usage_state().has_value());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(DegradationPreference::DISABLED);
processor_->StartResourceAdaptation();
resource_->set_usage_state(ResourceUsageState::kOveruse);
EXPECT_EQ(0u, processor_listener_.restrictions_updated_count());
EXPECT_FALSE(resource_->usage_state().has_value());
},
RTC_FROM_HERE);
}
TEST_F(ResourceAdaptationProcessorTest,
AdaptsDownWhenOtherResourceIsAlwaysUnderused) {
processor_.SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_.StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
// Does not trigger adapataion because there's no restriction.
EXPECT_EQ(0, processor_listener_.adaptation_counters().Total());
resource_adaptation_queue_.SendTask(
[this] {
processor_->SetDegradationPreference(
DegradationPreference::MAINTAIN_FRAMERATE);
processor_->StartResourceAdaptation();
SetInputStates(true, kDefaultFrameRate, kDefaultFrameSize);
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
// Does not trigger adapataion because there's no restriction.
EXPECT_EQ(0, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
resource_->set_usage_state(ResourceUsageState::kOveruse);
// Adapts down even if other resource asked for adapting up.
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
resource_->set_usage_state(ResourceUsageState::kOveruse);
// Adapts down even if other resource asked for adapting up.
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
// Doesn't adapt up because adaptation is due to another resource.
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
RestrictSource(processor_listener_.restrictions());
other_resource_->set_usage_state(ResourceUsageState::kUnderuse);
// Doesn't adapt up because adaptation is due to another resource.
EXPECT_EQ(1, processor_listener_.adaptation_counters().Total());
RestrictSource(processor_listener_.restrictions());
},
RTC_FROM_HERE);
}
} // namespace webrtc

65
call/adaptation/resource_unittest.cc
View File

@ -10,8 +10,12 @@
#include "call/adaptation/resource.h"
#include <memory>
#include "api/scoped_refptr.h"
#include "call/adaptation/test/fake_resource.h"
#include "rtc_base/event.h"
#include "rtc_base/task_queue_for_test.h"
#include "test/gmock.h"
#include "test/gtest.h"
@ -27,28 +31,49 @@ class MockResourceListener : public ResourceListener {
(rtc::scoped_refptr<Resource> resource));
};
TEST(ResourceTest, RegisteringListenerReceivesCallbacks) {
StrictMock<MockResourceListener> resource_listener;
rtc::scoped_refptr<FakeResource> fake_resource(
new FakeResource("FakeResource"));
fake_resource->SetResourceListener(&resource_listener);
EXPECT_CALL(resource_listener, OnResourceUsageStateMeasured(_))
.Times(1)
.WillOnce([](rtc::scoped_refptr<Resource> resource) {
EXPECT_EQ(ResourceUsageState::kOveruse, resource->usage_state());
});
fake_resource->set_usage_state(ResourceUsageState::kOveruse);
fake_resource->SetResourceListener(nullptr);
class ResourceTest : public ::testing::Test {
public:
ResourceTest()
: resource_adaptation_queue_("ResourceAdaptationQueue"),
encoder_queue_("EncoderQueue"),
fake_resource_(new FakeResource("FakeResource")) {
fake_resource_->Initialize(&encoder_queue_, &resource_adaptation_queue_);
}
protected:
const std::unique_ptr<TaskQueueFactory> task_queue_factory_;
TaskQueueForTest resource_adaptation_queue_;
TaskQueueForTest encoder_queue_;
rtc::scoped_refptr<FakeResource> fake_resource_;
};
TEST_F(ResourceTest, RegisteringListenerReceivesCallbacks) {
resource_adaptation_queue_.SendTask(
[this] {
StrictMock<MockResourceListener> resource_listener;
fake_resource_->SetResourceListener(&resource_listener);
EXPECT_CALL(resource_listener, OnResourceUsageStateMeasured(_))
.Times(1)
.WillOnce([](rtc::scoped_refptr<Resource> resource) {
EXPECT_EQ(ResourceUsageState::kOveruse, resource->usage_state());
});
fake_resource_->set_usage_state(ResourceUsageState::kOveruse);
fake_resource_->SetResourceListener(nullptr);
},
RTC_FROM_HERE);
}
TEST(ResourceTest, UnregisteringListenerStopsCallbacks) {
StrictMock<MockResourceListener> resource_listener;
rtc::scoped_refptr<FakeResource> fake_resource(
new FakeResource("FakeResource"));
fake_resource->SetResourceListener(&resource_listener);
fake_resource->SetResourceListener(nullptr);
EXPECT_CALL(resource_listener, OnResourceUsageStateMeasured(_)).Times(0);
fake_resource->set_usage_state(ResourceUsageState::kOveruse);
TEST_F(ResourceTest, UnregisteringListenerStopsCallbacks) {
resource_adaptation_queue_.SendTask(
[this] {
StrictMock<MockResourceListener> resource_listener;
fake_resource_->SetResourceListener(&resource_listener);
fake_resource_->SetResourceListener(nullptr);
EXPECT_CALL(resource_listener, OnResourceUsageStateMeasured(_))
.Times(0);
fake_resource_->set_usage_state(ResourceUsageState::kOveruse);
},
RTC_FROM_HERE);
}
} // namespace webrtc

45
video/adaptation/encode_usage_resource.cc
View File

@ -21,26 +21,19 @@ namespace webrtc {
EncodeUsageResource::EncodeUsageResource(
std::unique_ptr<OveruseFrameDetector> overuse_detector)
: rtc::RefCountedObject<Resource>(),
encoder_queue_(nullptr),
overuse_detector_(std::move(overuse_detector)),
is_started_(false),
target_frame_rate_(absl::nullopt) {
RTC_DCHECK(overuse_detector_);
}
void EncodeUsageResource::Initialize(rtc::TaskQueue* encoder_queue) {
RTC_DCHECK(!encoder_queue_);
RTC_DCHECK(encoder_queue);
encoder_queue_ = encoder_queue;
}
bool EncodeUsageResource::is_started() const {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
return is_started_;
}
void EncodeUsageResource::StartCheckForOveruse(CpuOveruseOptions options) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
RTC_DCHECK(!is_started_);
overuse_detector_->StartCheckForOveruse(TaskQueueBase::Current(),
std::move(options), this);
@ -49,14 +42,14 @@ void EncodeUsageResource::StartCheckForOveruse(CpuOveruseOptions options) {
}
void EncodeUsageResource::StopCheckForOveruse() {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
overuse_detector_->StopCheckForOveruse();
is_started_ = false;
}
void EncodeUsageResource::SetTargetFrameRate(
absl::optional<double> target_frame_rate) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
if (target_frame_rate == target_frame_rate_)
return;
target_frame_rate_ = target_frame_rate;
@ -66,7 +59,7 @@ void EncodeUsageResource::SetTargetFrameRate(
void EncodeUsageResource::OnEncodeStarted(const VideoFrame& cropped_frame,
int64_t time_when_first_seen_us) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
// TODO(hbos): Rename FrameCaptured() to something more appropriate (e.g.
// "OnEncodeStarted"?) or revise usage.
overuse_detector_->FrameCaptured(cropped_frame, time_when_first_seen_us);
@ -77,7 +70,7 @@ void EncodeUsageResource::OnEncodeCompleted(
int64_t time_sent_in_us,
int64_t capture_time_us,
absl::optional<int> encode_duration_us) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
// TODO(hbos): Rename FrameSent() to something more appropriate (e.g.
// "OnEncodeCompleted"?).
overuse_detector_->FrameSent(timestamp, time_sent_in_us, capture_time_us,
@ -85,21 +78,29 @@ void EncodeUsageResource::OnEncodeCompleted(
}
void EncodeUsageResource::AdaptUp() {
RTC_DCHECK_RUN_ON(encoder_queue_);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// PostTask the resource usage measurements.
OnResourceUsageStateMeasured(ResourceUsageState::kUnderuse);
RTC_DCHECK_RUN_ON(encoder_queue());
// Reference counting guarantees that this object is still alive by the time
// the task is executed.
resource_adaptation_queue()->PostTask(
[this_ref = rtc::scoped_refptr<EncodeUsageResource>(this)] {
RTC_DCHECK_RUN_ON(this_ref->resource_adaptation_queue());
this_ref->OnResourceUsageStateMeasured(ResourceUsageState::kUnderuse);
});
}
void EncodeUsageResource::AdaptDown() {
RTC_DCHECK_RUN_ON(encoder_queue_);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// PostTask the resource usage measurements.
OnResourceUsageStateMeasured(ResourceUsageState::kOveruse);
RTC_DCHECK_RUN_ON(encoder_queue());
// Reference counting guarantees that this object is still alive by the time
// the task is executed.
resource_adaptation_queue()->PostTask(
[this_ref = rtc::scoped_refptr<EncodeUsageResource>(this)] {
RTC_DCHECK_RUN_ON(this_ref->resource_adaptation_queue());
this_ref->OnResourceUsageStateMeasured(ResourceUsageState::kOveruse);
});
}
int EncodeUsageResource::TargetFrameRateAsInt() {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
return target_frame_rate_.has_value()
? static_cast<int>(target_frame_rate_.value())
: std::numeric_limits<int>::max();

11
video/adaptation/encode_usage_resource.h
View File

@ -34,10 +34,6 @@ class EncodeUsageResource : public rtc::RefCountedObject<Resource>,
explicit EncodeUsageResource(
std::unique_ptr<OveruseFrameDetector> overuse_detector);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// pass it in here.
void Initialize(rtc::TaskQueue* encoder_queue);
bool is_started() const;
void StartCheckForOveruse(CpuOveruseOptions options);
@ -60,11 +56,10 @@ class EncodeUsageResource : public rtc::RefCountedObject<Resource>,
private:
int TargetFrameRateAsInt();
rtc::TaskQueue* encoder_queue_;
const std::unique_ptr<OveruseFrameDetector> overuse_detector_
RTC_GUARDED_BY(encoder_queue_);
bool is_started_ RTC_GUARDED_BY(encoder_queue_);
absl::optional<double> target_frame_rate_ RTC_GUARDED_BY(encoder_queue_);
RTC_GUARDED_BY(encoder_queue());
bool is_started_ RTC_GUARDED_BY(encoder_queue());
absl::optional<double> target_frame_rate_ RTC_GUARDED_BY(encoder_queue());
};
} // namespace webrtc

117
video/adaptation/quality_scaler_resource.cc
View File

@ -18,12 +18,10 @@ namespace webrtc {
QualityScalerResource::QualityScalerResource()
: rtc::RefCountedObject<Resource>(),
encoder_queue_(nullptr),
adaptation_processor_(nullptr),
quality_scaler_(nullptr),
num_handled_callbacks_(0),
pending_callbacks_(),
processing_in_progress_(false),
adaptation_processor_(nullptr),
clear_qp_samples_(false) {}
QualityScalerResource::~QualityScalerResource() {
@ -31,33 +29,27 @@ QualityScalerResource::~QualityScalerResource() {
RTC_DCHECK(pending_callbacks_.empty());
}
void QualityScalerResource::Initialize(rtc::TaskQueue* encoder_queue) {
RTC_DCHECK(!encoder_queue_);
RTC_DCHECK(encoder_queue);
encoder_queue_ = encoder_queue;
}
void QualityScalerResource::SetAdaptationProcessor(
ResourceAdaptationProcessorInterface* adaptation_processor) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue());
adaptation_processor_ = adaptation_processor;
}
bool QualityScalerResource::is_started() const {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
return quality_scaler_.get();
}
void QualityScalerResource::StartCheckForOveruse(
VideoEncoder::QpThresholds qp_thresholds) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
RTC_DCHECK(!is_started());
quality_scaler_ =
std::make_unique<QualityScaler>(this, std::move(qp_thresholds));
}
void QualityScalerResource::StopCheckForOveruse() {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
// Ensure we have no pending callbacks. This makes it safe to destroy the
// QualityScaler and even task queues with tasks in-flight.
AbortPendingCallbacks();
@ -66,35 +58,41 @@ void QualityScalerResource::StopCheckForOveruse() {
void QualityScalerResource::SetQpThresholds(
VideoEncoder::QpThresholds qp_thresholds) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
RTC_DCHECK(is_started());
quality_scaler_->SetQpThresholds(std::move(qp_thresholds));
}
bool QualityScalerResource::QpFastFilterLow() {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
RTC_DCHECK(is_started());
return quality_scaler_->QpFastFilterLow();
}
void QualityScalerResource::OnEncodeCompleted(const EncodedImage& encoded_image,
int64_t time_sent_in_us) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
if (quality_scaler_ && encoded_image.qp_ >= 0) {
quality_scaler_->ReportQp(encoded_image.qp_, time_sent_in_us);
} else if (!quality_scaler_) {
// Reference counting guarantees that this object is still alive by the time
// the task is executed.
// TODO(webrtc:11553): this is a workaround to ensure that all quality
// scaler imposed limitations are removed once qualty scaler is disabled
// mid call.
// Instead it should be done at a higher layer in the same way for all
// resources.
OnResourceUsageStateMeasured(ResourceUsageState::kUnderuse);
resource_adaptation_queue()->PostTask(
[this_ref = rtc::scoped_refptr<QualityScalerResource>(this)] {
RTC_DCHECK_RUN_ON(this_ref->resource_adaptation_queue());
this_ref->OnResourceUsageStateMeasured(ResourceUsageState::kUnderuse);
});
}
}
void QualityScalerResource::OnFrameDropped(
EncodedImageCallback::DropReason reason) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
if (!quality_scaler_)
return;
switch (reason) {
@ -109,29 +107,37 @@ void QualityScalerResource::OnFrameDropped(
void QualityScalerResource::OnReportQpUsageHigh(
rtc::scoped_refptr<QualityScalerQpUsageHandlerCallbackInterface> callback) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
size_t callback_id = QueuePendingCallback(callback);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// PostTask the resource usage measurements.
RTC_DCHECK(!processing_in_progress_);
processing_in_progress_ = true;
clear_qp_samples_ = false;
// If this OnResourceUsageStateMeasured() triggers an adaptation,
// OnAdaptationApplied() will occur between this line and the next. This
// allows modifying |clear_qp_samples_| based on the adaptation.
OnResourceUsageStateMeasured(ResourceUsageState::kOveruse);
HandlePendingCallback(callback_id, clear_qp_samples_);
processing_in_progress_ = false;
// Reference counting guarantees that this object is still alive by the time
// the task is executed.
resource_adaptation_queue()->PostTask(
[this_ref = rtc::scoped_refptr<QualityScalerResource>(this),
callback_id] {
RTC_DCHECK_RUN_ON(this_ref->resource_adaptation_queue());
this_ref->clear_qp_samples_ = false;
// If this OnResourceUsageStateMeasured() triggers an adaptation,
// OnAdaptationApplied() will occur between this line and the next. This
// allows modifying |clear_qp_samples_| based on the adaptation.
this_ref->OnResourceUsageStateMeasured(ResourceUsageState::kOveruse);
this_ref->HandlePendingCallback(callback_id,
this_ref->clear_qp_samples_);
});
}
void QualityScalerResource::OnReportQpUsageLow(
rtc::scoped_refptr<QualityScalerQpUsageHandlerCallbackInterface> callback) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
size_t callback_id = QueuePendingCallback(callback);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// PostTask the resource usage measurements.
OnResourceUsageStateMeasured(ResourceUsageState::kUnderuse);
HandlePendingCallback(callback_id, true);
// Reference counting guarantees that this object is still alive by the time
// the task is executed.
resource_adaptation_queue()->PostTask(
[this_ref = rtc::scoped_refptr<QualityScalerResource>(this),
callback_id] {
RTC_DCHECK_RUN_ON(this_ref->resource_adaptation_queue());
this_ref->OnResourceUsageStateMeasured(ResourceUsageState::kUnderuse);
this_ref->HandlePendingCallback(callback_id, true);
});
}
void QualityScalerResource::OnAdaptationApplied(
@ -139,9 +145,9 @@ void QualityScalerResource::OnAdaptationApplied(
const VideoSourceRestrictions& restrictions_before,
const VideoSourceRestrictions& restrictions_after,
rtc::scoped_refptr<Resource> reason_resource) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue());
// We only clear QP samples on adaptations triggered by the QualityScaler.
if (!processing_in_progress_)
if (reason_resource != this)
return;
clear_qp_samples_ = true;
// If we're in "balanced" and the frame rate before and after adaptation did
@ -173,7 +179,7 @@ void QualityScalerResource::OnAdaptationApplied(
size_t QualityScalerResource::QueuePendingCallback(
rtc::scoped_refptr<QualityScalerQpUsageHandlerCallbackInterface> callback) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
pending_callbacks_.push(callback);
// The ID of a callback is its sequence number (1, 2, 3...).
return num_handled_callbacks_ + pending_callbacks_.size();
@ -181,24 +187,29 @@ size_t QualityScalerResource::QueuePendingCallback(
void QualityScalerResource::HandlePendingCallback(size_t callback_id,
bool clear_qp_samples) {
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// this method would be invoked on the adaptation queue and a PostTask would
// be used to resolve the callback.
RTC_DCHECK_RUN_ON(encoder_queue_);
if (num_handled_callbacks_ >= callback_id) {
// The callback with this ID has already been handled.
// This happens if AbortPendingCallbacks() is called while the task is
// in flight.
return;
}
RTC_DCHECK(!pending_callbacks_.empty());
pending_callbacks_.front()->OnQpUsageHandled(clear_qp_samples);
++num_handled_callbacks_;
pending_callbacks_.pop();
RTC_DCHECK_RUN_ON(resource_adaptation_queue());
// Reference counting guarantees that this object is still alive by the time
// the task is executed.
encoder_queue()->PostTask(
[this_ref = rtc::scoped_refptr<QualityScalerResource>(this), callback_id,
clear_qp_samples] {
RTC_DCHECK_RUN_ON(this_ref->encoder_queue());
if (this_ref->num_handled_callbacks_ >= callback_id) {
// The callback with this ID has already been handled.
// This happens if AbortPendingCallbacks() is called while the task is
// in flight.
return;
}
RTC_DCHECK(!this_ref->pending_callbacks_.empty());
this_ref->pending_callbacks_.front()->OnQpUsageHandled(
clear_qp_samples);
++this_ref->num_handled_callbacks_;
this_ref->pending_callbacks_.pop();
});
}
void QualityScalerResource::AbortPendingCallbacks() {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(encoder_queue());
while (!pending_callbacks_.empty()) {
pending_callbacks_.front()->OnQpUsageHandled(false);
++num_handled_callbacks_;

34
video/adaptation/quality_scaler_resource.h
View File

@ -33,9 +33,6 @@ class QualityScalerResource : public rtc::RefCountedObject<Resource>,
QualityScalerResource();
~QualityScalerResource() override;
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// pass it in here.
void Initialize(rtc::TaskQueue* encoder_queue);
void SetAdaptationProcessor(
ResourceAdaptationProcessorInterface* adaptation_processor);
@ -74,25 +71,22 @@ class QualityScalerResource : public rtc::RefCountedObject<Resource>,
void HandlePendingCallback(size_t callback_id, bool clear_qp_samples);
void AbortPendingCallbacks();
rtc::TaskQueue* encoder_queue_;
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// guard the processor by it instead.
ResourceAdaptationProcessorInterface* adaptation_processor_
RTC_GUARDED_BY(encoder_queue_);
std::unique_ptr<QualityScaler> quality_scaler_ RTC_GUARDED_BY(encoder_queue_);
// Members accessed on the encoder queue.
std::unique_ptr<QualityScaler> quality_scaler_
RTC_GUARDED_BY(encoder_queue());
// Every OnReportQpUsageHigh/Low() operation has a callback that MUST be
// invoked on the |encoder_queue_|.
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// handling a measurement entails a task queue "ping" round-trip between the
// encoder queue and the adaptation queue. Multiple callbacks in-flight would
// then be possible.
size_t num_handled_callbacks_ RTC_GUARDED_BY(encoder_queue_);
// invoked on the |encoder_queue_|. Because usage measurements are reported on
// the |encoder_queue_| but handled by the processor on the the
// |resource_adaptation_queue_|, handling a measurement entails a task queue
// "ping" round-trip. Multiple callbacks in-flight is thus possible.
size_t num_handled_callbacks_ RTC_GUARDED_BY(encoder_queue());
std::queue<rtc::scoped_refptr<QualityScalerQpUsageHandlerCallbackInterface>>
pending_callbacks_ RTC_GUARDED_BY(encoder_queue_);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// guard processing_in_progress_/clear_cp_samples_ by it instead.
bool processing_in_progress_ RTC_GUARDED_BY(encoder_queue_);
bool clear_qp_samples_ RTC_GUARDED_BY(encoder_queue_);
pending_callbacks_ RTC_GUARDED_BY(encoder_queue());
// Members accessed on the adaptation queue.
ResourceAdaptationProcessorInterface* adaptation_processor_
RTC_GUARDED_BY(resource_adaptation_queue());
bool clear_qp_samples_ RTC_GUARDED_BY(resource_adaptation_queue());
};
} // namespace webrtc

13
video/adaptation/quality_scaler_resource_unittest.cc
View File

@ -68,11 +68,15 @@ class QualityScalerResourceTest : public ::testing::Test {
public:
QualityScalerResourceTest()
: task_queue_factory_(CreateDefaultTaskQueueFactory()),
resource_adaptation_queue_(task_queue_factory_->CreateTaskQueue(
"ResourceAdaptationQueue",
TaskQueueFactory::Priority::NORMAL)),
encoder_queue_(task_queue_factory_->CreateTaskQueue(
"EncoderQueue",
TaskQueueFactory::Priority::NORMAL)),
quality_scaler_resource_(new QualityScalerResource()) {
quality_scaler_resource_->Initialize(&encoder_queue_);
quality_scaler_resource_->Initialize(&encoder_queue_,
&resource_adaptation_queue_);
rtc::Event event;
encoder_queue_.PostTask([this, &event] {
quality_scaler_resource_->StartCheckForOveruse(
@ -93,6 +97,7 @@ class QualityScalerResourceTest : public ::testing::Test {
protected:
const std::unique_ptr<TaskQueueFactory> task_queue_factory_;
rtc::TaskQueue resource_adaptation_queue_;
rtc::TaskQueue encoder_queue_;
rtc::scoped_refptr<QualityScalerResource> quality_scaler_resource_;
};
@ -115,9 +120,6 @@ TEST_F(QualityScalerResourceTest, ReportQpLow) {
callback->qp_usage_handled_event()->Wait(kDefaultTimeout);
}
// TODO(https://crbug.com/webrtc/11542): Callbacks are currently resolved
// immediately, but when we have an adaptation queue this test will ensure we
// can have multiple callbacks pending at the same time.
TEST_F(QualityScalerResourceTest, MultipleCallbacksInFlight) {
rtc::scoped_refptr<FakeQualityScalerQpUsageHandlerCallback> callback1 =
new FakeQualityScalerQpUsageHandlerCallback(&encoder_queue_);
@ -135,9 +137,6 @@ TEST_F(QualityScalerResourceTest, MultipleCallbacksInFlight) {
callback3->qp_usage_handled_event()->Wait(kDefaultTimeout);
}
// TODO(https://crbug.com/webrtc/11542): Callbacks are currently resolved
// immediately, but when we have an adaptation queue this test will ensure we
// can abort pending callbacks.
TEST_F(QualityScalerResourceTest, AbortPendingCallbacksAndStartAgain) {
rtc::scoped_refptr<FakeQualityScalerQpUsageHandlerCallback> callback1 =
new FakeQualityScalerQpUsageHandlerCallback(&encoder_queue_);

180
video/adaptation/video_stream_encoder_resource_manager.cc
View File

@ -147,6 +147,7 @@ VideoStreamEncoderResourceManager::PreventAdaptUpDueToActiveCounts::
void VideoStreamEncoderResourceManager::PreventAdaptUpDueToActiveCounts::
SetAdaptationProcessor(
ResourceAdaptationProcessorInterface* adaptation_processor) {
RTC_DCHECK_RUN_ON(resource_adaptation_queue());
adaptation_processor_ = adaptation_processor;
}
@ -155,26 +156,31 @@ bool VideoStreamEncoderResourceManager::PreventAdaptUpDueToActiveCounts::
const VideoSourceRestrictions& restrictions_before,
const VideoSourceRestrictions& restrictions_after,
rtc::scoped_refptr<Resource> reason_resource) const {
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// ensure that this is running on it instead.
RTC_DCHECK_RUN_ON(manager_->encoder_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue());
RTC_DCHECK(adaptation_processor_);
VideoAdaptationReason reason =
manager_->GetReasonFromResource(reason_resource);
// We can't adapt up if we're already at the highest setting.
// Note that this only includes counts relevant to the current degradation
// preference. e.g. we previously adapted resolution, now prefer adpating fps,
// only count the fps adaptations and not the previous resolution adaptations.
// TODO(hbos): Why would the reason matter? If a particular resource doesn't
// want us to go up it should prevent us from doing so itself rather than to
// have this catch-all reason- and stats-based approach.
int num_downgrades =
FilterVideoAdaptationCountersByDegradationPreference(
manager_->active_counts_[reason],
adaptation_processor_->effective_degradation_preference())
.Total();
RTC_DCHECK_GE(num_downgrades, 0);
return num_downgrades > 0;
{
// This is the same as |resource_adaptation_queue_|, but need to
// RTC_DCHECK_RUN_ON() both to avoid compiler error when accessing
// |manager_->active_counts_|.
RTC_DCHECK_RUN_ON(manager_->resource_adaptation_queue_);
// We can't adapt up if we're already at the highest setting.
// Note that this only includes counts relevant to the current degradation
// preference. e.g. we previously adapted resolution, now prefer adpating
// fps, only count the fps adaptations and not the previous resolution
// adaptations.
// TODO(hbos): Why would the reason matter? If a particular resource doesn't
// want us to go up it should prevent us from doing so itself rather than to
// have this catch-all reason- and stats-based approach.
int num_downgrades =
FilterVideoAdaptationCountersByDegradationPreference(
manager_->active_counts_[reason],
adaptation_processor_->effective_degradation_preference())
.Total();
RTC_DCHECK_GE(num_downgrades, 0);
return num_downgrades > 0;
}
}
VideoStreamEncoderResourceManager::
@ -189,18 +195,30 @@ VideoStreamEncoderResourceManager::
void VideoStreamEncoderResourceManager::
PreventIncreaseResolutionDueToBitrateResource::OnEncoderSettingsUpdated(
absl::optional<EncoderSettings> encoder_settings) {
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// update the state in a PostTask instead.
encoder_settings_ = std::move(encoder_settings);
RTC_DCHECK_RUN_ON(encoder_queue());
resource_adaptation_queue()->PostTask(
[this_ref =
rtc::scoped_refptr<PreventIncreaseResolutionDueToBitrateResource>(
this),
encoder_settings] {
RTC_DCHECK_RUN_ON(this_ref->resource_adaptation_queue());
this_ref->encoder_settings_ = std::move(encoder_settings);
});
}
void VideoStreamEncoderResourceManager::
PreventIncreaseResolutionDueToBitrateResource::
OnEncoderTargetBitrateUpdated(
absl::optional<uint32_t> encoder_target_bitrate_bps) {
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// update the state in a PostTask instead.
encoder_target_bitrate_bps_ = encoder_target_bitrate_bps;
RTC_DCHECK_RUN_ON(encoder_queue());
resource_adaptation_queue()->PostTask(
[this_ref =
rtc::scoped_refptr<PreventIncreaseResolutionDueToBitrateResource>(
this),
encoder_target_bitrate_bps] {
RTC_DCHECK_RUN_ON(this_ref->resource_adaptation_queue());
this_ref->encoder_target_bitrate_bps_ = encoder_target_bitrate_bps;
});
}
bool VideoStreamEncoderResourceManager::
@ -209,9 +227,7 @@ bool VideoStreamEncoderResourceManager::
const VideoSourceRestrictions& restrictions_before,
const VideoSourceRestrictions& restrictions_after,
rtc::scoped_refptr<Resource> reason_resource) const {
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// ensure that this is running on it instead.
RTC_DCHECK_RUN_ON(manager_->encoder_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue());
VideoAdaptationReason reason =
manager_->GetReasonFromResource(reason_resource);
// If increasing resolution due to kQuality, make sure bitrate limits are not
@ -250,15 +266,20 @@ VideoStreamEncoderResourceManager::PreventAdaptUpInBalancedResource::
void VideoStreamEncoderResourceManager::PreventAdaptUpInBalancedResource::
SetAdaptationProcessor(
ResourceAdaptationProcessorInterface* adaptation_processor) {
RTC_DCHECK_RUN_ON(resource_adaptation_queue());
adaptation_processor_ = adaptation_processor;
}
void VideoStreamEncoderResourceManager::PreventAdaptUpInBalancedResource::
OnEncoderTargetBitrateUpdated(
absl::optional<uint32_t> encoder_target_bitrate_bps) {
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// update the state in a PostTask instead.
encoder_target_bitrate_bps_ = encoder_target_bitrate_bps;
RTC_DCHECK_RUN_ON(encoder_queue());
resource_adaptation_queue()->PostTask(
[this_ref = rtc::scoped_refptr<PreventAdaptUpInBalancedResource>(this),
encoder_target_bitrate_bps] {
RTC_DCHECK_RUN_ON(this_ref->resource_adaptation_queue());
this_ref->encoder_target_bitrate_bps_ = encoder_target_bitrate_bps;
});
}
bool VideoStreamEncoderResourceManager::PreventAdaptUpInBalancedResource::
@ -266,9 +287,7 @@ bool VideoStreamEncoderResourceManager::PreventAdaptUpInBalancedResource::
const VideoSourceRestrictions& restrictions_before,
const VideoSourceRestrictions& restrictions_after,
rtc::scoped_refptr<Resource> reason_resource) const {
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// ensure that this is running on it instead.
RTC_DCHECK_RUN_ON(manager_->encoder_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue());
RTC_DCHECK(adaptation_processor_);
VideoAdaptationReason reason =
manager_->GetReasonFromResource(reason_resource);
@ -312,6 +331,7 @@ VideoStreamEncoderResourceManager::VideoStreamEncoderResourceManager(
new EncodeUsageResource(std::move(overuse_detector))),
quality_scaler_resource_(new QualityScalerResource()),
encoder_queue_(nullptr),
resource_adaptation_queue_(nullptr),
input_state_provider_(input_state_provider),
adaptation_processor_(nullptr),
encoder_stats_observer_(encoder_stats_observer),
@ -342,17 +362,29 @@ VideoStreamEncoderResourceManager::VideoStreamEncoderResourceManager(
VideoStreamEncoderResourceManager::~VideoStreamEncoderResourceManager() {}
void VideoStreamEncoderResourceManager::Initialize(
rtc::TaskQueue* encoder_queue) {
rtc::TaskQueue* encoder_queue,
rtc::TaskQueue* resource_adaptation_queue) {
RTC_DCHECK(!encoder_queue_);
RTC_DCHECK(encoder_queue);
RTC_DCHECK(!resource_adaptation_queue_);
RTC_DCHECK(resource_adaptation_queue);
encoder_queue_ = encoder_queue;
encode_usage_resource_->Initialize(encoder_queue_);
quality_scaler_resource_->Initialize(encoder_queue_);
resource_adaptation_queue_ = resource_adaptation_queue;
prevent_adapt_up_due_to_active_counts_->Initialize(
encoder_queue_, resource_adaptation_queue_);
prevent_increase_resolution_due_to_bitrate_resource_->Initialize(
encoder_queue_, resource_adaptation_queue_);
prevent_adapt_up_in_balanced_resource_->Initialize(
encoder_queue_, resource_adaptation_queue_);
encode_usage_resource_->Initialize(encoder_queue_,
resource_adaptation_queue_);
quality_scaler_resource_->Initialize(encoder_queue_,
resource_adaptation_queue_);
}
void VideoStreamEncoderResourceManager::SetAdaptationProcessor(
ResourceAdaptationProcessorInterface* adaptation_processor) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
adaptation_processor_ = adaptation_processor;
prevent_adapt_up_due_to_active_counts_->SetAdaptationProcessor(
adaptation_processor);
@ -465,16 +497,27 @@ void VideoStreamEncoderResourceManager::SetEncoderRates(
void VideoStreamEncoderResourceManager::OnFrameDroppedDueToSize() {
RTC_DCHECK_RUN_ON(encoder_queue_);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// PostTask the request to adapt due to frame drop.
adaptation_processor_->TriggerAdaptationDueToFrameDroppedDueToSize(
quality_scaler_resource_);
// The VideoStreamEncoder makes the manager outlive the adaptation queue. This
// means that if the task gets executed, |this| has not been freed yet.
// TODO(https://crbug.com/webrtc/11565): When the manager no longer outlives
// the adaptation queue, add logic to prevent use-after-free on |this|.
resource_adaptation_queue_->PostTask([this] {
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
if (!adaptation_processor_) {
// The processor nulled before this task had a chance to execute. This
// happens if the processor is destroyed. No action needed.
return;
}
adaptation_processor_->TriggerAdaptationDueToFrameDroppedDueToSize(
quality_scaler_resource_);
});
initial_frame_dropper_->OnFrameDroppedDueToSize();
}
void VideoStreamEncoderResourceManager::OnEncodeStarted(
const VideoFrame& cropped_frame,
int64_t time_when_first_seen_us) {
RTC_DCHECK_RUN_ON(encoder_queue_);
encode_usage_resource_->OnEncodeStarted(cropped_frame,
time_when_first_seen_us);
}
@ -610,11 +653,7 @@ void VideoStreamEncoderResourceManager::OnVideoSourceRestrictionsUpdated(
VideoSourceRestrictions restrictions,
const VideoAdaptationCounters& adaptation_counters,
rtc::scoped_refptr<Resource> reason) {
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// ensure that this is running on it instead, and PostTask back to the encoder
// queue if need be.
RTC_DCHECK_RUN_ON(encoder_queue_);
video_source_restrictions_ = restrictions;
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
VideoAdaptationCounters previous_adaptation_counters =
active_counts_[VideoAdaptationReason::kQuality] +
active_counts_[VideoAdaptationReason::kCpu];
@ -638,7 +677,14 @@ void VideoStreamEncoderResourceManager::OnVideoSourceRestrictionsUpdated(
RTC_DCHECK_EQ(adaptation_counters_total_abs_diff, 0);
}
RTC_LOG(LS_INFO) << ActiveCountsToString();
MaybeUpdateTargetFrameRate();
// The VideoStreamEncoder makes the manager outlive the encoder queue. This
// means that if the task gets executed, |this| has not been freed yet.
encoder_queue_->PostTask([this, restrictions] {
RTC_DCHECK_RUN_ON(encoder_queue_);
video_source_restrictions_ = restrictions;
MaybeUpdateTargetFrameRate();
});
}
void VideoStreamEncoderResourceManager::MaybeUpdateTargetFrameRate() {
@ -728,7 +774,7 @@ void VideoStreamEncoderResourceManager::OnAdaptationCountChanged(
void VideoStreamEncoderResourceManager::UpdateAdaptationStats(
const VideoAdaptationCounters& total_counts,
VideoAdaptationReason reason) {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
// Update active counts
VideoAdaptationCounters& active_count = active_counts_[reason];
VideoAdaptationCounters& other_active = active_counts_[OtherReason(reason)];
@ -777,29 +823,43 @@ void VideoStreamEncoderResourceManager::MaybePerformQualityRampupExperiment() {
try_quality_rampup = true;
}
}
// TODO(https://crbug.com/webrtc/11392): See if we can rely on the total
// counts or the stats, and not the active counts.
const VideoAdaptationCounters& qp_counts =
active_counts_[VideoAdaptationReason::kQuality];
const VideoAdaptationCounters& cpu_counts =
active_counts_[VideoAdaptationReason::kCpu];
if (try_quality_rampup && qp_counts.resolution_adaptations > 0 &&
cpu_counts.Total() == 0) {
RTC_LOG(LS_INFO) << "Reset quality limitations.";
adaptation_processor_->ResetVideoSourceRestrictions();
quality_rampup_done_ = true;
if (try_quality_rampup) {
// The VideoStreamEncoder makes the manager outlive the adaptation queue.
// This means that if the task gets executed, |this| has not been freed yet.
// TODO(https://crbug.com/webrtc/11565): When the manager no longer outlives
// the adaptation queue, add logic to prevent use-after-free on |this|.
resource_adaptation_queue_->PostTask([this] {
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
if (!adaptation_processor_) {
// The processor nulled before this task had a chance to execute. This
// happens if the processor is destroyed. No action needed.
return;
}
// TODO(https://crbug.com/webrtc/11392): See if we can rely on the total
// counts or the stats, and not the active counts.
const VideoAdaptationCounters& qp_counts =
active_counts_[VideoAdaptationReason::kQuality];
const VideoAdaptationCounters& cpu_counts =
active_counts_[VideoAdaptationReason::kCpu];
if (!quality_rampup_done_ && qp_counts.resolution_adaptations > 0 &&
cpu_counts.Total() == 0) {
RTC_LOG(LS_INFO) << "Reset quality limitations.";
adaptation_processor_->ResetVideoSourceRestrictions();
quality_rampup_done_ = true;
}
});
}
}
void VideoStreamEncoderResourceManager::ResetActiveCounts() {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
active_counts_.clear();
active_counts_[VideoAdaptationReason::kCpu] = VideoAdaptationCounters();
active_counts_[VideoAdaptationReason::kQuality] = VideoAdaptationCounters();
}
std::string VideoStreamEncoderResourceManager::ActiveCountsToString() const {
RTC_DCHECK_RUN_ON(encoder_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
RTC_DCHECK_EQ(2, active_counts_.size());
rtc::StringBuilder ss;

36
video/adaptation/video_stream_encoder_resource_manager.h
View File

@ -11,6 +11,7 @@
#ifndef VIDEO_ADAPTATION_VIDEO_STREAM_ENCODER_RESOURCE_MANAGER_H_
#define VIDEO_ADAPTATION_VIDEO_STREAM_ENCODER_RESOURCE_MANAGER_H_
#include <atomic>
#include <map>
#include <memory>
#include <string>
@ -33,6 +34,7 @@
#include "call/adaptation/resource_adaptation_processor_interface.h"
#include "call/adaptation/video_stream_adapter.h"
#include "call/adaptation/video_stream_input_state_provider.h"
#include "rtc_base/critical_section.h"
#include "rtc_base/experiments/quality_rampup_experiment.h"
#include "rtc_base/experiments/quality_scaler_settings.h"
#include "rtc_base/strings/string_builder.h"
@ -69,9 +71,8 @@ class VideoStreamEncoderResourceManager
std::unique_ptr<OveruseFrameDetector> overuse_detector);
~VideoStreamEncoderResourceManager() override;
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// pass it in here.
void Initialize(rtc::TaskQueue* encoder_queue);
void Initialize(rtc::TaskQueue* encoder_queue,
rtc::TaskQueue* resource_adaptation_queue);
void SetAdaptationProcessor(
ResourceAdaptationProcessorInterface* adaptation_processor);
@ -200,7 +201,8 @@ class VideoStreamEncoderResourceManager
// The |manager_| must be alive as long as this resource is added to the
// ResourceAdaptationProcessor, i.e. when IsAdaptationUpAllowed() is called.
VideoStreamEncoderResourceManager* const manager_;
ResourceAdaptationProcessorInterface* adaptation_processor_;
ResourceAdaptationProcessorInterface* adaptation_processor_
RTC_GUARDED_BY(resource_adaptation_queue());
};
// Does not trigger adaptations, only prevents adapting up resolution.
@ -230,8 +232,10 @@ class VideoStreamEncoderResourceManager
// The |manager_| must be alive as long as this resource is added to the
// ResourceAdaptationProcessor, i.e. when IsAdaptationUpAllowed() is called.
VideoStreamEncoderResourceManager* const manager_;
absl::optional<EncoderSettings> encoder_settings_;
absl::optional<uint32_t> encoder_target_bitrate_bps_;
absl::optional<EncoderSettings> encoder_settings_
RTC_GUARDED_BY(resource_adaptation_queue());
absl::optional<uint32_t> encoder_target_bitrate_bps_
RTC_GUARDED_BY(resource_adaptation_queue());
};
// Does not trigger adaptations, only prevents adapting up in BALANCED.
@ -261,8 +265,10 @@ class VideoStreamEncoderResourceManager
// The |manager_| must be alive as long as this resource is added to the
// ResourceAdaptationProcessor, i.e. when IsAdaptationUpAllowed() is called.
VideoStreamEncoderResourceManager* const manager_;
ResourceAdaptationProcessorInterface* adaptation_processor_;
absl::optional<uint32_t> encoder_target_bitrate_bps_;
ResourceAdaptationProcessorInterface* adaptation_processor_
RTC_GUARDED_BY(resource_adaptation_queue());
absl::optional<uint32_t> encoder_target_bitrate_bps_
RTC_GUARDED_BY(resource_adaptation_queue());
};
const rtc::scoped_refptr<PreventAdaptUpDueToActiveCounts>
@ -275,14 +281,13 @@ class VideoStreamEncoderResourceManager
const rtc::scoped_refptr<QualityScalerResource> quality_scaler_resource_;
rtc::TaskQueue* encoder_queue_;
rtc::TaskQueue* resource_adaptation_queue_;
VideoStreamInputStateProvider* const input_state_provider_
RTC_GUARDED_BY(encoder_queue_);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// guard the processor by it instead.
ResourceAdaptationProcessorInterface* adaptation_processor_
RTC_GUARDED_BY(encoder_queue_);
VideoStreamEncoderObserver* const encoder_stats_observer_
RTC_GUARDED_BY(encoder_queue_);
RTC_GUARDED_BY(resource_adaptation_queue_);
// Thread-safe.
VideoStreamEncoderObserver* const encoder_stats_observer_;
DegradationPreference degradation_preference_ RTC_GUARDED_BY(encoder_queue_);
VideoSourceRestrictions video_source_restrictions_
@ -298,7 +303,8 @@ class VideoStreamEncoderResourceManager
RTC_GUARDED_BY(encoder_queue_);
absl::optional<VideoEncoder::RateControlParameters> encoder_rates_
RTC_GUARDED_BY(encoder_queue_);
bool quality_rampup_done_ RTC_GUARDED_BY(encoder_queue_);
// Used on both the encoder queue and resource adaptation queue.
std::atomic<bool> quality_rampup_done_;
QualityRampupExperiment quality_rampup_experiment_
RTC_GUARDED_BY(encoder_queue_);
absl::optional<EncoderSettings> encoder_settings_
@ -325,7 +331,7 @@ class VideoStreamEncoderResourceManager
// thread-safe anyway, and active counts are used by
// PreventAdaptUpDueToActiveCounts to make decisions.
std::unordered_map<VideoAdaptationReason, VideoAdaptationCounters>
active_counts_ RTC_GUARDED_BY(encoder_queue_);
active_counts_ RTC_GUARDED_BY(resource_adaptation_queue_);
};
} // namespace webrtc

116
video/video_stream_encoder.cc
View File

@ -268,19 +268,21 @@ VideoStreamEncoder::VideoStreamEncoder(
std::move(overuse_detector)),
video_source_sink_controller_(/*sink=*/this,
/*source=*/nullptr),
resource_adaptation_queue_(task_queue_factory->CreateTaskQueue(
"ResourceAdaptationQueue",
TaskQueueFactory::Priority::NORMAL)),
encoder_queue_(task_queue_factory->CreateTaskQueue(
"EncoderQueue",
TaskQueueFactory::Priority::NORMAL)) {
RTC_DCHECK(encoder_stats_observer);
RTC_DCHECK_GE(number_of_cores, 1);
stream_resource_manager_.Initialize(&encoder_queue_);
stream_resource_manager_.Initialize(&encoder_queue_,
&resource_adaptation_queue_);
rtc::Event initialize_processor_event;
encoder_queue_.PostTask([this, &initialize_processor_event] {
RTC_DCHECK_RUN_ON(&encoder_queue_);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// initialize the processor on it instead.
resource_adaptation_queue_.PostTask([this, &initialize_processor_event] {
RTC_DCHECK_RUN_ON(&resource_adaptation_queue_);
resource_adaptation_processor_->InitializeOnResourceAdaptationQueue();
stream_resource_manager_.SetAdaptationProcessor(
resource_adaptation_processor_.get());
@ -307,10 +309,11 @@ VideoStreamEncoder::~VideoStreamEncoder() {
void VideoStreamEncoder::Stop() {
RTC_DCHECK_RUN_ON(&thread_checker_);
video_source_sink_controller_.SetSource(nullptr);
encoder_queue_.PostTask([this] {
RTC_DCHECK_RUN_ON(&encoder_queue_);
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// destroy the processor on it instead.
rtc::Event shutdown_adaptation_processor_event;
resource_adaptation_queue_.PostTask([this,
&shutdown_adaptation_processor_event] {
RTC_DCHECK_RUN_ON(&resource_adaptation_queue_);
if (resource_adaptation_processor_) {
resource_adaptation_processor_->StopResourceAdaptation();
for (Resource* resource : stream_resource_manager_.MappedResources()) {
@ -322,6 +325,11 @@ void VideoStreamEncoder::Stop() {
stream_resource_manager_.SetAdaptationProcessor(nullptr);
resource_adaptation_processor_.reset();
}
shutdown_adaptation_processor_event.Set();
});
shutdown_adaptation_processor_event.Wait(rtc::Event::kForever);
encoder_queue_.PostTask([this] {
RTC_DCHECK_RUN_ON(&encoder_queue_);
stream_resource_manager_.StopManagedResources();
rate_allocator_ = nullptr;
bitrate_observer_ = nullptr;
@ -359,8 +367,10 @@ void VideoStreamEncoder::SetSource(
RTC_DCHECK_RUN_ON(&thread_checker_);
video_source_sink_controller_.SetSource(source);
input_state_provider_.OnHasInputChanged(source);
encoder_queue_.PostTask([this, degradation_preference] {
RTC_DCHECK_RUN_ON(&encoder_queue_);
// Set the degradation preference on the adaptation queue.
resource_adaptation_queue_.PostTask([this, degradation_preference] {
RTC_DCHECK_RUN_ON(&resource_adaptation_queue_);
if (!resource_adaptation_processor_) {
// The VideoStreamEncoder was stopped and the processor destroyed before
// this task had a chance to execute. No action needed.
@ -368,8 +378,11 @@ void VideoStreamEncoder::SetSource(
}
resource_adaptation_processor_->SetDegradationPreference(
degradation_preference);
stream_resource_manager_.SetDegradationPreferences(
resource_adaptation_processor_->degradation_preference());
});
// This may trigger reconfiguring the QualityScaler on the encoder queue.
encoder_queue_.PostTask([this, degradation_preference] {
RTC_DCHECK_RUN_ON(&encoder_queue_);
stream_resource_manager_.SetDegradationPreferences(degradation_preference);
if (encoder_) {
stream_resource_manager_.ConfigureQualityScaler(
encoder_->GetEncoderInfo());
@ -702,12 +715,16 @@ void VideoStreamEncoder::ReconfigureEncoder() {
// invoked later in this method.)
stream_resource_manager_.StopManagedResources();
stream_resource_manager_.StartEncodeUsageResource();
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// PostTask ensuring it is started.
if (resource_adaptation_processor_) {
resource_adaptation_queue_.PostTask([this] {
RTC_DCHECK_RUN_ON(&resource_adaptation_queue_);
if (!resource_adaptation_processor_) {
// The VideoStreamEncoder was stopped and the processor destroyed before
// this task had a chance to execute. No action needed.
return;
}
// Ensures started. If already started this is a NO-OP.
resource_adaptation_processor_->StartResourceAdaptation();
}
});
pending_encoder_creation_ = false;
}
@ -786,12 +803,19 @@ void VideoStreamEncoder::ReconfigureEncoder() {
void VideoStreamEncoder::OnEncoderSettingsChanged() {
EncoderSettings encoder_settings(encoder_->GetEncoderInfo(),
encoder_config_.Copy(), send_codec_);
resource_adaptation_processor_->SetIsScreenshare(
encoder_config_.content_type == VideoEncoderConfig::ContentType::kScreen);
stream_resource_manager_.SetDegradationPreferences(
resource_adaptation_processor_->degradation_preference());
input_state_provider_.OnEncoderSettingsChanged(encoder_settings);
stream_resource_manager_.SetEncoderSettings(encoder_settings);
input_state_provider_.OnEncoderSettingsChanged(encoder_settings);
bool is_screenshare = encoder_settings.encoder_config().content_type ==
VideoEncoderConfig::ContentType::kScreen;
resource_adaptation_queue_.PostTask([this, is_screenshare] {
RTC_DCHECK_RUN_ON(&resource_adaptation_queue_);
if (!resource_adaptation_processor_) {
// The VideoStreamEncoder was stopped and the processor destroyed before
// this task had a chance to execute. No action needed.
return;
}
resource_adaptation_processor_->SetIsScreenshare(is_screenshare);
});
}
void VideoStreamEncoder::OnFrame(const VideoFrame& video_frame) {
@ -1717,9 +1741,7 @@ void VideoStreamEncoder::OnVideoSourceRestrictionsUpdated(
VideoSourceRestrictions restrictions,
const VideoAdaptationCounters& adaptation_counters,
rtc::scoped_refptr<Resource> reason) {
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// ensure that this is running on it instead.
RTC_DCHECK_RUN_ON(&encoder_queue_);
RTC_DCHECK_RUN_ON(&resource_adaptation_queue_);
video_source_sink_controller_.SetRestrictions(std::move(restrictions));
video_source_sink_controller_.PushSourceSinkSettings();
}
@ -1989,14 +2011,23 @@ void VideoStreamEncoder::CheckForAnimatedContent(
void VideoStreamEncoder::InjectAdaptationResource(
rtc::scoped_refptr<Resource> resource,
VideoAdaptationReason reason) {
rtc::Event inject_resource_event;
encoder_queue_.PostTask([this, resource, reason, &inject_resource_event] {
rtc::Event map_resource_event;
encoder_queue_.PostTask([this, resource, reason, &map_resource_event] {
RTC_DCHECK_RUN_ON(&encoder_queue_);
stream_resource_manager_.MapResourceToReason(resource, reason);
resource_adaptation_processor_->AddResource(resource);
inject_resource_event.Set();
map_resource_event.Set();
});
map_resource_event.Wait(rtc::Event::kForever);
resource_adaptation_queue_.PostTask([this, resource] {
RTC_DCHECK_RUN_ON(&resource_adaptation_queue_);
if (!resource_adaptation_processor_) {
// The VideoStreamEncoder was stopped and the processor destroyed before
// this task had a chance to execute. No action needed.
return;
}
resource_adaptation_processor_->AddResource(resource);
});
inject_resource_event.Wait(rtc::Event::kForever);
}
rtc::scoped_refptr<QualityScalerResource>
@ -2005,4 +2036,29 @@ VideoStreamEncoder::quality_scaler_resource_for_testing() {
return stream_resource_manager_.quality_scaler_resource_for_testing();
}
void VideoStreamEncoder::AddAdaptationListenerForTesting(
ResourceAdaptationProcessorListener* adaptation_listener) {
rtc::Event event;
resource_adaptation_queue_.PostTask([this, adaptation_listener, &event] {
RTC_DCHECK_RUN_ON(&resource_adaptation_queue_);
RTC_DCHECK(resource_adaptation_processor_);
resource_adaptation_processor_->AddAdaptationListener(adaptation_listener);
event.Set();
});
event.Wait(rtc::Event::kForever);
}
void VideoStreamEncoder::RemoveAdaptationListenerForTesting(
ResourceAdaptationProcessorListener* adaptation_listener) {
rtc::Event event;
resource_adaptation_queue_.PostTask([this, adaptation_listener, &event] {
RTC_DCHECK_RUN_ON(&resource_adaptation_queue_);
RTC_DCHECK(resource_adaptation_processor_);
resource_adaptation_processor_->RemoveAdaptationListener(
adaptation_listener);
event.Set();
});
event.Wait(rtc::Event::kForever);
}
} // namespace webrtc

28
video/video_stream_encoder.h
View File

@ -106,6 +106,9 @@ class VideoStreamEncoder : public VideoStreamEncoderInterface,
// Used for testing. For example the |ScalingObserverInterface| methods must
// be called on |encoder_queue_|.
rtc::TaskQueue* encoder_queue() { return &encoder_queue_; }
rtc::TaskQueue* resource_adaptation_queue() {
return &resource_adaptation_queue_;
}
void OnVideoSourceRestrictionsUpdated(
VideoSourceRestrictions restrictions,
@ -121,6 +124,11 @@ class VideoStreamEncoder : public VideoStreamEncoderInterface,
rtc::scoped_refptr<QualityScalerResource>
quality_scaler_resource_for_testing();
void AddAdaptationListenerForTesting(
ResourceAdaptationProcessorListener* adaptation_listener);
void RemoveAdaptationListenerForTesting(
ResourceAdaptationProcessorListener* adaptation_listener);
private:
class VideoFrameInfo {
public:
@ -405,11 +413,10 @@ class VideoStreamEncoder : public VideoStreamEncoderInterface,
VideoStreamInputStateProvider input_state_provider_;
// Responsible for adapting input resolution or frame rate to ensure resources
// (e.g. CPU or bandwidth) are not overused.
// This class is single-threaded.
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// guard the processor by it instead.
// This class is single-threaded on the resource adaptation queue.
std::unique_ptr<ResourceAdaptationProcessorInterface>
resource_adaptation_processor_ RTC_GUARDED_BY(&encoder_queue_);
resource_adaptation_processor_
RTC_GUARDED_BY(&resource_adaptation_queue_);
// Handles input, output and stats reporting related to VideoStreamEncoder
// specific resources, such as "encode usage percent" measurements and "QP
// scaling". Also involved with various mitigations such as inital frame
@ -417,18 +424,19 @@ class VideoStreamEncoder : public VideoStreamEncoderInterface,
// The manager primarily operates on the |encoder_queue_| but its lifetime is
// tied to the VideoStreamEncoder (which is destroyed off the encoder queue)
// and its resource list is accessible from any thread.
// TODO(https://crbug.com/webrtc/11542): When we have an adaptation queue,
// remove the RTC_GUARDED_BY to get resources on the adaptation queue.
VideoStreamEncoderResourceManager stream_resource_manager_
RTC_GUARDED_BY(&encoder_queue_);
VideoStreamEncoderResourceManager stream_resource_manager_;
// Carries out the VideoSourceRestrictions provided by the
// ResourceAdaptationProcessor, i.e. reconfigures the source of video frames
// to provide us with different resolution or frame rate.
// This class is thread-safe.
VideoSourceSinkController video_source_sink_controller_;
// All public methods are proxied to |encoder_queue_|. It must must be
// destroyed first to make sure no tasks are run that use other members.
// Public methods are proxied to the task queues. The queues must be destroyed
// first to make sure no tasks run that use other members.
// TODO(https://crbug.com/webrtc/11172): Move ownership of the
// ResourceAdaptationProcessor and its task queue to Call when processors are
// multi-stream aware.
rtc::TaskQueue resource_adaptation_queue_;
rtc::TaskQueue encoder_queue_;
RTC_DISALLOW_COPY_AND_ASSIGN(VideoStreamEncoder);

334
video/video_stream_encoder_unittest.cc
View File

File diff suppressed because it is too large Load Diff