[This CL is a rebase of an original CL by solenberg@:
https://codereview.webrtc.org/2948763002/ which in turn was a
rebase of an original CL by peah@:
https://chromium-review.googlesource.com/c/527032/]
Allow an external audio processing module to be used in WebRTC
This CL adds support for optionally using an externally created audio
processing module in a peerconnection. The ownership is shared
between the peerconnection and the external creator of the module.
As part of this the internal ownership of the audio processing module
is moved from VoiceEngine to WebRtcVoiceEngine.
BUG=webrtc:7775
Review-Url: https://codereview.webrtc.org/2961723004
Cr-Commit-Position: refs/heads/master@{#18837}
This CL corrects the initialization of the AEC3, as well
as for the other submodules in the whole audio processing module
in the sense that it properly update the submodule states also
for the case when reinitialization is trigger from the render
side of the audio processing module.
Bug: chromium:736889,webrtc:7879
Change-Id: I423e963835d0c3227caa8e186b29031bcb912515
Reviewed-on: https://chromium-review.googlesource.com/549315
Commit-Queue: Per Åhgren <peah@webrtc.org>
Reviewed-by: Ivo Creusen <ivoc@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#18784}
const int16_t* data() const;
int16_t* mutable_data();
- data() returns a zeroed static buffer on muted frames (to avoid unnecessary zeroing of the member buffer) and directly returns AudioFrame::data_ on unmuted frames.
- mutable_data(), lazily zeroes AudioFrame::data_ if the frame is currently muted, sets muted=false, and returns AudioFrame::data_.
These accessors serve to "force" callers to be aware of the mute state field, i.e. lazy zeroing is not the primary motivation.
This change only optimizes handling of muted frames where it is somewhat trivial to do so. Other improvements requiring more significant structural changes will come later.
BUG=webrtc:7343
TBR=henrika
Review-Url: https://codereview.webrtc.org/2750783004
Cr-Commit-Position: refs/heads/master@{#18543}
This CL ensures that it is not possible to run several echo canceller
solutions in cascade inside the audio processing module.
Bug: webrtc:7776
Change-Id: I1777f97aeacb8cdf5c6c3be4bf13eefcde7d69fb
Reviewed-on: https://chromium-review.googlesource.com/527053
Reviewed-by: Alex Loiko <aleloi@webrtc.org>
Commit-Queue: Per Åhgren <peah@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#18505}
configuration, inputs and outputs over a period of time. It is
activated by AudioProcessing::StartRecording. The data is stored in
binary protobuf format in a specified file. The file IO is, as of
this CL, done from the real-time audio thread.
This CL contains an interface for AecDump, a new APM submodule that
will handle the recordings. Calls to the new interface from the
AudioProcessingModule are added. These calls have no effect, and for a
short while, audio_processing_impl.cc will contain two copies of
recording calls.
The original calls are guarded by the WEBRTC_AUDIOPROC_DEBUG_DUMP
preprocessor define. They still have an effect, while the new ones do
not. In the following CLs, the old recording calls will be removed,
and an implementation of AecDump added.
The reasons for the refactoring is to move file IO operations from the
real-time audio thread, to add a top-level low-priority task queue for
logging tasks like this, to simplify and modularize audio_processing_impl.cc
and remove some of the preprocessor directives. These goals will be
archived by the upcoming CLs. The implementation is in
https://codereview.webrtc.org/2865113002.
BUG=webrtc:7404
Review-Url: https://codereview.webrtc.org/2778783002
Cr-Commit-Position: refs/heads/master@{#18233}
This CL ensures that the output of the level controller is kept
when no other submodules in APM are active
BUG=webrtc:7697,
Review-Url: https://codereview.webrtc.org/2902723002
Cr-Commit-Position: refs/heads/master@{#18230}
split domain and only implements floating point operations (to avoid spectral
leakage issues and unnecessary complexity).
The goal of this CL is adding the new sub-module into APM without providing an
implementation that could replace the existing gain control modules. The focus
is in fact on initialization, reset, and configuration of AGC2.
The module itself only applies a hard-coded gain value. This behavior will
change in the coming CLs.
BUG=webrtc:7494
Review-Url: https://codereview.webrtc.org/2848593002
Cr-Commit-Position: refs/heads/master@{#18222}
This CL adjusts the render processing rate such to avoid resampling of the
render signal when that is not needed.
Note that to avoid acquiring more locks than needed, this should be achieved
during initialization.
BUG=webrtc:7667
Review-Url: https://codereview.webrtc.org/2887693002
Cr-Commit-Position: refs/heads/master@{#18207}
This CL removes the residual echo detector from the list of
modules in APM that requires band-splitting.
BUG=webrtc:6220, webrtc:6183
Review-Url: https://codereview.webrtc.org/2884913002
Cr-Commit-Position: refs/heads/master@{#18164}
This CL moves the residual echo detector to reside outside of
the band-scheme in APM. The benefit of this is that the
residual echo detector will then no longer enforce the
band-splitting to be used when it is the only active component
inside APM.
This CL also introduces diagnostic dumping of data inside the
residual echo detector.
BUG=webrtc:6220, webrtc:6183
Review-Url: https://codereview.webrtc.org/2884593002
Cr-Commit-Position: refs/heads/master@{#18150}
This CL adds tuning to AEC3 for the purpose of reducing the impact of
gain changes in the analog microphone gain.
BUG=chromium:710818, webrtc:6018
Review-Url: https://codereview.webrtc.org/2811283003
Cr-Commit-Position: refs/heads/master@{#17673}
Changes in the microphone gain are effecting the AEC in the sense
that each change in the microphone gain is a change in the echo
path seen by the AEC. This CL utilizes the ability of AEC3 to
leverage information about known changes in the analog microphone
gain.
BUG=webrtc:6018
Review-Url: https://codereview.webrtc.org/2808073002
Cr-Commit-Position: refs/heads/master@{#17625}
This CL ensures that the number of bands
for the render side matches that for the capture side
when AEC3 is active. Without this, there was problems
when the render rate is different from the capture rate.
BUG=webrtc:6018
Review-Url: https://codereview.webrtc.org/2800033003
Cr-Commit-Position: refs/heads/master@{#17586}
Reason for revert:
Trying to re-land after solving some related issues.
There are no changes compared to the original CL.
Original issue's description:
> Revert of Loosening the coupling between WebRTC and //third_party/protobuf (patchset #16 id:300001 of https://codereview.webrtc.org/2747863003/ )
>
> Reason for revert:
> I will try to reland next week because it is causing some problems.
>
> Original issue's description:
> > To accommodate some downstream WebRTC users we need to loosen
> > the coupling between our code and the //third_party/protobuf.
> >
> > This includes using typedefs to define strings instead of
> > assuming std::string.
> >
> > After this refactoring it will be possible to link with other
> > protobuf implementations than the current one.
> >
> > We moved the PRESUBMIT check to another CL [1]. The goal of this
> > presubmit is to avoid the direct usage of google::protobuf outside
> > of the webrtc/base/protobuf_utils.h header file.
> >
> > [1] - https://codereview.webrtc.org/2753823003/
> >
> > BUG=webrtc:7340
> > NOTRY=True
> >
> > Review-Url: https://codereview.webrtc.org/2747863003
> > Cr-Commit-Position: refs/heads/master@{#17466}
> > Committed: 16ab93b952
>
> TBR=kjellander@webrtc.org,henrik.lundin@webrtc.org,kwiberg@webrtc.org,michaelt@webrtc.org,peah@webrtc.org
> # Not skipping CQ checks because original CL landed more than 1 days ago.
> BUG=webrtc:7340
>
> Review-Url: https://codereview.webrtc.org/2786363002
> Cr-Commit-Position: refs/heads/master@{#17483}
> Committed: d00aad5eb2TBR=kjellander@webrtc.org,henrik.lundin@webrtc.org,kwiberg@webrtc.org,michaelt@webrtc.org,peah@webrtc.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG=webrtc:7340
NOTRY=True
Review-Url: https://codereview.webrtc.org/2791963003
Cr-Commit-Position: refs/heads/master@{#17584}
This CL adds major render pipeline changes to the AEC3 code. The reason
for these are that
1) It allows the echo removal unit to receive information about the content
in bands beyond band 0, thereby allowing removal of high-frequency
echoes
2) It allows more controlled handling of the render buffers, allowing proper
buffer behaviour during capture glitches and clock-drift.
Unfortunately, the render pipeline caused a lot of related changes in much
of the rest of the AEC3 files. Most of these are, however, caused by
a change of class name.
Another unfortunate effect of this CL, is that a number of unittest cease to
compile. I chose to temporarily solve that by removing them from the
build using #if/#endif. The reason for that is that those will anyway again
need to be changed in the next review, and doing like this avoids them
having to be reviewed twice.
BUG=webrtc:6018
Review-Url: https://codereview.webrtc.org/2784023002
Cr-Commit-Position: refs/heads/master@{#17547}
Reason for revert:
I will try to reland next week because it is causing some problems.
Original issue's description:
> To accommodate some downstream WebRTC users we need to loosen
> the coupling between our code and the //third_party/protobuf.
>
> This includes using typedefs to define strings instead of
> assuming std::string.
>
> After this refactoring it will be possible to link with other
> protobuf implementations than the current one.
>
> We moved the PRESUBMIT check to another CL [1]. The goal of this
> presubmit is to avoid the direct usage of google::protobuf outside
> of the webrtc/base/protobuf_utils.h header file.
>
> [1] - https://codereview.webrtc.org/2753823003/
>
> BUG=webrtc:7340
> NOTRY=True
>
> Review-Url: https://codereview.webrtc.org/2747863003
> Cr-Commit-Position: refs/heads/master@{#17466}
> Committed: 16ab93b952TBR=kjellander@webrtc.org,henrik.lundin@webrtc.org,kwiberg@webrtc.org,michaelt@webrtc.org,peah@webrtc.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG=webrtc:7340
Review-Url: https://codereview.webrtc.org/2786363002
Cr-Commit-Position: refs/heads/master@{#17483}
the coupling between our code and the //third_party/protobuf.
This includes using typedefs to define strings instead of
assuming std::string.
After this refactoring it will be possible to link with other
protobuf implementations than the current one.
We moved the PRESUBMIT check to another CL [1]. The goal of this
presubmit is to avoid the direct usage of google::protobuf outside
of the webrtc/base/protobuf_utils.h header file.
[1] - https://codereview.webrtc.org/2753823003/
BUG=webrtc:7340
NOTRY=True
Review-Url: https://codereview.webrtc.org/2747863003
Cr-Commit-Position: refs/heads/master@{#17466}
Currently no lock is taken when returning echo likelihood stats, which causes a race condition between the thread getting the stats and the thread running the echo detector. This CL resolves the issue by adding locking.
BUG=webrtc:7346
Review-Url: https://codereview.webrtc.org/2749973003
Cr-Commit-Position: refs/heads/master@{#17270}
AEC3 and AEC2 are separate submodules in APM. This CL ensures that AEC3
deactivates AEC2 if both are active at the same time.
BUG=webrtc:6018
Review-Url: https://codereview.webrtc.org/2675863004
Cr-Commit-Position: refs/heads/master@{#16443}
that the level of the output in the audio processing
module is monitored. This CL adds that.
BUG=webrtc:6181, webrtc:6183, webrtc:6220
Review-Url: https://codereview.webrtc.org/2549143004
Cr-Commit-Position: refs/heads/master@{#15718}
It will be followed by a number of other CLs that extends this framework.
BUG=webrtc:6018
Review-Url: https://codereview.webrtc.org/2567513003
Cr-Commit-Position: refs/heads/master@{#15593}
Reason for revert:
The downstream problem is now fixed, and this should be good to land again.
Original issue's description:
> Revert of APM: Change 3 UMA metrics to fewer but linearly distributed buckets (patchset #2 id:20001 of https://codereview.webrtc.org/2547593002/ )
>
> Reason for revert:
> Breaks down-stream dependencies.
>
> Original issue's description:
> > APM: Change 3 UMA metrics to fewer but linearly distributed buckets
> >
> > In this change WebRTC.Audio.ApmCaptureInputLevel{Average,Peak} are
> > changed to 64 buckets, while WebRTC.Audio.AgcLevel is changed to 50
> > buckets. All three are changed to have linear spacing between buckets.
> >
> > Also, the metrics are renamed to avoid stats conflicts because of different bucket schemes:
> > - WebRTC.Audio.AgcLevel -> WebRTC.Audio.AgcSetLevel
> > - WebRTC.Audio.ApmCaptureInputLevelAverage -> WebRTC.Audio.ApmCaptureInputLevelAverageRms
> > - WebRTC.Audio.ApmCaptureInputLevelPeakRms -> WebRTC.Audio.ApmCaptureInputLevelPeakRms
> >
> > BUG=webrtc:6622
> > CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_rel_ng
> >
> > Committed: https://crrev.com/49715fe3be17d8579586d5bc954d626126d53415
> > Cr-Commit-Position: refs/heads/master@{#15418}
>
> TBR=peah@webrtc.org,rkaplow@chromium.org
> # Skipping CQ checks because original CL landed less than 1 days ago.
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
> BUG=webrtc:6622
>
> Committed: https://crrev.com/63407a9b6ae6f3fc096e01d64e46c6d21d86b517
> Cr-Commit-Position: refs/heads/master@{#15420}
TBR=peah@webrtc.org,rkaplow@chromium.org
BUG=webrtc:6622
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_rel_ng
Review-Url: https://codereview.webrtc.org/2551863003
Cr-Commit-Position: refs/heads/master@{#15442}
This change enables experimentation with the clipping minimum level
parameter in the gain control.
BUG=webrtc:6622
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_rel_ng
Review-Url: https://codereview.webrtc.org/2543753006
Cr-Commit-Position: refs/heads/master@{#15426}
Reason for revert:
Breaks down-stream dependencies.
Original issue's description:
> APM: Change 3 UMA metrics to fewer but linearly distributed buckets
>
> In this change WebRTC.Audio.ApmCaptureInputLevel{Average,Peak} are
> changed to 64 buckets, while WebRTC.Audio.AgcLevel is changed to 50
> buckets. All three are changed to have linear spacing between buckets.
>
> Also, the metrics are renamed to avoid stats conflicts because of different bucket schemes:
> - WebRTC.Audio.AgcLevel -> WebRTC.Audio.AgcSetLevel
> - WebRTC.Audio.ApmCaptureInputLevelAverage -> WebRTC.Audio.ApmCaptureInputLevelAverageRms
> - WebRTC.Audio.ApmCaptureInputLevelPeakRms -> WebRTC.Audio.ApmCaptureInputLevelPeakRms
>
> BUG=webrtc:6622
> CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_rel_ng
>
> Committed: https://crrev.com/49715fe3be17d8579586d5bc954d626126d53415
> Cr-Commit-Position: refs/heads/master@{#15418}
TBR=peah@webrtc.org,rkaplow@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=webrtc:6622
Review-Url: https://codereview.webrtc.org/2548333002
Cr-Commit-Position: refs/heads/master@{#15420}
In this change WebRTC.Audio.ApmCaptureInputLevel{Average,Peak} are
changed to 64 buckets, while WebRTC.Audio.AgcLevel is changed to 50
buckets. All three are changed to have linear spacing between buckets.
Also, the metrics are renamed to avoid stats conflicts because of different bucket schemes:
- WebRTC.Audio.AgcLevel -> WebRTC.Audio.AgcSetLevel
- WebRTC.Audio.ApmCaptureInputLevelAverage -> WebRTC.Audio.ApmCaptureInputLevelAverageRms
- WebRTC.Audio.ApmCaptureInputLevelPeakRms -> WebRTC.Audio.ApmCaptureInputLevelPeakRms
BUG=webrtc:6622
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_rel_ng
Review-Url: https://codereview.webrtc.org/2547593002
Cr-Commit-Position: refs/heads/master@{#15418}
This CL adds WebRTC.Audio.ApmCaptureInputLevelAverage and
WebRTC.Audio.ApmCaptureInputLevelPeak. The metrics are updated once
every 10 seconds.
BUG=webrtc:6622
Review-Url: https://codereview.webrtc.org/2534473004
Cr-Commit-Position: refs/heads/master@{#15300}
There's no longer any need to make the two arguments have the same
signedness, so we can drop the "u" suffix on literal integer
arguments.
NOPRESUBMIT=true
BUG=webrtc:6645
Review-Url: https://codereview.webrtc.org/2535593002
Cr-Commit-Position: refs/heads/master@{#15280}
the APM parameters to the high-pass filter.
The introduction will be done in three steps:
1) This CL which introduces the new scheme and
changes the code in webrtcvoiceengine.cc to use it.
2) Introduce the scheme into upstream code.
3) Remove the HighPassFilter interface in APM.
BUG=webrtc::6220, webrtc::6296, webrtc::6297, webrtc::6181, webrtc::5298
Review-Url: https://codereview.webrtc.org/2415403002
Cr-Commit-Position: refs/heads/master@{#15197}
This adds two command-line flags to audioproc_f: -red and -red_graph, which can be used to enable/disable the RED, and to set the output path for the graph. The graph is generated as a python script that depends on matplotlib and numpy to display the graph.
BUG=webrtc:6525
Review-Url: https://codereview.webrtc.org/2486763002
Cr-Commit-Position: refs/heads/master@{#15069}
In the new APM statistics interface, the default values did not match those previously used in AudioSendStream::Stats.
BUG=webrtc:6525
Review-Url: https://codereview.webrtc.org/2469783002
Cr-Commit-Position: refs/heads/master@{#14896}
This CL does not contain the actual algorithm, but only creates an empty processing component and connects the right signals to it. The algorithm will be added in a follow-up CL.
BUG=webrtc:6525
Review-Url: https://codereview.webrtc.org/2405403003
Cr-Commit-Position: refs/heads/master@{#14820}
Reason for revert:
This CL breaks internal dependencies.
Original issue's description:
> New statistics interface for APM
>
> This adds functions to enable and retrieve statistics from APM. These functions are not yet called, which will happen in a follow-up CL.
>
> BUG=webrtc:6525
>
> Committed: https://crrev.com/8b8d3e4c30e8ea3846b58dfd36d1fd35a7799df4
> Cr-Commit-Position: refs/heads/master@{#14810}
TBR=peah@webrtc.org,solenberg@webrtc.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=webrtc:6525
Review-Url: https://codereview.webrtc.org/2456333002
Cr-Commit-Position: refs/heads/master@{#14814}
aecdump recordings in an efficient manner, it is
important to be able to use a standardized analysis
script. For this to be feasible, data log points should
be present.
This CL adds those logpoints as well as the framework
needed to for those to work.
BUG=webrtc:6564
Review-Url: https://codereview.webrtc.org/2457783003
Cr-Commit-Position: refs/heads/master@{#14812}
This adds functions to enable and retrieve statistics from APM. These functions are not yet called, which will happen in a follow-up CL.
BUG=webrtc:6525
Review-Url: https://codereview.webrtc.org/2433153003
Cr-Commit-Position: refs/heads/master@{#14810}
Several subcomponents inside APM copy render audio from
the render side to the capture side using the same
pattern. Currently this is done independently for the
submodules.
This CL moves the the AGC functionality for this into
APM.
BUG=webrtc:5298, webrtc:6540
Review-Url: https://codereview.webrtc.org/2444283002
Cr-Commit-Position: refs/heads/master@{#14770}
the render side to the capture side using the same
pattern. Currently this is done independently for the
submodules.
This CL moves the the AECM functionality for this into
APM.
BUG=webrtc:5298, webrtc:6540
Review-Url: https://codereview.webrtc.org/2444793005
Cr-Commit-Position: refs/heads/master@{#14768}
the render side to the capture side using the same
pattern. Currently this is done independently for the
submodules.
This CL moves the the AEC functionality for this into
APM.
BUG=webrtc:5298, webrtc:6540
Review-Url: https://codereview.webrtc.org/2427553003
Cr-Commit-Position: refs/heads/master@{#14726}
to the functionality in the audio processing module.
Therefore, it should be a pure interface.
This CL ensures that is the case.
BUG=webrtc:6515
Review-Url: https://codereview.webrtc.org/2406193002
Cr-Commit-Position: refs/heads/master@{#14608}
receive a signal level to use initially, instead of the
default initial signal level.
The initial form of the CL
(https://codereview.webrtc.org/2254973003/) was reverted
due to down-stream dependencies. These have been resolved,
but the CL needed to be revised according to the new scheme
for passing parameters to the audio processing module.
Therefore, please review this CL as if it is new.
TBR=aleloi@webrtc.org
BUG=webrtc:6386
Review-Url: https://codereview.webrtc.org/2337083002
Cr-Commit-Position: refs/heads/master@{#14579}
Before this CL, the IntelligibilityEnhancer introduced a processing delay to the lower band, without compensating for it in the higher bands. This CL corrects this.
BUG=b/30780909
R=henrik.lundin@webrtc.org, peah@webrtc.org
Review URL: https://codereview.webrtc.org/2320833002 .
Cr-Commit-Position: refs/heads/master@{#14311}
methods inside the audio processing module for the purpose of
increasing code readability.
BUG=
Review-Url: https://codereview.webrtc.org/2335633002
Cr-Commit-Position: refs/heads/master@{#14269}
This requires marking a bunch of compile-time constants "constexpr"
instead of just "const".
Review-Url: https://codereview.webrtc.org/2335483003
Cr-Commit-Position: refs/heads/master@{#14199}
behavior of the audio processing module is quite complex and hard to
implement in a threadsafe and efficient manner. Therefore a new
scheme for setting the parameters in the audio processing module is
introduced in this CL.
The idea is to roll this scheme out gradually and as a first functionality
in the audio processing module where this is applied the level controller
was chosen. This CL includes the replacement of the Config-based
level controller scheme with the new scheme.
TBR=henrik.lundin@webrtc.org, solenberg@webrtc.org,
BUG=webrtc:5298
Review-Url: https://codereview.webrtc.org/2338493002
Cr-Commit-Position: refs/heads/master@{#14190}
