Parcourir la source

feat: add LaughterFeatureExtractor

This adds functionality for getting laughter by using jrgillick's
laughter detection library

NB python expects all of the feature extractor's dependencies to be
available; perhaps in future we can do something even fancier like
activating another python env

[retroactive commit]
main
Rob Hallam il y a 3 mois
Parent
révision
f892d8da41
1 fichiers modifiés avec 86 ajouts et 0 suppressions
  1. +86
    -0
      pipeline/feature_extractors.py

+ 86
- 0
pipeline/feature_extractors.py Voir le fichier

@@ -13,6 +13,7 @@ class FeatureExtractor(ABC):

def teardown(self):
pass

class LaughterFeatureExtractor(FeatureExtractor):
"""Feature extractor for laughter detection.

@@ -28,3 +29,88 @@ class LaughterFeatureExtractor(FeatureExtractor):

See: https://github.com/jrgillick/laughter-detection for the laughter-detection library
"""

def __init__(self, input_files=None, config=None):
"""It is expected that input_files is a SourceMedia object"""
self.input_files = input_files
self.config = config
self.features = []

def _laughdetect(self, audio_file):
"""Run laughter detection on the audio file"""
laugh_detector_dir = "/home/robert/mounts/980data/code/laughter-detection/"
laugh_detector_script = "segment_laughter.py"
# fake output for testing
# laugh_detector_path = "tests/fake_segment_laughter.py"
laugh_detector_cmd = ["python", f"{laugh_detector_dir}{laugh_detector_script}",
f"--input_audio_file={audio_file}"]

# run command, capture output, ignore exit status
laugh_output = subprocess.run(laugh_detector_cmd,
stdout=subprocess.PIPE,
cwd=laugh_detector_dir).stdout.decode("utf-8")
# ↑ have to include cwd to keep laughter-detection imports happy
# also, it isn't happy if no output dir is specified but we get laughs so it's grand

# laughs are lines in stdout that start with "instance:", followed by a space and a 2-tuple of floats
# so jump to the 10th character and evaluate the rest of the line
return [literal_eval(instance[10:])
for instance in laugh_output.splitlines()
if instance.startswith("instance: ")]

def _adjust_features(self):
"""Adjust features according to config

Generically, this ensures features conform to config - min/max feature length, etc.

In the context of LaughterFeatureExtractor, there is some secret sauce: things that
cause a laugh generally /precede/ the laugh, so we want more team before the detected start
than at the end. For example, for a minimum feature length of 15s, we might prepend 10 seconds,
and append 5 seconds (for example), or 12s and 3s. We may wish to do this pre/post adjustment
for all laughter features found, regardless of length.

TODO: figure out how we're going to handle length adjustments
TODO: config for length adjustments per design doc
TODO: play with numbers more to see what works best
"""
PREPEND = 7.0
APPEND = 3.0

for feature in self.features:
# do the pre & post adjustment
feature.interval.move_start(-PREPEND, relative=True)
feature.interval.move_end(APPEND, relative=True)

def setup(self):
"""Setup the laughter feature extractor -- validate input files & config

jrgillick's laughter-detection library can work with AV files directly

TODO: validate input files
TODO: handle config
"""
logger.debug("LaughterFeatureExtractor setup")

# Validate input files
if not self.input_files:
raise ValueError("No input files provided")

# TODO: convert video to audio if needed

def run(self):
"""Extract laughter features for each input file"""
if self.input_files:
for file in self.input_files:
laughs = self._laughdetect(file.path)
for laugh in laughs:
start, end = laugh
self.features.append(Feature(interval=Interval(start=start, end=end),
source="laughter", path=file.path))
# TODO: implement options eg minimum feature length

# adjust features
self._adjust_features()

def teardown(self):
pass


Chargement…
Annuler
Enregistrer