diff --git a/highlight-pipeline-classes.org b/highlight-pipeline-classes.org new file mode 100644 index 0000000..210652c --- /dev/null +++ b/highlight-pipeline-classes.org @@ -0,0 +1,322 @@ +* Class Overview + +** Prepipeline Class + +Placeholder, implied from [[*Post-Pipeline Actions]]. + +User could action any setup needed here, eg mounting of cource file media. + +** InputFiles + +Approach: Described elsewhere, collect files and if relevant map to options (which feature extractor, duration etc). + +Options: per-class options (eg ~config_file_path~ to be used with an ~InputFiles~-class which takes user input from a config file) + +*** Class Sketch + +#+begin_src plantuml :results output :file inputfiles.svg +scale 1000 height + +InputFiles -- InputFilesArgs +InputFiles -- InputFilesJSON +InputFiles -- InputFilesYAML + +abstract class InputFiles { + InputFilesOptions options + + {abstract} get_files(*args, **kwargs) + +} + +note right of InputFiles::get_files + returns JSON +end note + +class InputFilesArgs {} + +class InputFilesJSON {} + +class InputFilesYAML {} +#+end_src + +#+RESULTS: +[[file:inputfiles.svg]] + +InputFiles + + InputFilesOptions + (eg config_file_path for eg InputFilesJSON or InputFilesYAML) + + get_files(**kwargs): JSON + +** FeatureExtractor Classes + +Basic approach is the usual: + + - setup / prepare / pre-run (eg create an audio file if the tool does not like AV media) + - work / run (find the features) + - teardown / cleanup / post-run + +The /work / run/ phase will need to either capture stdout or read files created by the tool to collect timestamps/intervals. [implementation detail for each FeatureExtractor] + +Options (with suggested defaults): + + - working directory (~/tmp/highlightgen/~) + - cleanup temporary files (~True~) + - log to stdout / file / none (~None~) + - padding for minimum feature duration (ie lengthen any short features to this duration, ~5s~) + - trimming for maximum feature duration (ie chop the end[s?] off anything longer than this, ~-1s~) + - reject/drop features shorter than (~0.1s~) + - reject/drop features longer than (~0.1s~) + - reject/drop features with a lower `score' than (~-1~) + +Notes on options: + + - All options should be optional due to defaults + - Not all options will apply to all FeatureExtractors (eg some may not produce a `score' or equivalent) + - Options that are not relevant to an encoder can be specified but will be ignored (consider emitting a ~WARN~ loglevel) + +*** Class Sketch + +#+begin_src plantuml :results output :file featureextractor.svg +scale 1000 height + +abstract class FeatureExtractor { + {field} FeatureExtractorOptions options + {field} Logger logger + + {abstract} setup() + + {abstract} run() + + {abstract} teardown() +} + +'@dataclass +struct FeatureExtractorOptions { + + working_directory : str + + do_cleanup : boolean + + log_level : int / enum + + minimum_feature_padding : float + + maximum_feature_trimming : float + + reject_shorter_than : float + + reject_longer_than : float + + reject_scoring_less_than : float +} + + +FeatureExtractor::options <|-- FeatureExtractorOptions +#+end_src + +#+RESULTS: +[[file:featureextractor.svg]] + +** Consolidator + +/(tl;dr: clustering? aggregation?)/ + +Basic approach: any time intervals produced earlier in the pipeline which overlap, or are within some specified /delta/ should be combined into one interval. + +Overlap example: + +~(10 - 15, 13 - 20) → (10-20)~ + +Delta = 5 example: + +~(10 - 15, 18 - 25) → (10-25)~ + +Non-overlap, non-within-delta=5 example: + +~(10 - 15, 21 - 30) → (10-15, 21-30)~ + +This is essentially reduction or transform on 1D data (time). It might make sense to consider two approaches (overlap, overlap after delta) separately. + +Since taking any action on consolidation (or whatever term) is potentially making an inaccurate or unwarranted value/content decision†, the option to skip this stage entirely (or effectively, in the form of a ~Consolidator~ class which replicates input to output unchanged) should be included. + +I am not sure if a ~Consolidator~ of any strategy should be permitted to output zero items / null. Similarly, I am not sure if trying to apply a ~Consolidator~ to any zero-sized / null set is well-defined. + +Would using some kind of set theory definition be useful, or just a distraction? + +*** Class sketch + +#+begin_src plantuml :results output :file consolidator.svg +scale 1000 height + +abstract class Consolidator { + ConsolidatorOptions options + {method} run() +} +#+end_src + +#+RESULTS: +[[file:consolidator.svg]] + +~Consolidator~: + - ConsolidatorOptions + + ConsolidatorSpecificOptions + - (eg delta) + + run() + +** Other Operators + +For example: + + - ~Join~ (combine/group/associate time intervals -- ie produce one highlight video) + +*Note*: this and the next step needs some thinking as to how the output would 'look' for being passed to VideoProducer. I had originally envisioned temporary files being written by intermediate stages, but I then hoped to avoid this and only `produce' a video at the last possible moment. This last part is notionally possible but may be introducing unwarranted complexity. + +** VideoProducer + +Approach: take definitions abolve and reify / actualise them- translate something along the lines of "take video /foo/bar.mp4 to produce and take segments A, B, C... and join them to produce a video file", expressed in representation/serialised class object/DSL-definition. + +On an implementation level, translate what we have to call out to a program or API, eg ffmpeg MLT libavuser (etc). + +Consideration: if video files (however temporary) can be produced earlier in the pipeline, there should perhaps be a ~VideoProducer~ that applies a 'nothing' definition -- that is, effectively it simply copies a (temporary) video to an output video (permanent). + +** Post-Pipeline Actions + +Placeholder, not sure of any yet (maybe show log or info? something user-friendly but technically optional?) + +** Additional Classes +*** Logging + +Setup on init- eg ~FileLogger(dest="/path/to/file.log")~ or on ~.setup()~ method ? + +Used by classes via eg D-I. + +Sketch: + +#+BEGIN_SRC plantuml :results output :file /tmp/testuml.png +'!theme spacelab +scale 1000 height + +FileLogger <|-- Logger +'note "throws LoggingError" as LE + + +abstract class Logger { + {abstract} void log() +} + +note right of Logger::log() + throws LoggingError +end note + +class FileLogger { + -_dest : String +} +#+END_SRC + +#+RESULTS: +[[file:/tmp/testuml.png]] + + +*** Interval + +Convenience class for highlights, around some data like: + +#+begin_src json +{ + "file": "/path/to/video", + "start": 10, + "end": 15, + "duration": 5, + "highlight_type": "laugh", + "score": 0.8, +} +#+end_src + +Advantages include: + + - can set start and duration or end + - makes it clearer what is being passed around + +Disadvantages: + + - class proliferation? + +** Additional Considerations + +Would it be desirable to add custom/user pre/post steps for each part of the pipeline? + +Pros: lots of flexibility +Cons: complexity for ?practical benefit (WLtH) + +** Overview / Recap + +#+begin_src plantuml :results output :file pipeline-overview.svg +scale 1000 height +title Video Highlight Generation Pipeline +allowmixing + +abstract class Logger { + {abstract} log() +} + +actor User + +User -> PrePipelineAction +PrePipelineAction -> InputFiles +InputFiles -> FeatureExtractor +FeatureExtractor -> Consolidator +Consolidator -> Operators +Consolidator -> VideoProducer +Operators -> VideoProducer +VideoProducer -> PostPipelineAction +VideoProducer -> User : Output video(s) +PostPipelineAction --> User : Output video(s) + +abstract class PrePipelineAction {} + +abstract class InputFiles {} + +abstract class FeatureExtractor {} +#+end_src + +#+RESULTS: +[[file:pipeline-overview.svg]] + +#+begin_src plantuml :results output :file highlight-pipeline2.svg +scale 1000 height +!theme cerulean + +actor User +action PrePipelineAction +process InputFiles +file Video as V1 +file Video as V2 +file Video as VN +process FeatureExtractor +collections Features +process Consolidator +collections "Consolidated Features" as ConsolidatedFeatures +process Operators +process VideoProducer +file Highlight as H1 +file Highlight as H2 +file Highlight as H3 +process PostPipelineAction + + +User -> PrePipelineAction +PrePipelineAction -> InputFiles +InputFiles <.. V1 +InputFiles <.. V2 +InputFiles <.. VN + +InputFiles -> FeatureExtractor +FeatureExtractor .. Features +FeatureExtractor -> Consolidator +Consolidator .. ConsolidatedFeatures +Consolidator -> Operators +Consolidator -> VideoProducer +Operators -> VideoProducer +Operators .. ConsolidatedFeatures +VideoProducer -> PostPipelineAction +VideoProducer ..> H1 +VideoProducer ..> H2 +VideoProducer ..> H3 +#+end_src + +#+RESULTS: +[[file:highlight-pipeline2.svg]] diff --git a/meetings.org b/meetings.org index a90cf25..c43068e 100644 --- a/meetings.org +++ b/meetings.org @@ -140,3 +140,7 @@ Next steps for coming week: see [[file:highlight-pipeline-planning.org][highlight-pipeline-planning.org]] +*** Pipeline Class Sketches + +see [[file:highlight-pipeline-classes.org]] +