[meeting 2] add pipeline classes doc

highlight-pipeline-classes.org

@@ -0,0 +1,322 @@
+* Class Overview
+
+** Prepipeline Class
+
+Placeholder, implied from [[*Post-Pipeline Actions]].
+
+User could action any setup needed here, eg mounting of cource file media.
+
+** InputFiles
+
+Approach: Described elsewhere, collect files and if relevant map to options (which feature extractor, duration etc).
+
+Options: per-class options (eg ~config_file_path~ to be used with an ~InputFiles~-class which takes user input from a config file)
+
+*** Class Sketch
+
+#+begin_src plantuml :results output :file inputfiles.svg
+scale 1000 height
+
+InputFiles -- InputFilesArgs
+InputFiles -- InputFilesJSON
+InputFiles -- InputFilesYAML
+
+abstract class InputFiles {
+  InputFilesOptions options
+
+  {abstract} get_files(*args, **kwargs)
+
+}
+
+note right of InputFiles::get_files
+  returns JSON
+end note
+
+class InputFilesArgs {}
+
+class InputFilesJSON {}
+
+class InputFilesYAML {}
+#+end_src
+
+#+RESULTS:
+[[file:inputfiles.svg]]
+
+InputFiles
+  + InputFilesOptions
+    (eg config_file_path for eg InputFilesJSON or InputFilesYAML)
+
+  get_files(**kwargs): JSON
+
+** FeatureExtractor Classes
+
+Basic approach is the usual:
+
+ - setup / prepare / pre-run (eg create an audio file if the tool does not like AV media)
+ - work / run (find the features)
+ - teardown / cleanup / post-run
+
+The /work / run/ phase will need to either capture stdout or read files created by the tool to collect timestamps/intervals. [implementation detail for each FeatureExtractor]
+
+Options (with suggested defaults):
+
+ - working directory (~/tmp/highlightgen/~)
+ - cleanup temporary files (~True~)
+ - log to stdout / file / none (~None~)
+ - padding for minimum feature duration (ie lengthen any short features to this duration, ~5s~)
+ - trimming for maximum feature duration (ie chop the end[s?] off anything longer than this, ~-1s~)
+ - reject/drop features shorter than (~0.1s~)
+ - reject/drop features longer than (~0.1s~)
+ - reject/drop features with a lower `score' than (~-1~)
+
+Notes on options:
+
+ - All options should be optional due to defaults
+ - Not all options will apply to all FeatureExtractors (eg some may not produce a `score' or equivalent)
+ - Options that are not relevant to an encoder can be specified but will be ignored (consider emitting a ~WARN~ loglevel)
+
+*** Class Sketch
+
+#+begin_src plantuml :results output :file featureextractor.svg
+scale 1000 height
+
+abstract class FeatureExtractor {
+  {field} FeatureExtractorOptions options
+  {field} Logger logger
+
+  {abstract} setup()
+
+  {abstract} run()
+
+  {abstract} teardown()
+}
+
+'@dataclass
+struct FeatureExtractorOptions {
+   + working_directory : str
+   + do_cleanup : boolean
+   + log_level : int / enum
+   + minimum_feature_padding : float
+   + maximum_feature_trimming : float
+   + reject_shorter_than : float
+   + reject_longer_than : float
+   + reject_scoring_less_than : float
+}
+
+
+FeatureExtractor::options <|-- FeatureExtractorOptions
+#+end_src
+
+#+RESULTS:
+[[file:featureextractor.svg]]
+
+** Consolidator
+
+/(tl;dr: clustering? aggregation?)/
+
+Basic approach: any time intervals produced earlier in the pipeline which overlap, or are within some specified /delta/ should be combined into one interval.
+
+Overlap example:
+
+~(10 - 15,  13 - 20) → (10-20)~
+
+Delta = 5 example:
+
+~(10 - 15, 18 - 25) → (10-25)~
+
+Non-overlap, non-within-delta=5 example:
+
+~(10 - 15, 21 - 30) → (10-15, 21-30)~
+
+This is essentially reduction or transform on 1D data (time). It might make sense to consider two approaches (overlap, overlap after delta) separately.
+
+Since taking any action on consolidation (or whatever term) is potentially making an inaccurate or unwarranted value/content decision†, the option to skip this stage entirely (or effectively, in the form of a ~Consolidator~ class which replicates input to output unchanged) should be included.
+
+I am not sure if a ~Consolidator~ of any strategy should be permitted to output zero items / null. Similarly, I am not sure if trying to apply a ~Consolidator~ to any zero-sized / null set is well-defined.
+
+Would using some kind of set theory definition be useful, or just a distraction?
+
+*** Class sketch
+
+#+begin_src plantuml :results output :file consolidator.svg
+scale 1000 height
+
+abstract class Consolidator {
+  ConsolidatorOptions options
+  {method} run()
+}
+#+end_src
+
+#+RESULTS:
+[[file:consolidator.svg]]
+
+~Consolidator~:
+ - ConsolidatorOptions
+   + ConsolidatorSpecificOptions
+     - (eg delta)
+
+ run()
+
+** Other Operators
+
+For example:
+
+ - ~Join~ (combine/group/associate time intervals -- ie produce one highlight video)
+
+*Note*: this and the next step needs some thinking as to how the output would 'look' for being passed to VideoProducer. I had originally envisioned temporary files being written by intermediate stages, but I then hoped to avoid this and only `produce' a video at the last possible moment. This last part is notionally possible but may be introducing unwarranted complexity.
+
+** VideoProducer
+
+Approach: take definitions abolve and reify / actualise them- translate something along the lines of "take video /foo/bar.mp4 to produce and take segments A, B, C... and join them to produce a video file", expressed in representation/serialised class object/DSL-definition.
+
+On an implementation level, translate what we have to call out to a program or API, eg ffmpeg MLT libavuser (etc).
+
+Consideration: if video files (however temporary) can be produced earlier in the pipeline, there should perhaps be a ~VideoProducer~ that applies a 'nothing' definition -- that is, effectively it simply copies a (temporary) video to an output video (permanent).
+
+** Post-Pipeline Actions
+
+Placeholder, not sure of any yet (maybe show log or info? something user-friendly but technically optional?)
+
+** Additional Classes
+*** Logging
+
+Setup on init- eg ~FileLogger(dest="/path/to/file.log")~ or on ~.setup()~ method ?
+
+Used by classes via eg D-I.
+
+Sketch:
+
+#+BEGIN_SRC plantuml :results output :file /tmp/testuml.png
+'!theme spacelab
+scale 1000 height
+
+FileLogger <|-- Logger
+'note "throws LoggingError" as LE
+
+
+abstract class Logger {
+  {abstract} void log()
+}
+
+note right of Logger::log()
+  throws LoggingError
+end note
+
+class FileLogger {
+  -_dest : String
+}
+#+END_SRC
+
+#+RESULTS:
+[[file:/tmp/testuml.png]]
+
+
+*** Interval
+
+Convenience class for highlights, around some data like:
+
+#+begin_src json
+{
+  "file": "/path/to/video",
+  "start": 10,
+  "end": 15,
+  "duration": 5,
+  "highlight_type": "laugh",
+  "score": 0.8,
+}
+#+end_src
+
+Advantages include:
+
+ - can set start and duration or end
+ - makes it clearer what is being passed around
+
+Disadvantages:
+
+ - class proliferation?
+
+** Additional Considerations
+
+Would it be desirable to add custom/user pre/post steps for each part of the pipeline?
+
+Pros: lots of flexibility
+Cons: complexity for ?practical benefit (WLtH)
+
+** Overview / Recap
+
+#+begin_src plantuml :results output :file pipeline-overview.svg
+scale 1000 height
+title Video Highlight Generation Pipeline
+allowmixing
+
+abstract class Logger {
+  {abstract} log()
+}
+
+actor User
+
+User -> PrePipelineAction
+PrePipelineAction -> InputFiles
+InputFiles -> FeatureExtractor
+FeatureExtractor -> Consolidator
+Consolidator -> Operators
+Consolidator -> VideoProducer
+Operators -> VideoProducer
+VideoProducer -> PostPipelineAction
+VideoProducer -> User : <i>Output video(s)</i>
+PostPipelineAction --> User : <i>Output video(s)</i>
+
+abstract class PrePipelineAction {}
+
+abstract class InputFiles {}
+
+abstract class FeatureExtractor {}
+#+end_src
+
+#+RESULTS:
+[[file:pipeline-overview.svg]]
+
+#+begin_src plantuml :results output :file highlight-pipeline2.svg
+scale 1000 height
+!theme cerulean
+
+actor User
+action PrePipelineAction
+process InputFiles
+file Video as V1
+file Video as V2
+file Video as VN
+process FeatureExtractor
+collections Features
+process Consolidator
+collections "Consolidated Features" as ConsolidatedFeatures
+process Operators
+process VideoProducer
+file Highlight as H1
+file Highlight as H2
+file Highlight as H3
+process PostPipelineAction
+
+
+User -> PrePipelineAction
+PrePipelineAction -> InputFiles
+InputFiles <.. V1
+InputFiles <.. V2
+InputFiles <.. VN
+
+InputFiles -> FeatureExtractor
+FeatureExtractor .. Features
+FeatureExtractor -> Consolidator
+Consolidator .. ConsolidatedFeatures
+Consolidator -> Operators
+Consolidator -> VideoProducer
+Operators -> VideoProducer
+Operators .. ConsolidatedFeatures
+VideoProducer -> PostPipelineAction
+VideoProducer ..> H1
+VideoProducer ..> H2
+VideoProducer ..> H3
+#+end_src
+
+#+RESULTS:
+[[file:highlight-pipeline2.svg]]

meetings.org

@@ -140,3 +140,7 @@ Next steps for coming week:
 
 see [[file:highlight-pipeline-planning.org][highlight-pipeline-planning.org]]
 
+*** Pipeline Class Sketches
+
+see [[file:highlight-pipeline-classes.org]]
+