itproject-docs/meetings.org

* 2024-06-18 1330: Meeting 0

Present:

 - Matthew Barr (MB)
 - Rob Hallam (RH)

** Project Choice

RH gave a quick (ish) pitch of several of the ideas listed:

 - Queen Bee Finder :: interesting, but data access issues
 - BorgBackup Visualiser :: lack of familiarity of Emacs Lisp for that portion
 - FAIR principles `data monkey' :: the `fallback' option, though still interesting to RH
 - Linux find fonts by descriptive tags :: slightly niche, and nebulous in the data gathering/scraping aspect
 - Video Highlight Finder :: interesting but potentially tricky; more *research*-oriented _(A)_
 - Cybersec Learning Game :: less well-defined in RH's head than it was, can the mechanics be interesting (MB mentioned the potential pitfall of `chocolate-covered broccoli', see [fn:chocobroc])
 - Patient Questionnaire App :: best defined and circumscribed, plenty of potential to do with; more *development*-oriented _(B)_
 - Google Fit exporter :: fairly standard, `speak to an API and do something interesting with the output' type project

Several may well be propitious.

*Outcome*: RH to investigate _(A)_ and _(B)_ further for prior art, feasibility etc and report back to MB with in 1-2 days.


[fn:chocobroc] This seems to have originated from Prof Amy Bruckman's 1999 paper, /`Can Educational Be Fun?'/ -- ``Most attempts at making software both educational and fun end up being neither. Fun is often treated like a sugar coating to be added to an educational core. Which makes about as much sense as chocolate-dipped broccoli.'' [[https://faculty.cc.gatech.edu/~asb/papers/bruckman-gdc99.pdf][Can Educational Be Fun? (PDF)]]

** Additional Considerations

RH mentioned a recent bereavement which happened during exams, with the knock-on effects of needing to sit exams during summer diet in August, and having to tend to affairs (clear house, etc) over the coming weeks and months. MB highlighted Good Cause options. RH mentioned he had been in contact with Isabella Widger, who had provided useful support advice previously and who he felt comfortable approaching again if needed.

*Outcome*: RH to balance requirements, will keep MB apprised if necessary and contact support if needed

** AOB

MB's communication preference is email for anything needing specifically `actioned'.

* 2024-06-26 1530: Meeting 1

** Agenda

 - [Questionnaire app] :: v basic React PoC done; other features seem implementable in React based on checking; next steps: user stories, plan UI
 - [Gaming highlight generator] :: laughter-detection does find laughs (when targeted: some FPs and FNs, when not targetd: many FPs, unknown FNs)
 - [Next steps] :: (app) pretty standard dev workflow, more focus on rapid prototypes & early feedback before formal user testing / evaluation later on; (highlights) focus on latter parts of `pipeline' first instead of feature detection- ie processing timestamps (consolidating & into clips), maybe UI for user to adjust clips (selection, times) before highlights made

** Pre-Meeting

*** Very Basic React App Proof of Concept

I decided to try out React since I haven't before just to see if I could get something that read in a questionnaire from JSON and could present it as a questionnaire.

http://inquisitor.roberthallam.org

Screenshot:

[[file:~/downloads/inquisitor-demo-example1.png]]

(alternative link)

*** Highlight Generator Pipeline / Workflow

[[file:~/downloads/highlightgeneration-process.svg]]

([[https://roberthallam.com/files/highlightgeneration-process.svg][alternative link]])

Not much to look at as it's only a few hours' work but lays the foundation for rapid prototyping / iterating once design is in place!

*** Laughter Detection

Tricky mix of specific package versions needed to get this working in 2024! Also needs a minor change (1 line) due to librosa API update. Can also be run on Colab:

[[file:~/downloads/colab-laughdetect.png]]

([[https://roberthallam.com/files/colab-laughdetect.png][alternative link]])

Observations:

 - running on a ~5 minute audio clip in AAC format takes ~30s (so a 3 hour video would take ~18 minutes)
 - qualitative observation: default parameters have a reasonable mix of detecting obvious laughs with a small number of FPs (and seems to have a few FNs too)

To test this I generated five audio clips- four were five minutes in duration, selected from longer clips, intended to be representative of obvious laughter, non-obvious / subtle laughter. The final audio clip was the full audio track of around 2½ hours.

Results

| Index | Duration | Context                                              | № Detected | FPs | Comments                                                                 |
|-------+----------+------------------------------------------------------+------------+-----+--------------------------------------------------------------------------|
|     1 |     5:06 | Multiple laughs from different speakers around ~1min |         15 |   3 | Seems inconsistent in detection when laughter is ongoing and overlapping |
|     2 |     5:08 | Mostly discussion, couple chuckles etc               |          0 | N/A | Arguably ~4/5 FNs                                                        |
|     3 |     5:10 | TBC                                                  |          8 |   5 | Detected segments are short                                              |
|     4 |     5:11 | One bit of obvious laughter                          |          2 |   1 | Detects the obvious bit of laughter                                      |
|     5 |  2:37:11 | Full-length video of gaming session                  |         74 |  65 | Quite a lot of FPs!                                                      |

/Note: Clips are not exactly five minutes due to the way ffmpeg cuts when doing a stream copy/

The results of that testing suggests two things about the default parameters:

 - laughter can be detected, even when it's coming from multiple speakers
 - those parameters produce a lot of FPs when not targeted

Given that, a two-pass approach might yield better results.


* 2024-07-11 1400: Meeting 2

** Agenda

/(note: GitLab has been down since Monday)/

For scoping, done:
 - conceptual overview of pipeline
 - class sketch
 - some user story cards (more TBD)

Still to do:
 - plan UI
 - collect and test other 'feature extractors' than laughter-detection

Also done:
 - simple proof-of-concept of highlight pipeline that can take input videos and produce output
 - ffmpeg invocation is a bit slow, might change approach

Next steps for coming week:
 - replace PoC version (turn class sketch into code)
 - start UI plan
 - begin integrating more feature extractors

** Pre-Meeting

*** Conceptual Overview Infographic

[[file:~/downloads/highlights-infographic-process.png]]

[[https://roberthallam.com/files/highlights-infographic-process.png]]

*** Pipeline Planning

[[file:highlight-pipleline2.svg]]

[[https://roberthallam.com/files/highlight-pipeline2.svg]]

see [[file:highlight-pipeline-planning.org][highlight-pipeline-planning.org]]

*** Pipeline Class Sketches

see [[file:highlight-pipeline-classes.org]]

*** Pipeline User Stories

see [[file:highlight-pipeline-userstories.org]]


* Footnotes