Bout review design

[skanderm]

This document specifies an outline of both frontend and backend design specifications for creating and reviewing audio segments from our hydrophones that have been flagged by listener-submitted audio detections. A "bout" is a general period of audio activity for all categories that can span anywhere from minutes to hours - whether it's whales, vessel noise, or environmental sounds. The goal of this audio review step are:

• to provide a first filter for whether there's a real-time ongoing bout (whales, vessel, etc)
• flag the bout for further finer-grained labeling
• optionally send a timely notification to subscribers for that bout type (e.g. send to listeners for an orca bout that's likely to continue for another 10-30+ minutes)

The design and requirements have so far been inspired by Scott's current workflow in Audacity to create bouts for later analysis. Please let me know if there's anything I can change that might make things easier in your context!

Bout review UI

Bout review index

The index page for the bout review will have a queue of candidates per feed that haven't been either rejected or added to a bout. We'll also add a section for current (ongoing) and past bouts, since we need a moderator to eventually label the end time for each bout.

Bout review page

The bout review page will allow the moderator to see a seek-able spectrogram of the feed and with filterable detections as markers on top of the spectrogram. Since an acoustic event can have many different types of bouts, we'll make it easy to create and switch between categories. Moderators can then seek to where the bout started and label the beginning, whether the bout is ongoing (no end time yet), as well as the end time. Moderators can also send a notification if they think the bout is likely to continue for a threshold amount of time. We can also display the Acartia map centered around the feed (with AIS/ship information) for contextualizing the bout.

Bout review backend

The backend will include:

• a FeedStream resource (feed_streams) for tracking HLS stream metadata to seek forward and backward in time
• a Bout resource (bouts) per feed and category to record long-duration bouts of ships, whales, etc.
• a join table between FeedStream and Bout for a many-to-many relationship
• A GraphQL API for reading, creating, and updating Bout and FeedStream data.

FeedStream

Attributes will include (pseudo-gql schema)

start_time: datetime
end_time: datetime
duration: decimal # seconds; nullable, might be a lot of work to backfill. We can have the client POST data about stream duration
feed: Feed
bucket_url: string # fully qualified uri for S3 bucket (https://s3-region.amazon[...]/stream-bucket-name/)
playlist_path: string # S3 Object path for playlist dir (e.g. /rpi_orcasound_lab/hls/1541027406/)
playlist_timestamp: string # UTC Unix epoch (e.g. 1541027406)
playlist_m3u8_path: string # S3 Object path for playlist file (e.g. /rpi_orcasound_lab/hls/1541027406/)
next_feed_stream: FeedStream
prev_feed_stream: FeedStream
inserted_at: datetime
updated_at: datetime

Bout

start_time: datetime
end_time: datetime
duration: decimal # seconds
ongoing: boolean # note: `true` is mutually exclusive with end_time being present
category: enum # biophony, anthrophony, geophony
feed: Feed
created_by_user: User
inserted_at: datetime
updated_at: datetime

# We should probably allow moderators to add some type of free-form info for the bout
tags: [Tag]
notes: [Note]

Spectrograms

Currently under-specified, but we can either do this in-browser or start generating pngs with librosa in an AWS Lambda job or switch our prod instance to Fly and use FLAME to spin up scalable librosa spectrogram jobs.

That's all I've got for now - please let me know if you have any suggestions or questions!

[scottveirs]

Sounds good overall! Two quick thoughts, @skanderm :

1. I think it's implied by "we need a moderator to eventually label the end time for each bout" -- but to be explicit, the UI will also need to enable the moderator to determine and define the start time as well.

2. For free-form info, one place to start could be the tag cloud we've developed through the OrcaHello moderation UI.

[scottveirs]

@ben-hendricks @valentina-s @veirs @dbainj1 @pastorep
We'd welcome your input on definition of bioacoustic bouts! Ideally, we'd end up with human and machine events similar to those currently defined by Ben's algorithms and in a format that would allow easy visualization through the open bioacoustic dashboard. For example, this view of humpback and orca events at the SIMRES East Point hydrophone:

@tsuize It may be worth starting to thinking about how the image classifications from the orca-eye-aye model could help define vessel bouts, too!

[ben-hendricks]

Here are some of the specs I use to define "bouts" (aka. bubbles) for bio-acoustic activity:

The clustering algorithm I use is DBSCAN. Below is the Python implementation:
https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html

DBSCAN uses two native clustering parameters
eps - the scanning window (here: time)
min-n - the minimum cluster size acceptable for a True Positive

I typically refine the conditions by also adding
min-dt - the minimum duration of the cluster (to add confidence)
min-rho - the minimum callrate within the cluster (to reject events that are too sporadic, and more likely False Positives)

Depending on the species, I set these parameters to approx.
eps = 300-600 sec
min-n = 10 - 20
min-dt = 60 - 300 sec
min-rho = 1-3 (per 60 seconds)

----------

It looks like you should be able to generate very similar info from visual/manual entries. One of the aspects that I do not trace is whether or not the bout is "ongoing". This could be as simple as comparing the time of latest detection with current time and compare the delta to the eps parameter. Since my code typically runs only once a day or once an hour, this is irrelevant. I am curious to see how you will implement that feature for your setup!

Cheers,
Ben

[dbainj1]

Scott, I noticed I wasn't getting false positives from Point Robinson, and Dataplicity said it was offline. I pulled the plug, plugged it back in, and Dataplicity says it's back, but when I try to listen on Orcasound, I get the exclamation point instead of sound. Can you try a remote reboot? This is starting to happen often enough that it's probably worth teaching me to do it so I don't have to bug you about it. Also, it may be worth figuring out why a simple power on doesn't automatically work anymore. Dave Sent from Samsung Galaxy smartphone. Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Scott Veirs ***@***.***> Sent: Wednesday, May 15, 2024 11:33:04 AM To: orcasound/orcasite ***@***.***> Cc: David Bain ***@***.***>; Mention ***@***.***> Subject: Re: [orcasound/orcasite] Bout review design (Discussion #454) @ben-hendricks<https://github.com/ben-hendricks> @valentina-s<https://github.com/valentina-s> @veirs<https://github.com/veirs> @dbainj1<https://github.com/dbainj1> @pastorep<https://github.com/pastorep> We'd welcome your input on definition of bioacoustic bouts! Ideally, we'd end up with human and machine events similar to those currently defined by Ben's algorithms and in a format that would allow easy visualization through the open bioacoustic dashboard. For example, this view of humpback and orca events at the SIMRES East Point hydrophone<https://whalesound.ca/map/?showFullReadings=true&stationId=East-Point>: Screenshot.2024-05-15.at.11.29.01.AM.png (view on web)<https://github.com/orcasound/orcasite/assets/14044595/59183696-1cbd-45d2-b936-d1091020af7a> @tsuize<https://github.com/tsuize> It may be worth starting to thinking about how the image classifications from the orca-eye-aye<https://github.com/orcasound/orca-eye-aye> model could help define vessel bouts, too! — Reply to this email directly, view it on GitHub<#454 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AWBBIJY64E54KWVIH5OPGTTZCOS6BAVCNFSM6AAAAABHGDTK46VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TINBZGQ3TS>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[paulcretu]

@skanderm this looks like a pretty good plan, some thoughts:

• How are you planning to get FeedStream end _time? We know when the streams start (playlist_timestamp), but figuring out when they stop will take more work (parsing the manifest file). Should it be nullable just like duration?
• We should try to move over to Cloudfront for serving the files to users, so we may want to keep track of the Cloudfront distribution URL as well.
• Can bouts overlap? (e.g. there are multiple sources of noise at the same time)
• How does this schema align with the catalog @valentina-s is working on? Back-filling FeedStreams seems like half of the cataloging effort. I imagine we'll eventually have another table to track individual stream chunks. It would be nice if the Orcasite schema and the catalog schema aligned, because then we could use something like parquet_fdw or pg_lakehouse to query over archived data

[skanderm]

• I'm thinking we can lean on the client to send info to the server, including setting the end_time for a stream. I was planning to open up a GraphQL endpoint to allow updating incomplete FeedStreams
• I can add a column for cloudfront_url. Or I wonder if we can pull the mapping from S3 if it's available?
• Yes, bouts can overlap. We can have any number of bouts (fish noises, orca vocalizations, vessel noise) happening concurrently.
• I'm not quite sure how it overlaps, though the current plan for backfilling the FeedStreams is to scan for all the epochs under each feed's object directory (e.g. <node_slug>/hls/<epoch_timestamp/) @valentina-s it'd be great to get your thoughts if possible!