Joy R Kim

20220292830, Sep 15, 2022

Jun 2, 2022

17/805075

- SAN JOSE CA, US
Xue Bai - Bellevue WA, US
Aseem Agarwala - Seattle WA, US
Joel R. Brandt - Venice CA, US
Jovan Popovic - Seattle WA, US
Lubomira Dontcheva - Seattle WA, US
Joy Oakyung Kim - Menlo Park CA, US
Seth Walker - Oakland CA, US

G06V 20/40
G06K 9/62
G11B 27/19
G11B 27/00
G10L 25/78

Embodiments are directed to segmentation and hierarchical clustering of video. In an example implementation, a video is ingested to generate a multi-level hierarchical segmentation of the video. In some embodiments, the finest level identifies a smallest interaction unit of the video—semantically defined video segments of unequal duration called clip atoms. Clip atom boundaries are detected in various ways. For example, speech boundaries are detected from audio of the video, and scene boundaries are detected from video frames of the video. The detected boundaries are used to define the clip atoms, which are hierarchically clustered to form a multi-level hierarchical representation of the video. In some cases, the hierarchical segmentation identifies a static, pre-computed, hierarchical set of video segments, where each level of the hierarchical segmentation identifies a complete set (i.e., covering the entire range of the video) of disjoint (i.e., non-overlapping) video segments with a corresponding level of granularity.

20220292831, Sep 15, 2022

Jun 2, 2022

17/805080

- SAN JOSE CA, US
Xue Bai - Bellevue WA, US
Aseem Agarwala - Seattle WA, US
Joel R. Brandt - Venice CA, US
Jovan Popovic - Seattle WA, US
Lubomira Dontcheva - Seattle WA, US
Joy Oakyung Kim - Menlo Park CA, US
Seth Walker - Oakland CA, US

G06V 20/40
G06K 9/62
G11B 27/19
G11B 27/00
G10L 25/78

Embodiments are directed to segmentation and hierarchical clustering of video. In an example implementation, a video is ingested to generate a multi-level hierarchical segmentation of the video. In some embodiments, the finest level identifies a smallest interaction unit of the video—semantically defined video segments of unequal duration called clip atoms. Clip atom boundaries are detected in various ways. For example, speech boundaries are detected from audio of the video, and scene boundaries are detected from video frames of the video. The detected boundaries are used to define the clip atoms, which are hierarchically clustered to form a multi-level hierarchical representation of the video. In some cases, the hierarchical segmentation identifies a static, pre-computed, hierarchical set of video segments, where each level of the hierarchical segmentation identifies a complete set (i.e., covering the entire range of the video) of disjoint (i.e., non-overlapping) video segments with a corresponding level of granularity.

20210142827, May 13, 2021

Nov 8, 2019

16/679013

- San Jose CA, US
Lubomira DONTCHEVA - Seattle WA, US
Haiqing LU - Brooklyn NY, US
Zachary Platt MCCULLOUGH - Jersey City NJ, US
David R. STEIN - Brooklyn NY, US
Christopher NUUJA - San Anselmo CA, US
Benoit AMBRY - San Jose CA, US
Joel Richard Brandt - Venice CA, US
Cristin Ailidh FRASER - San Diego CA, US
Joy Oakyung KIM - Menlo Park CA, US
Hijung SHIN - Arlington MA, US

G11B 27/034
G11B 27/036
G06F 3/0484

Systems and methods provide for capturing and presenting content creation tools of an application used in a video. Application data from the application for the duration of the video is received. The application data includes data identifiers and time markers corresponding to user interaction with an application in a video. The application data is processed to detect tool identifiers identifying tools used in the video based on the data identifiers. For each a tool identifier, a tool label and a corresponding time in the timeline is determined. A tool record storing the tool labels and the corresponding times in association with the video is generated. When a viewer requests to watch the video, the tool record is presented to the viewer in conjunction with the video.