layout: true <div class="my-footer"> <div class="my-footer-box"><a href="https://openvolley.org/"><img style="display:inline;" src="extra/ovoutline-w.png"/>openvolley.org</a></div> <div class="my-footer-box"><a href="https://https://volleyball.ca/"><img src="extra/vc-w-wide.png"/></a></div> <div class="my-footer-box"><a href="https://untan.gl/"><img src="extra/su_title-w.png"/></a></div> </div> --- class: inverse, logo, center <img src="extra/3logo2.png" style="width:65%; margin-bottom:50px;" /> ## Session 5: Computer vision in R ### Ben Raymond, Adrien Ickowicz ##### with valuable contributions from many others... --- ## Session 5 setup Update your copy of the workshop repo: If you are in the 'R_Workshop_2022' project, then from the RStudio menu: `Tools` -> `Version Control` -> `Pull Branches` Or from the command line in your 'R_Workshop_2022' directory: `git pull` <br /> Install the ovml package if you don't already have it: ```r install.packages("ovml", repos = c("https://openvolley.r-universe.dev", "https://cloud.r-project.org")) ``` --- ## Limitations of scout data - only the player playing the ball - doesn't capture all details - unavoidable subjectivity - manually intensive --- ## A brief history of computer vision, AI and 'deep learning' #### AI, deep learning: 1940s–1980s: artificial neural networks, backpropagation 1990s–2000s: use of GPUs, quantities of data Around 2010: "modern" computer vision approaches <br /> <hr /> See also: [Peter Norvig - The Unreasonable Effectiveness of Data](https://www.youtube.com/watch?v=yvDCzhbjYWs) --- ## A biased perspective For a prospective user of these approaches: - developing new models is hard - there are models available (with code), many to choose from - but they likely don't do quite what (or everything) you want - training networks is fiddly, time consuming, and requires lots of data ... but likely feasible - beyond trivial usage, most require domain-specific coding (Python) --- ## Openvolley aims - to make these approaches more accessible - in R - not necessarily the most up-to-date or best (from a ML perspective) - not necessarily the most efficient implementation - BUT integrated with all the other openvolley tools --- ## The ovml package ```r library(ovml) dn <- ovml_yolo() image_file <- ovml_example_image() res <- ovml_yolo_detect(dn, image_file) res ``` ``` image_number class score xmin xmax ymin ymax 1 1 person 0.9997922 559.29555 655.0639 78.05564 259.7696 2 1 person 0.9996709 828.48042 953.8777 91.11566 280.6085 3 1 person 0.9996420 742.20545 837.7763 207.96229 397.8552 4 1 person 0.9998480 655.88811 718.8102 249.73282 449.8958 5 1 person 0.9996897 81.39468 123.2998 245.04192 408.4309 6 1 person 0.9991708 982.23695 1086.7565 148.34806 346.2865 7 1 person 0.9984150 622.42059 674.1802 268.68086 430.3307 8 1 person 0.9973027 802.84929 862.9306 310.86588 469.4443 9 1 person 0.9997010 355.70534 471.9602 99.31968 256.9752 10 1 person 0.9996446 742.64417 800.3177 419.89205 577.3802 11 1 person 0.9996887 1162.08036 1223.8335 450.21562 529.3872 ``` --- ```r ovml_ggplot(image_file, res) ``` <!-- --> --- ## The ovml package Helpers — extracting video frames ```r my_video_file <- ovdata::ovdata_example_video("190301_kats_beds") my_video_file ``` ``` [1] "/home/ben/R/x86_64-pc-linux-gnu-library/4.2/ovdata/extdata/video/2019_03_01-KATS-BEDS-clip.mp4" ``` ```r library(ovideo) image_file <- ov_video_frame(my_video_file, t = 3.2) image_file ``` ``` [1] "/tmp/Rtmpu8a1Kh/file3560a3855f2cc.jpg" ``` --- ## The ovml package Helpers — converting to court coordinates Also in the ovideo package: - `ov_shiny_court_ref` to define the transformation function - `ov_transform_points` to transform from image coordinates to court coordinates and vice-versa --- ## The ovml package Helpers — converting to court coordinates ```r ref <- ov_shiny_court_ref(image_file) ref$court_ref ``` ``` image_x image_y court_x court_y 1 0.05397063 0.02129301 0.5 0.5 2 0.95402573 0.02294600 3.5 0.5 3 0.75039756 0.52049712 3.5 6.5 4 0.28921230 0.51884413 0.5 6.5 ``` --- ## The ovml package Helpers — putting those together ```r res <- ovml_yolo_detect(dn, image_file) head(res) ``` ``` image_number class score xmin xmax ymin ymax 1 1 person 0.9987562 249.3282 370.6062 93.12426 298.4178 2 1 person 0.9997475 796.2333 869.6249 260.13521 433.4992 3 1 person 0.9996364 712.9836 786.1387 224.97480 396.0520 4 1 person 0.9996997 535.7915 595.6890 266.06239 430.9295 5 1 person 0.9999343 652.9980 724.1900 247.44727 452.2819 6 1 person 0.9990908 1125.6339 1217.8855 170.70665 379.4568 ``` --- ## The ovml package Helpers — putting those together ```r library(magick) image_meta <- image_info(image_read(image_file)) res <- res %>% mutate(x = (xmin + xmax) / 2 / image_meta$width, y = ymin / image_meta$height) head(res) ``` ``` image_number class score xmin xmax ymin ymax 1 1 person 0.9987562 249.3282 370.6062 93.12426 298.4178 2 1 person 0.9997475 796.2333 869.6249 260.13521 433.4992 3 1 person 0.9996364 712.9836 786.1387 224.97480 396.0520 4 1 person 0.9996997 535.7915 595.6890 266.06239 430.9295 5 1 person 0.9999343 652.9980 724.1900 247.44727 452.2819 6 1 person 0.9990908 1125.6339 1217.8855 170.70665 379.4568 x y 1 0.2421619 0.1293393 2 0.6507259 0.3612989 3 0.5855947 0.3124650 4 0.4419846 0.3695311 5 0.5379641 0.3436768 6 0.9154373 0.2370926 ``` --- ## The ovml package ```r court_xy <- ov_transform_points(res[, c("x", "y")], ref = ref$court_ref, direction = "to_court") res <- bind_cols(res, setNames(court_xy, c("court_x", "court_y"))) head(res) ``` ``` image_number class score xmin xmax ymin ymax 1 1 person 0.9987562 249.3282 370.6062 93.12426 298.4178 2 1 person 0.9997475 796.2333 869.6249 260.13521 433.4992 3 1 person 0.9996364 712.9836 786.1387 224.97480 396.0520 4 1 person 0.9996997 535.7915 595.6890 266.06239 430.9295 5 1 person 0.9999343 652.9980 724.1900 247.44727 452.2819 6 1 person 0.9990908 1125.6339 1217.8855 170.70665 379.4568 x y court_x court_y 1 0.2421619 0.1293393 1.012716 1.244800 2 0.6507259 0.3612989 2.679859 3.633504 3 0.5855947 0.3124650 2.338898 3.004974 4 0.4419846 0.3695311 1.632314 3.757907 5 0.5379641 0.3436768 2.117389 3.400127 6 0.9154373 0.2370926 3.706100 2.172419 ``` --- ## The ovml package Helpers — putting those together ```r ggplot(res %>% filter(court_y < 7 & court_x < 4), aes(x = court_x, y = court_y)) + ggcourt(labels = NULL) + geom_point() ``` <!-- --> --- ## ovml — Application ideas #### Court positioning Recall from yesterday, the volleydef app output:  - uses one defender position per dug attack - what can we get via ovml? --- ## ovml — Court positioning ```r library(datavolley) library(ovdata) dv <- ovdata_example("190301_kats_beds-clip", as = "parsed") vt <- plays(dv) %>% filter(attack_code %in% c("X6", "V6") & team == "MKS Będzin") %>% pull(video_time) image_files <- ov_video_frame(my_video_file, t = vt) res <- ovml_yolo_detect(dn, image_files) res <- res %>% mutate(x = (xmin + xmax) / 2 / image_meta$width, y = ymin / image_meta$height) court_xy <- ov_transform_points(res[, c("x", "y")], ref = ref$court_ref, direction = "to_court") res <- bind_cols(res, setNames(court_xy, c("court_x", "court_y"))) res <- res %>% filter(class == "person" & court_y >= 3.5 & court_y < 7 & court_x > 0 & court_x < 4) ``` --- ## ovml — Court positioning ```r ggplot(res, aes(x = court_x, y = court_y)) + ggcourt(labels = NULL, court = "upper") + geom_point(color = "blue") ``` <!-- --> --- ## ovml — Court positioning Positions vs left-side attacks: <img src="extra/x5-v5-def.png" style="max-height:45vh;" /> --- ## ovml — Court positioning (beach) Images courtesy Tyler Widdison <img src="extra/tw-image1.png" style="float:left; clear:none; width:60%;" /> <img src="extra/tw-image2.png" style="float:right; clear:none; max-height:50vh;" /> --- ## Next steps - tracking of players, and the ball - identify individuals, follow their movements - fill in gaps (occlusions, missed detections) - some unique challenges --- ## Next steps — tracking <img src="extra/players-tracked.png" style="max-height:55vh;" /> --- ## Next steps — tracking <video src="extra/kats3_tracked.mp4" controls style="margin-top:-20vh;"/> --- ## Next steps — height <img src="/home/ben/R/x86_64-pc-linux-gnu-library/4.2/ovdata/extdata/images/2019_03_01-KATS-BEDS-frame.png" style="max-height:45vh;" /> - multiple cameras? --- class: logo <img src="extra/leal_two_views.png" /> --- ## Next steps — two camera example <img src="extra/leal_apparent_positions.png" /> --- ## Next steps — two camera example <img src="extra/leal_resolved_positions.png" style="max-height:100vh; margin-top:-20px;" /> Estimated height: 58cm --- ## Next steps — two camera example <video id="vid2c" src="extra/demo_ball_tracking.mp4" controls onplay="var vid = document.getElementById('vid2c'); vid.playbackRate = 0.5; "/> --- ## Next steps — two camera example Estimated contact heights: - serve height (Isac, Brazil #12) is 3.49m (top of the ball) - back-row spike by Alan (Brazil #21) at 3.43m (See https://untan.gl/multicamera-volleyball-tracking.html) --- class: center, middle ## Enough with the machine learning already --- ## openvolley - GitHub issues - analytics snippets: https://openvolley.github.io/volley-analytics-snippets