On the final day of the “Multimodal Models for the Study of Culture” week, Tim Tangherlini, BIDS Associate Faculty Director and Cultural Analytics Group Lead, welcomed Miguel Escobar Varela to campus, describing him as a fellow “computational folklorist” in addition to his official positions as an associate professor at the Department of English, Linguistics and Theatre Studies and the deputy director of the Centre for Computational Social Science and Humanities at the National University of Singapore. He researches the changing landscape of Southeast Asian cultural heritage by combining fieldwork with computational methods such as natural language processing, computer vision and network analysis.
Escobar Varela began his talk with context, laying the groundwork for understanding the value of how cultural traditions change over time, how these traditions adapt tropes from other media, and how they respond to external cultural influences. After setting the stage, he moved onto the computational analysis and the results of his research, plus encouragement for the use of the open source tools on other cultural analytics projects, specifically those with video-based motion capture.
Photo: Miguel Escobar Varela shares images of Wayang Kulit with the talk attendees at the I School.
An introduction to Wayang Kulit (Javanese shadow puppets)
In his talk “What Dance Scholars Can Learn from Warehouse Surveillance: Emic Approaches to Temporal Action Segmentation” at the UC Berkeley School of Information, Escobar Varela focused on the data from Wayang Kulit (Javanese shadow puppets) performances posted on YouTube. A typical performance has 1,000 people in the audience and can last up to 7 hours long, so there is a huge amount of data to explore. The puppets, both painted and carved, are illuminated, so the puppets’ shadows are projected on a screen. The tradition is at least 1,000 years old, and Wayang Kulit is a very prestigious art form that uses complex and poetic language, and is semi-improvised. Even the fight scenes can get rowdy! The structure includes segments called pathets, a way of dividing the performance. The part of that structure Escobar Varela researched for this project are the comic interludes. Many followers of this tradition assert that comic interludes are increasingly longer over time, specifically since the 1990s.
An exploration of the art form: Are the comic interludes getting longer?
This is a question that matters to an interpretive community in Indonesia and beyond. Wayang has always changed over time and now there's a way to operationalize this question by using only two dimensions of the visual information of the frame and using time as a modality. Escobar Varela frames the research like this: “We have these seven hours. We have these frames. And we want to be able to label the beginning and the end of the interlude segments…” He noted that this kind of work is already done at tech companies. For example, they may want to know how long it takes a human to pack something. Convolutional neural networks are used to extract the pose of a person on a frame. They look at sequences of things that happen, instead of looking at individual frames, and predict what things are likely to come before other things. This process can be used to segment a video. It is as accurate as other models, but at a fraction of the compute costs.
Image: Slide from Miguel Escobar Varela’s talk spotlighting the Open Pack dataset found at https://open-pack.github.io/ as a valuable resource.
Computational analysis and motivations behind cultural analytics research
In their research, Escobar Varela and Shawn Liew (PhD Student at National University of Singapore) worked with samples of 100 performances each year for the previous five years, which provided a large amount of data to be processed and inspected. The result, which they found surprising, is that the comic interlude has indeed increased over time. Escobar Varela remarked that, building on this success, the process can be applied to other highly codified movement traditions like Hokkien opera and Cantonese opera where different character types move in particular ways. Additionally, he explained the unique opportunity we have now in computational humanities to finetune open-weight models, allowing us to operationalize emic theories. This is one kind of research that will help us understand how cultural traditions evolve, from Scandinavian films to silent-era movies, using data from sources like YouTube.
Watch Miguel Escobar Varela’s cultural analytics talk on the I School website. To stay in touch and join conversations of critical cultural importance, please join our Cultural Analytics mailing list by visiting this page or emailing bids-cultural-analytics+subscribe@lists.berkeley.edu, follow us on Bluesky & LinkedIn, and subscribe to the BIDS newsletter.