CLARIAH Media Studies and MIMEHIST in Zürich – A Report

Early September, Liliana Melgar and I received an invitation from Barbara Flückiger , Professor in Film Studies at the University of Zürich, to participate in the “Colloquium Visualization Strategies for the Digital Humanities”. The aim of the day was to bring together experts to discuss film data visualization opportunities in relation to Professor Flückiger’s current research projects on the history of film colors. Currently, Flückiger leads two large-scale projects on this topic: the ERC Advanced Grant FilmColors (2015-2020) and the Filmfarben project funded by the Swiss National Science Foundation (2016-2020). A presentation of the projects’ team members can be found here .

As a scholar, Barbara Flückiger has in-depth expertise on the interrelation between film technology, aesthetics and culture covering especially aspects of film sound, special effects, film digitization and film colors in her research. In recent years, her research has increasingly focussed on film colors, especially since the launch of the online database of film colors Timeline of Historical Film Colors in 2012 after a successful crowdfunding campaign. The Timeline of Historical Film Colors has since grown to become one of the leading authoritative resources on the history and aesthetics of film colors – it is presented as “a comprehensive resource for the investigation of film color technology and aesthetics, analysis and restoration”. It is now consolidating this position as it is being followed up by the two large-scale research projects mentioned above which merge perspectives from film digitization, restoration, aesthetic and cultural history.

These projects are entering a phase in which the involved researchers are beginning to conceive ways of visualizing the data they have created so far and need to consider the potential value which data visualization may have for historical research on film color aesthetics, technology and reception.

In the following we share a few impressions from the day and thoughts on the discussions which arose.

Organising and mining film data

The first speaker of the day was Eric Hoyt , Associate Professor of Media and Cultural Studies at the University of Wisconsin-Madison. In spite of having just arrived the previous day, Hoyt was not visibly jetlagged and gave a fantastic presentation on his research activities in the Arclight and Lantern projects. Both projects work with materials made available through the online Media History Digital Library (MHDL) and offer different ways of visualizing patterns in them. The Lantern project involved the digitisation and OCR’ing of 900.000 pages from public domain trade journals made available in this collection, while Arclight works with around 2 million pages also available through MDHL.

The tools developed in these projects enable data mining and visualisation in combination with simpler, standard search queries to supplement traditional archival research on silent era periodicals – among other periods of media history – in order to advance our understanding of film’s cultural history. Using the visualization options developed by these projects it is possible to analyse word patterns within the MDHL’s journals, so as to understand how certain topics and trends emerged and understand the historical networks of popular cultures that conditioned them. Taking a distant reading approach, the projects thus explore “the great unread” of tradepapers to broaden the horizon of media historians and to open new research avenues. As Hoyt has poignantly pointed out, previous scholarship and access projects on relying on analogue formats, especially microfilm, have established a reference frame in which periodicals such as Variety and Photoplay appear as canonical source material. This has had the effect that a wide range of magazines which were published in large numbers back in the day have been largely neglected in contemporary scholarship on film exhibition, distribution and reception. For example, as Hoyt has discussed in one of his articles, the magazine Film Fun , which – while published in relatively large numbers – was never cited in any article available in the widely used academic journal database JSTOR. There is much more to read on the development and use of Arclight and Lantern here and here .

Based on his experience in these projects, Hoyt presented a set of concrete and highly intriguing suggestions for further development of the Filmcolors-project and the functionalities of the Timeline of Historical Film Colors. The suggestions ranged from specific ways of organising data in the Timeline’s database so as to make it more searchable to different ways of visualizing the data created by the Filmcolors/Filmfarben researchers.

As Hoyt pointed out, when it comes to allowing users to explore a digitised collection through visualizations it helps their reasoning to be able to do this by making use of a wider range of visualization formats. This allows researchers to see different views and perspectives and make them understand that there is not one finite and ideal representation of data. On the other hand, Hoyt also pointed out that while we tend to favour interactive visualizations and multiplicity of viewpoints because they highlight and facilitate research as (contingent) processes, rather than finite end products, we may also consider whether static visualizations are not sometimes more efficient for getting our point across poignantly to a broader public, in order to create discussion among researchers. In both cases it is key to think about who we are making visualizations for and presenting results to.

Another very interesting suggestion was to enable the comparison of scanned film frames included in the timeline with computational methods. In the Timeline, users can currently select frames for comparison manually based on their impressions of the images’ visual features. To develop this feature, it would, as Hoyt pointed out, be interesting to see how a computer would compare images as a way to both assist and challenge human vision. With these suggestions, Hoyt nicely built a bridge from the more information theoretical parts of his talk to the rest of the day’s presentations which would focus to a greater degree on the visualisation of film data and pattern recognition in images.

Visualising film data

Following Hoyt’s presentation the focus of the day’s program turned to video annotation, analysis and visualization. This comprised a presentation on the Filmcolors project with a general introduction and discussion by Barbara Flückiger which outlined the goals and components of her research projects. As Barbara explained, the research projects’ comprise different steps, in particular off-line video segmentation and annotation, the creation of a Filemaker database based on the annotation work as well as (semi-) automatic color analysis and visualization. This was followed by brief introductions to the programming work which supports the annotation and visualization procedures of the projects’ film analyses.

In the first short presentation following Barbara’s introduction, Gaudenz Halter, programmer in the Filmcolors project, explained how he is currently developing an extension for the video annotation software ELAN, used in the projects for film segmentation and labelling, to tweak it in such a way that it better fits the project’s purposes. ELAN is a video annotation software developed at the Max Planck Institute for Psycholinguistics in Nijmegen in the Netherlands used mainly for annotating video and audio material in linguistic research. For this reason it is not the best suited for film and media historians’ purposes, and has some drawbacks in particular when it comes to color analysis (this is a topic which Barbara discusses in an article forthcoming in the Moving Image journal). However, ELAN is attractive to work with because its userbase and institutional backing secures its sustainability, something which has not been the case for the otherwise excellent annotation software programs such as Anvil, Lignes de Temps and Advene which were tailored to film and media analysis to a greater degree.

Subsequently, Enrique Paredes gave a presentation of the work he is currently involved in carrying out at the University of Zürich’s Department of Informatics’ Visualization and Multimedia Lab to support the projects’ visualizations work, which explores the YOLO (You Only Look Once) detection method. This extremely exciting work entails the use of computational procedures to detect and distinguish between foreground and background and to visualize patterns in their interplay as a basis for film analysis.

The interplay between foreground and background in the use of colors in mise-en-scène may be seen as a particularly important one as Flückiger explained, for instance in the way they contrast each other. The results achieved with these procedures also seem to support this assumption. The detection of foreground and background allows the researchers to produce image sets consisting of respectively foreground and background categories. Subsequently, the researchers can use ImagePlot to visualize the interplay between foreground and background color schemes throughout a film’s duration, making visualizations of the respective image sets separately or in combination as a basis for comparative analysis. So far the team has experimented with ImagePlot analysis of three films, among which Gentlemen Prefer Blondes (dir.: Howard Hawks, US, 1953) and Jigokumon (dir.: Teinosuke Kinugasa, Japan, 1953). These visualization results are still not published and can therefore not be shared here. But they were impressive and striking in their ingenious use and appropriation of ImagePlot to reflect film’s temporal dimension and in their support of a close reading analysis of films attending to color patterns in foreground and background.

Following up on this presentation after a break, Everardo Reyes , Associate Professor in Digital Humanities at the Paris 8 University, gave an introduction to his work with ImageJ and ImagePlot , which he has carried out in the context of media scholar Lev Manovich’s Cultural Analytics Lab . As a trained semiotician and highly skilled Java programmer, Reyes has tweaked ImageJ to be able to give shape to his image sets in multifarious ways experimenting with shapes, different levels of interactivity and interface design. His work is absolutely stunning and explores a broad range of subjects in visual culture from the covers of the skater magazine Trasher , the visual features of rock album covers and bands – in particular of Nirvana – to the history of modern art works and painters and patterns of hue, saturation and brightness in them. Reyes’ work can be explored here .

Considered in connection to Paredes’ presentation, Reyes’ work seemed to underline the versatility and broad range of possibilities offered by ImageJ for media studies. While ImageJ has been in use for approximately a decade now in media studies – if not more – it is intriguing to see how scholars find new ways of tweaking it to different analytical ends and reach a level of theorization of the visualizations they produce which can accommodate for the reflexive and ambiguous perspectives which are cherished and necessary in humanistic interpretation. Reyes’ visualization work for instance, as was discussed after the presentation, show how scholars may try to display their work processes and procedures rather than producing finite epistemic images.

In this part of the workshop, Liliana and I also presented on our work in CLARIAH. Liliana presented the Media Suite and the research pilot projects, and we made a short demo of the video annotation functionalities that are being built as part of the CLARIAH research infrastructure. This was followed by a presentation of the MIMEHIST project which has previously been discussed in depth on this blog.

Closing expert panel discussion at the Colloquium “Visualization Strategies for the Digital Humanities” in Zürich.

To conclude the colloquium, the presenters answered questions from Professor Flückiger and her team about the interrelation of her project with the tools we presented, and inquired about our opinions of what the next steps should be in relation to video annotation – manual, semi-automatic and automatic – in a collaborative perspective.

Film Scanning and/as Source Criticism

Outside of the colloquium program, and benefiting from the fact that we had almost a full day available before going back to Amsterdam, Bregt Lameris, Postdoctoral Researcher in the Filmcolors project, helped us organizing a visit to the lab where Giorgio Trumpy, also Postdoctoral Research in Filmcolors, is currently carrying out his research on illumination and film scanning techniques. The Filmcolors project is working closely with the Cinegrell film development lab (formerly Egli Film) located on the outskirts of Zürich, which is the last exisiting film lab in Switzerland. Here we were given a tour by staff members who generously shared their time to show the lab’s facilities and talk about their work.

At Cinegrell, Giorgio Trumpy – together with Martin Weiss, Senior Researcher and Restorer in Filmcolors – have their own work space where the project’s Kinetta Archival Scanner is being used and modified to experiment with different scanning techniques for the films selected for the project. This work is fundamental not only for perfecting existing restoration methods but also in questioning and reconsidering already established practices.

Basically, one of the fundamental hypotheses being explored in Trumpy’s research is that the scanning techniques which are currently used for archival films are not the best suited to bring out the visual properties of the material of films stocks used during the silent period. Consequently, film restorers and historians may need to rethink their assumptions about present-day color film restoration a great deal as well as the way in which we understand silent films as historical sources. The results of this research are extremely illuminating (pun intended!) in terms of understanding the intricate relations between film scanning and the appearance of historical film colors.

Yet, these results – while partially published – are not entirely public at this point and – as Trumpy explained – still need a firmer empirical foundation, so the details of this work will have to wait until they are presented by the team in media studies conferences and in the archival film festival circuit. As with the research team’s visualization work this is really something to look forward to!

Concluding impressions and thoughts

We got back from the colloquium with many fresh perspectives on what it means to do film history in an increasingly digital age with new research tools and methods in a process of interdisciplinary collaboration. From the hermeneutical work which goes into a close reading analysis of films through segmentation and labelling to the scientific processes through which the evidence – the films – we work with are shaped, Filmcolors promises – and will surely also deliver – a fundamental historical reconsideration of a significant aspect of film technology and aesthetics. It is impressive to see an entire research area being reviewed and scrutinized on so many levels in such a large-scale research project, something which is also the goal of a “high risk” funding scheme as the ERC Advanced Grant.

Moreover, from a film studies perspective, where archival research is often a lone endeavor which produces exegetic readings of a microscopic area and body of evidence it is intriguing to see how a project such as Filmcolors allows for thinking media history on a grander scale in a large team. In particular, in a time when it is customary for researchers in film studies to cautiously add an “a” before history or to use the plural “histories” to distance oneself from the universalizing aspirations of historical research in early film studies – think for instance of the recurrent critiques of Georges Sadoul’s world film histories or the universal filmographies of a French historians such as Jean Mitry – the presentation of the Timeline of Historical Film Colors as a comprehensive resource may initially seem bold. Yet, it is justified and exciting exactly because of the wide range of scientific activities, processes and critical theoretical perspectives it covers: spanning innovative scientific research on spectroscopy to cultural studies perspectives on film colors. The Timeline is certainly the most qualified and convincing attempt to produce a comprehensive resource on historical film colors to date.

Beyond the Timeline of Historical Film Colors one may also tentatively ponder if the project can be seen as reflecting an emerging renewed interest in the comprehensiveness promised by encyclopedic formats for the exploration of film’s technological history. For instance, the recently launched project of the Canadian research program Technès to develop an encyclopedia of cinema techniques ( Encyclopédie raisonnée des techniques du cinéma ) seems to share ambitions with the research projects currently being carried out under Professor Flückiger’s supervision in its aspiration to cover the broad historical lines of film technology in a multi-authored, interdisciplinary database format.

Based on these examples, one may contend that one of the great advantages which the digital humanities’ emergence has had for film studies is that it allows for collaborative forms of database creation which brings notions of comprehensiveness back into historical research in refreshingly dynamic and engaging ways by involving a greater number of researchers in the creation of reference resources and by nurturing critical discussions of the foundations of previous and current historical research across disciplinary boundaries in this process. This will be extremly interesting to follow in the coming years, especially with regard to the Timeline’s future development and results.

References

Durteste, Pierre, “Faut-il oublier Georges Sadoul ?”, 1895. Mille huit cent quatre-vingt-quinze , no. 44 (2004), 29-46.

Hoyt, Eric, “Lenses for Lantern: Data Mining, Visualization, and Excavating Film History’s Neglected Sources”, in Film History: An International Journal, Volume 26, Number 2 (2014): 146-168.

Hoyt, Eric, Kit Hughes, Derek Long and Anthony Tran, “Scaled Entity Search: A Method for Media Historiography and Response to Critiques of Big Humanities Data Research.” IEEE BigData 2014 Proceedings. (October 978-1-4799-5666-1): 56-64.

Flueckiger, Barbara, “Material Properties of Historical Film in the Digital Age”, in NECSUS – European Journal of Media Studies , no. 2, vol. 1 (2012)

Stutz, Olivia Kristina, Algorithmische Farbfilmästhetik. Historische sowie experimentell-digitale Notations- und Visualisierungssysteme des Farbfilms im Zeichen der Digital Humanities 2.0 und 3.0 . MA Thesis, Universität Zürich, 2016.