Rock Art in Three Dimensions – Documentation, Research, and Outreach
This infrastructure project will establish a collaboration between archaeology, data science, and digital humanities to develop an effective tool that uses 3D data to explore rock art to its fullest extent. The Svenskt Hällristningsforskningsarkiv (SHFA) will create an interdisciplinary platform for data-driven analysis in collaboration with the Centre for Digital Humanities at Gothenburg University and the Department of Computer Sciences and Engineering at Chalmers. The platform will use methods from the fields of Computer Vision and Machine Learning for image recognition, object detection, and segmentation. This not only provides the best visualization of engraved images currently possible, but the trained algorithm will eventually be able to recognize images, for example, boats from different periods.
The project aims to integrate the interdisciplinary platform into the research infrastructure of the SHFA and make use of the data visualization in a mobile app that will be developed during the project. By integrating 3D data, modern data-driven visualization, and analysis, the suggested project strengthens and enhances the infrastructure of the SHFA to account for all three dimensions of rock art studies: documentation, research, and outreach.
Final report
Final report: Rock Art in Three Dimensions – Documentation, Research, and Outreach (IN18-0557:1)
1. Aims, development, and results
Sweden’s more than 20,000 rock art sites provide one of the richest and best-preserved legacies for the Bronze Age (BA) in Europe with the densest occurrence in the UNESCO world heritage area “Rock art in Tanum”. The arrival of new 3D documentation made older reductive methods insufficient posing three specific challenges:
1. Updating the record with more precise and uniform documentations especially of the world heritage.
2. New documentation is leading to new research results that need to be conveyed to the public.
3. The amount of data is increased by orders of magnitude which requires new research infrastructure, i.e., tools and methods, to analyse the material.
To solve these issues Swedish Rock Art Research Archives (SHFA) collaborated with its partners to create a close interdisciplinary collaboration between archaeology, data science, and digital humanities to develop effective tools using 3D data, Computer Vision, and Machine Learning to explore rock art to its fullest extent. This work has driven 3D documentation and the development of methodologies and tools. Finished results have all been made publicly available and the outcome of the project became a cornerstone for the modernisation of the SHFA itself.
The BA rock art were mostly carvings which were produced by taking away material from the bedrock. Thus, they have depth which older reductive methods discard. To update the record and document the depth of carvings we have created c. 800 new records at different scales from individual images to full sites in collaboration with the County Administrative Board for West Sweden, Vitlycke Museum, and Tanums Hällristningsmuseum Underslös.
3D files can become very large which increases computing power requirements and times for AI training. To make this more feasible, two new 3D data visualization methods were developed which also helped to make the results easier visible in publications and for the public. Both methods were based on depth maps which are images that use the metadata of every pixel to store its own depth value. While the output looks like older rubbings, the colour distribution is based on numerical values rather than the bias of the documenter, and therefore, the colour distribution is equal across the entire surface allowing to study depth differences. Both methods have been bundled up into the Topographic Visualization Toolbox (TVT; https://tvt.dh.gu.se/).
All AI models were trained on the outputs of the two visualization techniques. As this was pioneering work for Scandinavian rock art it was not possible to know beforehand which approach would work best. Therefore, we trained three different models based on convolutional neural networks in collaboration with Chalmers Technical University and the Centre for Digital Humanities (GU; now Gothenburg Research Infrastructure for the Digital Humanities or GRIDH). The training used annotated data, i.e., sets of documentation in which the rock art location had been marked and given a class label by an expert. These were simple bounding boxes at first, but later the annotation was more actualistic.
All models worked reasonably well, but we found out that AI has profound difficulties dealing with the outcomes of human creativity which can be fluid, ambiguous and that can be transformed over centuries. While this may lead to misclassification, it is worth engaging with the results because it can teach us about our biases, BA rock art making, and even lead to new results (see publications). These results were used to spark a new discussion about traditional archaeological methods, and about the capability of Ais to deal with complex material such as rock art.
The trained models were implemented in an app prototype that automatically identifies and classifies rock art which is currently in a pre-release version. 3D models and visualizations are also used in prototypes for two apps showing rock art and research results to the wider public. Both use augmented reality with one being used on-site to display rock art to visitors without the need to paint the rocks, the other is for off-site use at home. All this work highlighted the need to modernize the SHFA’s website and database, which is a process now underway.
2. Use, integration, and long-term maintenance
The code for the visualization and the first trained AI model were uploaded to GitHub (https://github.com/Swedish-Rock-Art-Research-Archives). All other code based on this project will be uploaded there soon. GRIDH and SHFA collaborate on the long-term maintenance and continued development of the AI models, the visualization app TVT, and the segmentation app, for example in the project “Tracing the carvers on the rocks” (VR). TVT provides visualizations to several projects (see publications) and supported three PhD students and three MA theses. The on-going development and maintenance of the new infrastructure at the SHFA was recognised by the faculty which increased its financial support from one to three million SEK for the years 2022-2023. The prototypes of the two apps aimed at outreach will be taken over by Vitlycke Museum for further development and maintenance. 1164 sites and visualizations directly related to this project have been made publicly available already (www.shfa.se). Overall, more than 60.000 scans, database entries, and publications of documentation on the website have been made. The upload of the 3D models and visualizations helped the SHFA website to increase its unique page views to two million.
3. Deviations from original plan and staff
Initial staff and technical solutions changes have taken place (see the follow-up report). The project was severely hampered by the COVID-19 pandemic caused interruptions in fieldwork. This meant during 2020 and early 2021 fewer documentations than intended were produced and field tests for the outreach app had to be delayed. Furthermore, it was decided that during the transition to the new SHFA website publishing is temporarily suspended on the old website to avoid data loss and double work. The staff who have worked in the project on varying percentages of full-time are the documentation team (Ellen Meijer, Jacob Alvå, Rich Potter), research engineers for AI training and visualization (Ashely Green, Rich Potter, Oscar Ivarsson, Victor Wåhlstrand Skärström), AR and VR app development (Jonathan Westin, Aliisa Råmark), coordination, administration and advising (Christian Horn, Johan Ling, Ulf Bertilsson).
4. Accessibility
TVT is available long-term through a website hosted by GRIDH (https://tvt.dh.gu.se/) which includes guidelines and a tutorial. The segmentation app and guidelines will be delivered the same way. The code for the visualization and AI training is available open source through Github. TVT and the segmentation app will eventually be integrated into the new website.
All documentation uploaded documentations are available through a CC-BY-NC-ND license which will soon be changed to CC-BY which is less restrictive. 3D models have so far been hosted on Sketchfab which will change when the new website to 3dhop to display models in higher quality.
5. Publications
The project has led to 19 directly and indirectly related publications including 10 peer reviewed articles, one anthology in Open Archaeology, and two public outreach articles (see publication list).
1. Aims, development, and results
Sweden’s more than 20,000 rock art sites provide one of the richest and best-preserved legacies for the Bronze Age (BA) in Europe with the densest occurrence in the UNESCO world heritage area “Rock art in Tanum”. The arrival of new 3D documentation made older reductive methods insufficient posing three specific challenges:
1. Updating the record with more precise and uniform documentations especially of the world heritage.
2. New documentation is leading to new research results that need to be conveyed to the public.
3. The amount of data is increased by orders of magnitude which requires new research infrastructure, i.e., tools and methods, to analyse the material.
To solve these issues Swedish Rock Art Research Archives (SHFA) collaborated with its partners to create a close interdisciplinary collaboration between archaeology, data science, and digital humanities to develop effective tools using 3D data, Computer Vision, and Machine Learning to explore rock art to its fullest extent. This work has driven 3D documentation and the development of methodologies and tools. Finished results have all been made publicly available and the outcome of the project became a cornerstone for the modernisation of the SHFA itself.
The BA rock art were mostly carvings which were produced by taking away material from the bedrock. Thus, they have depth which older reductive methods discard. To update the record and document the depth of carvings we have created c. 800 new records at different scales from individual images to full sites in collaboration with the County Administrative Board for West Sweden, Vitlycke Museum, and Tanums Hällristningsmuseum Underslös.
3D files can become very large which increases computing power requirements and times for AI training. To make this more feasible, two new 3D data visualization methods were developed which also helped to make the results easier visible in publications and for the public. Both methods were based on depth maps which are images that use the metadata of every pixel to store its own depth value. While the output looks like older rubbings, the colour distribution is based on numerical values rather than the bias of the documenter, and therefore, the colour distribution is equal across the entire surface allowing to study depth differences. Both methods have been bundled up into the Topographic Visualization Toolbox (TVT; https://tvt.dh.gu.se/).
All AI models were trained on the outputs of the two visualization techniques. As this was pioneering work for Scandinavian rock art it was not possible to know beforehand which approach would work best. Therefore, we trained three different models based on convolutional neural networks in collaboration with Chalmers Technical University and the Centre for Digital Humanities (GU; now Gothenburg Research Infrastructure for the Digital Humanities or GRIDH). The training used annotated data, i.e., sets of documentation in which the rock art location had been marked and given a class label by an expert. These were simple bounding boxes at first, but later the annotation was more actualistic.
All models worked reasonably well, but we found out that AI has profound difficulties dealing with the outcomes of human creativity which can be fluid, ambiguous and that can be transformed over centuries. While this may lead to misclassification, it is worth engaging with the results because it can teach us about our biases, BA rock art making, and even lead to new results (see publications). These results were used to spark a new discussion about traditional archaeological methods, and about the capability of Ais to deal with complex material such as rock art.
The trained models were implemented in an app prototype that automatically identifies and classifies rock art which is currently in a pre-release version. 3D models and visualizations are also used in prototypes for two apps showing rock art and research results to the wider public. Both use augmented reality with one being used on-site to display rock art to visitors without the need to paint the rocks, the other is for off-site use at home. All this work highlighted the need to modernize the SHFA’s website and database, which is a process now underway.
2. Use, integration, and long-term maintenance
The code for the visualization and the first trained AI model were uploaded to GitHub (https://github.com/Swedish-Rock-Art-Research-Archives). All other code based on this project will be uploaded there soon. GRIDH and SHFA collaborate on the long-term maintenance and continued development of the AI models, the visualization app TVT, and the segmentation app, for example in the project “Tracing the carvers on the rocks” (VR). TVT provides visualizations to several projects (see publications) and supported three PhD students and three MA theses. The on-going development and maintenance of the new infrastructure at the SHFA was recognised by the faculty which increased its financial support from one to three million SEK for the years 2022-2023. The prototypes of the two apps aimed at outreach will be taken over by Vitlycke Museum for further development and maintenance. 1164 sites and visualizations directly related to this project have been made publicly available already (www.shfa.se). Overall, more than 60.000 scans, database entries, and publications of documentation on the website have been made. The upload of the 3D models and visualizations helped the SHFA website to increase its unique page views to two million.
3. Deviations from original plan and staff
Initial staff and technical solutions changes have taken place (see the follow-up report). The project was severely hampered by the COVID-19 pandemic caused interruptions in fieldwork. This meant during 2020 and early 2021 fewer documentations than intended were produced and field tests for the outreach app had to be delayed. Furthermore, it was decided that during the transition to the new SHFA website publishing is temporarily suspended on the old website to avoid data loss and double work. The staff who have worked in the project on varying percentages of full-time are the documentation team (Ellen Meijer, Jacob Alvå, Rich Potter), research engineers for AI training and visualization (Ashely Green, Rich Potter, Oscar Ivarsson, Victor Wåhlstrand Skärström), AR and VR app development (Jonathan Westin, Aliisa Råmark), coordination, administration and advising (Christian Horn, Johan Ling, Ulf Bertilsson).
4. Accessibility
TVT is available long-term through a website hosted by GRIDH (https://tvt.dh.gu.se/) which includes guidelines and a tutorial. The segmentation app and guidelines will be delivered the same way. The code for the visualization and AI training is available open source through Github. TVT and the segmentation app will eventually be integrated into the new website.
All documentation uploaded documentations are available through a CC-BY-NC-ND license which will soon be changed to CC-BY which is less restrictive. 3D models have so far been hosted on Sketchfab which will change when the new website to 3dhop to display models in higher quality.
5. Publications
The project has led to 19 directly and indirectly related publications including 10 peer reviewed articles, one anthology in Open Archaeology, and two public outreach articles (see publication list).