EnviLOD: Summary of User Feedback on the Semantic Search Interface

We have now completed the analysis of the user feedback received during two evaluation sessions, carried out towards the end of the 7 month long #EnviLOD project:

• The first smaller user evaluation session was held internally at HR Wallingford and was used as an initial test of our semantic search user interface (UI). Following this, some minor adjustments were made to the UI.
• We held  a much larger user workshop at the British Library at the end of January 2013. The core of our findings are derived from this second, much larger user evaluation.

Overall, workshop participants found the EnviLOD semantic search UI (see a screen shot at the end of this post) easy to learn and use. In some detail, 87.5% of users disagreed or strongly disagreed with the statement that the UI is unnecessarily complex. Similarly, 81.25% agreed or strongly agreed that the EnviLOD UI is easy to use. 93.75% of participants also felt they can use the system without needing to learn more about it first. With respect to quality, we asked whether the results returned by the semantic search UI made sense to the users, where 75% agreed or strongly agreed with this statement. 

We also carried out a task completion analysis, comparing semantic search versus keyword search. For details on task definitions, success rates, and more in-depth questionnaire results, please see our WP2 User Feedback Report.

The group discussions and the three written feedback questions on the user feedback forms allowed us to elicit a number of small, easy to implement changes to the UI, which we hope will lead to improved usability in the future. We are planning to implement these during follow-up research  in the next 1-2 months and then carry out a second user-based evaluation. This time we will include users recruited online, who will not be shown a demonstration of the semantic search UI in advance.

In addition, we also elicited a number of more challenging ideas for future improvements, which cannot be easily addressed within the scope of  short, informal follow-up work. The most substantial of these include the implementation of a natural language interface, map-based visualisations, support for user feedback on search results and search query  refinement, for example. These are all valuable possible extensions to this work, including building a natural language interface (see (Damljanovic et al, 2013) for some preliminary work which we have done on this already). However, to fully address these, a much longer and larger, two or three year project would be required.

Lastly, a known limitation of this evaluation which could have impacted on our results came from the limited amount of content which we could index in the experimental system. This was due to copyright issues with the majority of environmental science content at the British Library. Due to this limited content, participants were not always able to judge how comprehensive or accurate the semantic search was, especially if compared against results offered by Google. Since the British Library is now planning to integrate the EnviLOD semantic enrichment tools within the advanced Envia Labs functionality, future work on this tool could potentially be able to evaluate on this more comprehensive data, through the Envia system.