OK , as promised, here is our phase 2 update for the Big Data VR Challenge (click here to read phase 1 before proceeding).
So what’s new? Well, we had a one-on-one consultation with our team of scientist advisors to determine if we were on the right track in order to produce something practical that an actual scientist could and would use. The answer was ‘erm..maybe.’ It appears that our approach was (as designed) geared towards accessing an individual’s data; what samples they had provided over the years, what information from surveys was provided and what bio-data was given. Now this would be a very useful tool for a particular sort of data research, i.e. a private medical practice for instance (which we are now also thinking about as a side project) but for the ALSPAC data set in particular, researchers are generally more concerned with looking for trends within the overall data set and less concerned with any individual data sample. This Cohort Study approach would require more variables to be ‘eyeballed’ at once in order for trends to emerge.
So we requested a new set of data which would better reflect this form of research. While we waited for this data to be simulated (the actual live data is confidential and so we can only work with simulated data for our project), we went back to the drawing board to look at ways to visualise multiple variables within our VR environment.
We started at the beginning with the simplest piece of 3D geometry; the pyramid (only 4 vertices and 4 triangle faces), and so what can we do with this shape? For starters we can move the verts around in constrained axis to reflect different values.
This means that we can start to build up geometry from multiple pieces of data which we can then quickly assess visually.
In the above image, the 2D triangles show different amounts of data variables to give the overall shape. Example 4 (top right) shows 4 data types in the brightly coloured triangles: width (at the base line) reflects waist size, height (peak above the base line) shows height of the individual, Inverse to this, the length of the peak below the base line shows Blood Pressure and the overall colour shows the sex of the individual. Furthermore, by overlaying darker triangles on top, we can show comparison data for previous entries, last year perhaps. To take this into the 3rd dimension, we extend out the top and bottom vertices towards the viewer to show another variable, say BMI. This means we now have a 3D shape that we can view at two different angles, that will display 5 separate pieces of data, as well as show a comparison for that same data (colour is not reflected in the image but that would be possible with blending) for the previous year. This is quite a lot for the humble pyramid.
We now had our concept of displaying multiple variables for an individual at once in order to compare them against each other and against other individuals. However, this still does not lend itself to looking at general trends and patterns and so now we had to take this concept and apply it to a design that would be more cohort study friendly.
Virtual Reality provides many benefits over traditional 2d screens and even interactive 2d content, one of which (as discussed previously in blog posts) is the 360 degree environment. This alone allows us more space in which to pack in lots and lots of data. Rather than scrolling down endless spreadsheets of numbers, or projecting plotted graphs onto the largest wall you can find in order to see more dots, we can spread information above, beside, below and behind the user. This also gives us more variables to play with, positional, time and audio based (we are after all creating a data environment and so as masters of our domain, we can make up the rules ourselves). Things can float, warp, move, pulsate, flash, glow, emit chimes or anything we can think of. These are all potential ways to show data in our VR world. We set to work testing this theory with our newly provided simulated data set. This new set was for 1000 individuals, each with 9 separate data variables and taken over two years; one at 7 yrs old and again at 11 yrs old. Time was now a variable and so with this we tested the idea of animating between the two with the arrow keys on the keyboard.
The above animation shows pyramids moving from a position in space generated from the data at 7 yrs to another position generated at 11 yrs. This is tough to read not only as the pyramids are small, but also because by spreading the geometry over a wide plane, perspective now warps our sense of scale, meaning we can no longer accurately compare the size of the pyramids at the front to those at the back. We were keen on minimising the amount of navigating that the user needs to do, but this now forces us to move towards the back in order to see the actual size of the pyramids. For these reasons we returned to our wrap-around style as this means we can keep the majority of data equidistant from the central user, which in turn means they do not have to move in order to compare accurately the sizes of our geometry. But would this work in practice? Our next step was to build some quick and dirty helix rings in 3dsmax and see what they looked like and felt like in VR.
3DsMax Screenshot to show variables.
How that looked in VR. (Along with some other versions we tested).
Some interesting things emerged from this test. One was confirmation that we can now start to recognise patterns in our data and notice trends appearing. For instance (difficult to tell in the video) but our brains can swiftly pick out spatial, colour and size information without too much effort on our part, so we immediately identify that most of the reds appear on the higher rings, with almost none on the bottom; telling us that perhaps blood pressure is linked somehow to hip size. Next observation was that the larger shapes were also clustered towards the top, indicating that waist size also plays a part in higher blood pressure. Closer examination shows that more large, red shapes are on the right than the left, so perhaps taller people show higher blood pressure, waist and hip size within our sample group. These rings were populated using fake data in 3dsMax and so they do not reflect any kind of scientifically accurate reading, but the concept was now proven to be effective for quick pattern recognition. As mentioned previously, there are many other potentials that we can explore to show data in this format. We are keen to put in things like, pulsing geometry to reflect pulse rate data, shape type (cuboid or spheroid) to show male or female samples, 3D spatial sound to indicate data values (pitch for instance could indicate weight), opacity, time (same as above tweening animation) to show historical data and more.
Spatial sound in particular is very interesting as a means to show data patterns, and NASA has also explored this concept by using sound chirps to inform engineers of data flow when they are not necessarily looking at the monitors. This concept could be useful to us in VR as our brains can pick up sound location in 3D space quite well. These ideas will hopefully be tested in the following phase.
Some more ideas around comparison options.
The final piece of design we looked at this phase was around interacting with the data itself. We always had in mind a UI system that would allow the user to customise the variables and filter the data themselves, therefore refining the search (meandering search was the term we coined). The newer design does away with the complete wrap around rings in favour of arcs that stacked at an angle. This design is meant to accommodate UI in the gaps between and immediately behind the user, as well as trying to keep the vast majority of the data in front of the user to lessen the need to keep turning full circle to see everything (a compromise between comfort and utilising space). The razor hydra integration has already been done and works in unreal (as you can see in the above video with the blue ‘fencing sword’ wafting around) and so next we plan to look at manipulating the data itself. The bottom of the above image shows an idea for a docking mechanic, whereby floating screens can be stretched and rotated around the user in order to display data placed there, by the user. The placement could be done by a lasso selection type method where the user draws a lasso around the interesting data to highlight it. Then the menu will allow options to cut and paste it to a docking screen for later analysis. Another method could be movable brackets that mark the start and end of the selection. Voice control could also be eventually used to quickly interact with or contextualise menus to minimise text on screen.
Where are we now?
We are still importing all the actual data into unreal to drive the visualisation. We are working towards the stacked arc design currently but this may change again as we get more feedback. The below screenshot shows the new data generating three arcs within unreal. We also have some basic UI implemented that can be used to swap out a few variables via the Razor Hydra controls. Following the screenshot is a video showing where we are currently.
Early days but we expect things to move along a little smoother from now on as we zone in on our final design.
Where to next?
The first thing is to present the above to our project scientist for another round of feedback and then we can take it from there. Provided no major pivots are required, we can look at adding in more variable, UI elements, filters and eventually a multi-user functionality so that we can harness the power of collaborative analysis within VR.
Read about the Challenge result in Phase 3.