Guiding Deep Brain Stimulation Interventions
by Fusing Multimodal Uncertainty Regions
Alexander Bock∗
Norbert Lang†
Gianpaolo Evangelista‡
Ralph Lehrke§
Timo Ropinski¶
Scientific
Visualization Group
Linköping University
St. Barbara Hospital
Hamm
Sound Technology
Group
Linköping University
St. Barbara Hospital
Hamm
Scientific
Visualization Group
Linköping University
A BSTRACT
Deep Brain Stimulation (DBS) is a surgical intervention that is
known to reduce or eliminate the symptoms of common movement
disorders, such as Parkinson’s disease, dystonia, or tremor. During the intervention the surgeon places electrodes inside of the patient’s brain to stimulate specific regions. Since these regions span
only a couple of millimeters, and electrode misplacement has severe
consequences, reliable and accurate navigation is of great importance. Usually the surgeon relies on fused CT and MRI data sets, as
well as direct feedback from the patient. More recently Microelectrode Recordings (MER), which support navigation by measuring
the electric field of the patient’s brain, are also used. We propose
a visualization system that fuses the different modalities: imaging
data, MER and patient checks, as well as the related uncertainties,
in an intuitive way to present placement-related information in a
consistent view with the goal of supporting the surgeon in the final
placement of the stimulating electrode. We will describe the design
considerations for our system, the technical realization, present the
outcome of the proposed system, and provide an evaluation.
Index Terms: I.3.7 [Three-Dimensional Graphics and Realism]—
1
I NTRODUCTION
Due to advances in medicine, society is now facing the problems
of an aging population suffering from an increasing occurrence of
age-related diseases. One group of frequently occurring age-related
diseases are movement disorders such as Parkinson’s Disease (PD),
dystonia, or tremors that greatly affect the patients’ quality of life.
As both symptoms affecting motor skills, and the psychological
consequences of these diseases have a great impact on everyday
life, successful treatment strategies are of increasing importance.
Deep Brain Stimulation (DBS) is a well-established procedure
for reducing the symptoms of these diseases [10, 2] and capable of
improving the overall quality of life in the cases where medication
is not a viable option. To conduct a DBS, electrodes are implanted
into specific regions of the brain and then emit electrical signals to
stimulate these regions. In the case of PD the most effective target areas for electrode placement are the subthalamic nuclei (STN)
with sizes of a few millimeters [27] and are located in both hemispheres. As misplaced stimulation electrodes can have severe side
effects, including speech difficulties, increased tremor, or long term
memory problems [1], accurate electrode placement is crucial [31].
Nowadays the access path to the target region is planned using magnetic resonance imaging (MRI). However, in addition to the low
signal-to-noise ratio, a major downside of this imaging modality is
∗ alexander.bock@liu.se
† nlang@barbaraklinik.de
‡ giaev@itn.liu.se
§ rlehrke@barbaraklinik.de
¶ timo.ropinski@liu.se
that the STN is not be visible in every patient [36]. Therefore, surgeons often use an atlas-based approach to locate the STN within
the MRI data, a procedure which is prone to registration errors, is
not patient specific, and lacks visual assessment. To further verify
the electrode placement, the patient performs simple tasks during
the operation that are monitored by the surgical staff and are used
to localize the electrode’s position. Depending on the location of
the electrode, a different region of the brain is stimulated, resulting in measurable responses from the patient. These responses can
be cognitive, e. g., memory impairment, or motor-related, e. g., increased tremor or speech impairment. To further reduce the uncertainty of the electrode placement intra-operative x-ray scans can be
performed to confirm the electrode placement.
In recent years Microelectrode Recording (MER) has emerged
as an additional technique allowing the surgeon to better locate the
target region for DBS intra-operatively [17]. MER measures the
electrical field of the brain during the surgery by inserting electrodes into the access path. The measured information is presented
to the surgical staff by showing the amplitude in the time domain
for each electrode, from which the expert can differentiate functional regions of the brain. In order to allow for an intuitive and
accurate placement of the electrodes, it is important that the surgeon has access to all these modalities in a unified manner.
Within this paper we propose an interactive visualization system
to support the surgeon during a DBS placement procedure. From
the surgeon’s point of view it is of great importance to obtain system support when identifying the optimal placement location for
the emitting electrode, such that higher accuracy and precision can
be achieved. Therefore the system should support the surgeon’s
mental registration between the different modalities and combine
the uncertainties in a quantitative way so that confidence in the displayed data is increased.We address these technical objectives, by
proposing a visualization system which provides two views enhancing the final placement of the electrode, by employing two separate
fusion techniques. In the first view, we combine the structural and
functional information about the target region gathered from different sources, such as pre-operative scans, x-ray scans, MER, patient
checks, and present them to the surgeon in order to guide them to
the optimal placement location. The view fuses this data with the
structural information surrounding the intended target region in a
multimodal visualization. This enables a mental registration of the
measured data with the imaging modalities and, thereby, provides
context that the surgeon can draw upon during the placement decision. In the second view we fuse the information about the occurring uncertainty and provide an information visualization approach
to present the data in a quantitative way. The original data and derived information is presented as profile plots allowing the surgeon
to see the optimal placement position at one glance and, thus, enable a more effective placement of the electrode.
In the remainder of this paper we will briefly explain the DBS
process, derive uncertainty regions for the measurements, and show
how to display them in a unified approach.
(a) Thalamus
(b) Zona incerta
(c) Nucleus subthalamicus
(d) Substantia nigra
Figure 1: The MER discharge pattern changes depending on the
functional region of the brain. The level of background activity and
single-cell activity varies when entering or leaving specific regions
and can be used to identify these region [3, 14].
2
R ELATED W ORK
An important step in the DBS intervention consists of the, usually
manual, trajectory planning. There has been a lot of work on (semi)automatic trajectory planning in recent years. Gemmar et al. determine the mid-sagittal plane and the localization of the anterior
commissures (AC) and the posterior commissures (PC) [9]. Their
method is based on a region-growing based segmentation algorithm
and requires a nonlinear anisotropic filtering kernel. The entry
point is varied while, for each trajectory, a cost function is evaluated and the system operator selects the best trajectory. Bruneberg
et al. facilitated a segmentation of the STN and determined the
optimal trajectory by avoiding ventricles, gyri, and blood vessels,
which need to be segmented beforehand [5]. Khlebnikov et al. suggested a system to find optimal access paths by interpreting tumors
as light sources and determining the cost-function based on light
transport [15]. A visualization system for the pre-operative planning was provided by Beyer et al., who proposed a multi-volume
renderer employing cut-away views to allow for visual access to
the brain structures of interest [4]. Furthermore, they included a
skull peeling algorithm, which we have also incorporated into our
system. Serra et al. created a neurosurgery planning tool for tumor resections, that uses a virtual workbench to increase user immersion and which provides additional 3D interaction tools [33].
Watanabe et al. developed a computer assisted surgery tool to treat
cortical lesions, and used a curvilinear reformatting to allow direct
access to the data [39]. More recently, Rieder et al. proposed a planning tool for neurosurgical tumor treatment, that uses a cylindrical
cut for a better view of the target region [28]. Additionally, they
introduce a distance ring to denote the relative depth of a specific
region of interest. Furthermore, the same authors introduce visualization techniques to enhance the perception of structures when
using multimodal rendering setups [29].
In the area of MER data integration Miocinovic et al. provide a
system for a guided DBS electrode implantation in non-human [23]
as well as in human primates [22]. They utilitize metaphors for visualizing region information but base the information on a 3D atlas
only instead of using the MER signal. They also do not employ
patient checks for feedback. In contrast, we provide a unified guidance system based on multi-modal measurements. Furthermore,
their main focus is MER-based region detection for generating an
atlas of the brain. D’Haese et al. presented a system for use in human surgery but focus on the comparison between MER selected
targets and atlas-based target selection [7]. They also do not incorporate the patient test information into their system, which is
required for a sufficient level of accuracy and confidence. Sperka
et al. also presented a system to improve the planning phase for
stereotaxic surgery [35]. Lastly, the use of fMRI data together with
other modalities was investigated in the context of stereotactic interventions on macaques [25].
Figure 3: Screenshots of our system during the three phases as described in Section 3.1. The planning phase (Section 3.2 in the top
left, recording phase (Section 3.3) on the right, and the placement
phase (Section 3.4) on the lower left.
3
S YSTEM OVERVIEW
The main goal of the designed system is to support the surgeon by
reducing cognitive load by fusing the available modalities and thus
facilitating mental registration [37]. The current situation in the
operating theater is that the methods for inspecting the structural
modalities, such as MRI, CT, and biplanar x-ray scans, are fairly
advanced and widely used. However, the systems for recording and
analyzing the MER signals and patient checks are decoupled from
the other modalities. This forces the operating team to perform
tasks serially, which prolongs the tiring procedure for the patient.
By integrating the temporal data of the recording in the spatial context of the scans, the surgeon is relieved of the burden of mental
registration and can perform the operation faster and with greater
confidence.
The second method through which we support the electrode
placement is to present all collected data in a unified way, and show
it in such a way that the optimal placement location is immediately
visible and the surgeon is guided towards it. The collected data
consists of information about the intended target region, the MER
recording, patient checks, and their associated uncertainties.
The traditional use of a planning system and an MER system side
by side allows for the mental coregistration of MER signals, but this
requires high levels of concentration fromt the surgeon. The situation is made worse because up to five parallel trajectories maybe
recorded at the same time. Correlation of the MER signal of interest
with the corresponding trajectory (for example anterior, posterior
lateral central or medial) becomes a confusing task for the surgeon.
The simple and intuitive display of a significant MER signal within
the space of the planned trajectory is therefore a new feature with
direct clinical benefit.
3.1 DBS Intervention Procedure
Before describing our system, we will first give a high-level
overview of the DBS intervention procedure as it is specified in
surgery guidelines [12]. With respect to the steps in this procedure
a DBS intervention can be divided into roughly three subsequent
phases: planning, in which the intended target region is selected,
recording, in which the MER signal is obtained, and placement,
in which the emitting electrode is inserted, the patient checks are
performed, and the placement is evaluated (see also Figure 2).
As accuracy is of uttermost importance when performing DBS
interventions, the patient is mounted into a stereotaxic frame that
is rigidly fixed to the patient’s skull. This frame has sockets that
hold and precisely guide the equipment used during the intervention. Due to the known geometry of the stereotaxic frame, it serves
as the basis for the coordinate transformation from imaging data to
the patient. An important property of the frame is that it is visible in
both CT and MRI scans, and can thereby serve as one of the land-
Figure 2: The workflow of the proposed DBS intervention system can be divided into three phases: planning, recording and placement. To
support each of these phases, we employ dedicated views that are arranged in a multiple view setup as shows in Figure 3. External components
interacting with our visualization system are shown for each phase.
marks used in the registration process. In the planning phase the
surgeon plans the access path based on the pre-operatively acquired
T1 - and T2 - weighted MRI scans. In most cases, however, the target region is not directly visible on either of the MRI scans and the
surgeon has to rely on experience and patient-neutral heuristics to
locate the target area. This step introduces some uncertainty, of an
unknown scale, into the position of the intended target region.
In addition to the MRI scans, a pre-operative CT scan, and orthogonal reference x-ray scans (see Figure 4) are acquired. Following the scans the access path is drilled, the MER electrodes
are inserted, and the recording is started. During the recording
phase the electrodes are moved forward towards the intended target region while they continuously record the discharge pattern of
the surrounding tissue. One member of the surgical team observes
the signal, analyzes it, and informs the surgeon about the findings.
We describe the developed visualization techniques supporting the
recording phase in Section 3.3.
After the recording the MER electrodes are removed, the emitting electrode is inserted, and, in the placement phase, the electrode
placement is performed. Upon reaching the target depth, the electrodes are activated and start emitting. Changing the position of the
electrode, the surgeon tests the patient’s higher brain functions in
order to narrow down the optimal electrode location. This is done
based on expert knowledge of the relative positions of functional
brain regions. To monitor and verify this process and the electrode’s actual position, orthogonal x-ray scans are obtained. We describe the visualization techniques developed to support the placement phase within Section 3.4. This part of the surgery is very tiring
for the conscious patient. The operation time varies between 6 to
10 hours and success may be limited by the ability of the patient to
cooperate. Thus, speeding up the testing phase using an integrated
planning/MER visualization tool is highly desireable.
3.2.1
3.2
3.3 Recording Phase
Planning Phase
The goal of the planning phase is to identify the desired target region based on pre-operative scans, and to plan a trajectory reaching
that target region while minimizing the distance to critical structures. There are a number of highly specialized tools available to
perform this task, and there is also active research in the field of automatic trajectory planning [34]. Since our focus is on other parts
of the procedure, we implemented a basic planning tool, capable
of importing the data from more sophisticated tools. The surgeon
selects the intended target location and the burr hole position on
a slice representation of the MRI scan. The resulting access path
between the two selected points is shown in a 3D multimodal visualization of the pre-operative scans (see Figure 5). As the 3D view
contains all contextual information relevant during each phase of
the DBS intervention procedure, it is used within all phases and
provides continuity through the phases of the system.
Contextual View
The contextual view is the common element that is present in all
three phases of our system. It is based on a multimodal visualization
of the pre-operatively acquired CT and MRI scans. To improve spatial comprehension, the three modalities are not fused completely
but vertically separated such that, for each layer, the modality of
highest interest is visible (see Figure 5). At the lowest layer we
render the T1 -weighted MRI with a transfer function that shows the
whole head of the patient. The spin-lattice time was chosen since
it allows for an easier classification of the skin and the relatively
stable gradients allow for adequate shading. This part communicates the orientation of the patient intuitively and thereby reduces
dangerous left-right mismatches by the surgical staff. The lower
layer is capped with a slice view that provides the surgeon with
direct access to structural information embedded within the spatial context. The skull is partly shown in the middle layer, based
on the CT scan, to serve as a smooth transition between the lower
layer and the brain. To obtain an occlusion-free view of the brain,
we employ a skull stripping approach, described in Section 4, at the
highest layer. As MRI has a fairly low signal-to-noise ratio [13], we
decided to employ depth darkening [19] instead of gradient-based
shading as it increases the depth perception of the brain’s structures.
The depth of the intended target region is visible as a yellow band
surrounding the skull and serves as another depth cue.
We present the access path to the surgeon by removing the structures around the previously determined trajectory [40, 28]. The
removed area is cylinder-shaped and possesses a variable, userdefined radius To increase the amount of information the surgeon
gains from the cylinder walls, the same transfer function as used
in the slice representations is applied here. More importantly, this
increased access path allows more information to be shown, and remain visible, in a spatial context. As described in the next section,
we can include glyphs within the removed area to provide information about attributes along the trajectory.
The recording phase is the next step during a DBS intervention.
While advancing the electrodes towards the target region, each detects the signals from the surrounding tissue and transmits them to
the controlling device. To obtain knowledge about the current region, the received signals are constantly analyzed and classified.
Methods exist to automatically detect the region. The analysis
and computation of these metrics, however, is not in the main focus of this paper so it is not elaborated on in detail. Instead, any
of the following variants can be chosen and a comparative study
might be the focus of future work. One possible set of metrics consist of the firing rate, burst index, pause ratio, and the pause index,
which were introduced by Haese et al. [7]. They also found out
that these metrics are useful to map the signals to specific brain
regions. These values are computed using a denoised signal and
applying a non-linear energy operator as presented by Maragos et
al. [20] and a spike detection with a subsequent thresholding presented by Mukhopadhyay et al. [24]. Several other approaches for
this technique exist and these make no systematical difference for
our system.
3.3.1
Additions to the Contextual View
The contextual view as employed in the recording phase is, in most
parts, identical to the one used in the planning phase. As this phase
of the intervention process requires additional information, however, the view differs in some ways. First, since the current depth
of the MER electrodes is known, we can compute and display a representation of the electrodes. As the extent of the electrodes is very
small compared to the whole head, and we are only interested in
the general position and orientation of the electrodes, just a single
representative proxy electrode is rendered.
Using the MER signal analysis, we can classify depth values by
their functional areas. Doing this continuously for every point along
the access path would introduce a lot of visual clutter that would not
give additional insight. Therefore we classify and summarize segments along the trajectory that we can classify up to an acceptable
certainty. For each classified segment we display a glyph, representation, which we refer to as a bead, in the center of the canal.
Each bead is rendered as a shaded sphere with a specific color. This
visual metaphor has been proposed before [23, 7] and provides contextual and functional information. The bead’s color correlates with
the classified region. A black bead is created if there has been a
significant distance without a reliable classification. These intermediate beads are important to maintain the analogy, i.e., the spatial relationship, of a bead string. Different shades of red are used
for areas that lie outside of the desired target area. It is important
to present this information to the surgeon as they can deduce more
information from the changes between different areas. The first
choice for a color would be different shades of gray, since it does
not draw as much attention. But since the canal wall is gray and
the human visual system is not well adapted to differentiate gray
tones, we chose red as the hue for these regions. As soon as the
analysis of the spike signals indicates that the MER electrodes are
in the target region, green beads are rendered that draw attention
to those structures. Furthermore, this color coding resembles the
traffic light color scheme proposed by Rieder et al. [30], which intuitively assigns green to good areas, while red is assigned to bad
areas. To further reduce visual clutter only one bead is rendered for
all electrodes. This is a viable simplification as the different functional regions of the brain are oriented such that either all electrodes
detect the same signal (either type of tissue or undefined) or a subset of electrodes detects a regional signal and the others detect an
undefined signal.
Even with the application of the depth darkening shading tech-
(a) Anterior-Posterior
(b) Dorsal-Ventral
Figure 4: Two x-ray images are created to calibrate the patient’s position in the operating room using the perspective distortion of the
reference plates. Subsequent x-ray images are used to verify the
electrodes position inside the head and estimate the uncertainty of
the segmentation.
Figure 5: In the recording phase the contextual view additionally
shows the intra-operatively acquired x-ray scans. It further renders
the electrode while it is being advanced towards the target. The
beads behind the electrodes represent the results of the MER signal analysis.
nique described by Luft et al. [19], the depth perception of the electrode is not optimal. One possible solution would be the introduction of a distance ring as described by Rieder et al. [28]. Instead, in
order to reduce occlusion within the focus of the view, we decided
to show a linear scale outside of the view’s focus, where it does not
block any relevant structures. The vertical white bar in this scale
gives immediate feedback about the electrode’s position. This widget is the first feature used within our system, which employs a
normalized view. This means that the left-most position in the view
corresponds to the entry point of the access path and the right-most
position is slightly further than the intended target region. As soon
as a functional area has been detected the background is colored
with the respective color. Although the beads convey a good absolute spatial orientation, their relative orientation is not immediately
obvious because of occlusion and perspective distortion. On the
other hand, the distance widget does not contain any information
about the absolute location. The combination of the beads in the
contextual view and the color in the distance widget provide the
relevant information to the surgeon.
Another addition to this view is the inclusion of the intraoperative x-ray images, which can be easily registered with the
patient using the external and internal camera matrices and the
known geometry of the two reference plates, which are shown in
Figure 4 [6, 41]. Using the complete camera matrix, it is possible
to select a point in both images and reconstruct, up to a certain accuracy, a 3D position within the volume from it [11]. The position
of the reconstructed electrode position can be shown on demand in
this view by rendering a second electrode in gray that is blurred
to accommodate for the uncertainty in reconstruction. The specific
uncertainty is dependent on, among other things, the geometry and
resolution of the x-ray detectors and is therefore different for each
operating theater. In our test case the accuracy in the position is of
the order of < 1mm. As these x-ray scans are used to guide the surgeon to the target region and provide one important means to verify
the final position of the electrode, the integration with the other
spatial modalities is important. In addition, we use the discrepancy
between the expected electrode position and the reconstructed position as an uncertainty for the target region’s segmentation in the
later phase.
3.3.2
2D Temporal Audio Visualization
In this view we present the raw data collected by the MER electrodes in real time. This signal is usually inspected by the surgeon
as an audio signal. One graph is shown for each electrode, which
shows the electric potential difference plotted over time. The scaling factor of the ordinate can be manually adjusted using a slider to
account for patient and equipment specific differences. The identifying names for each electrode are shown on the left hand side next
to the graphs.
Since surgeons are generally more interested in the distribution
of spikes than the background noise, we visually enhance spikes
such that they are the immediate focus of attention. This visual enhancement is done by defining a threshold value below which all
values are de-emphasized, a darker color being chosen, and above
which the values are emphasized with a brighter color. There is
an exponential transition in the area around the threshold to reduce
any unwanted attention that would otherwise result from a discontinuity. This threshold value can be varied by the user to include
a wider or narrower range of values in the focus area. The color
scheme matches the color on the electrode representations and at
the same time intensifies the perceptual distance between the colors
used for the thresholded area and the spikes.
To further facilitate linking between all views containing a representation of the MER signal, the surgeon can select electrodes
which are then highlighted in all views (see Figure 6). A problem
with current systems is that the surgeon must maintain a mental
model relating the horizontal oscilloscope graphs with the geometric orientation of the electrodes in the patient’s head. By creating a
linked view between those two representations, we reduce the mental burden on the surgeon without preventing him access to any of
the information.
3.3.3
3D Spatiotemporal Audio Visualization
It is important to bridge the gap between the MER signal, defined
over the time domain, and the structual data, contained in the imaging data. We achieve this by employing an audio visualization
that directly combines the spatial information of the electrodes with
their temporal electric signal. Each electrode shows its electric signal as concentric discs around the centerline. The discs start at the
back end of the electrode with a radius that scales linearly with the
absolute value of the amplitude. As soon as the next measurement
is registered the discs move away from the electrode and a new disc
is inserted at the back end. This way the signal originates from the
electrode and moves away from it over time. Considering the absolute value of the amplitude is sufficient as the spikes are distributed
symmetrically and surgeons are not interested in whether the spikes
occur because of a negative potential difference or a positive one.
The discs are distorted since they are drawn in perspective. This is
not a serious problem because, on the one hand the actual amplitude
Figure 7: Combining the spatial orientation and layout of the recording electrodes with the temporal signal relieves the surgeon of the
burden to keep this association in mind. To enhance the perception
of the spikes, only the values above a certain threshold are shown in
this view so that they are visible pre-attentively. The inset shows the
result for one electrode without thresholding.
of the signal is not as important as the spike frequency and, on the
other, the 2D signal is presented to the surgeon as a backup as well.
Creating a disc for every measurement point would clutter the
whole view (see inset in Figure 7). Therefore we apply the same
thresholding technique as described in the previous section, but discard all signals below the threshold completely.
3.4 Placement Phase
In the placement phase the surgeon needs to consider all acquired
information to determine the optimal position of the stimulating
electrode. This decision is based on three different factors that are
all affected by varying uncertainties:
First, the surgeon’s experience and knowledge in selecting and
segmenting the intended target region during the planning phase.
The segmentation is uncertain as, for example, the brain shifts from
its scanned position during the operation. There exist techniques to
reduce this uncertainty and we estimate it by measuring the difference between the expected electrode position and the reconstructed
position based on the x-ray images. Second, the results of the MER
recording and the area along the trajectory. The uncertainty for this
step is highly dependent on both the algorithm used to analyze the
MER signal and the electrode configuration. The third factor is
acquired when the stimulating electrode is inserted and activated.
The surgical staff will test different brain functions of the patient.
Since it is well-known which areas affect which capabilities while
stimulated, the surgeon can further narrow down the location of the
optimal target region. In our example we use two brain functions,
the ability to recall events from long-term memory and an increase
in tremor of the patient’s hand. Each of the functions is tested separately along the access path and, based on the surgeon’s experience
and a patient-neutral heuristic, it is possible to deduce the target region’s position relative to the measurement point. The level of uncertainty for this information is relatively constant and in the same
range as the extent of the electrodes,< 1.5mm for our setup. For
each decision factor we independently measure the probability, and
uncertainty, of the target region’s position along the access path.
The contextual view introduced above is shown to maintain the
surgeon’s mental registration. Since the whole trajectory has been
measured in the previous phase, however, all beads and all information previously acquired in the canal tube is presented independently of the electrode position.
3.4.1
Figure 6: The MER signal in the time domain is shown using an
oscillograph-like representation in which time is on the horizontal axis
and the electric potential difference on the vertical. The visual perception of the signal is enhanced by de-emphasizing the background
noise and guiding the attention to the more important spikes. The
signal of the currently selected electrodes is highlighted.
Target Closeup
The localization procedure must not only communicate the electrode’s position with respect to anatomical structures, but also with
regard to the intended target region. An optimal placement of the
stimulation electrode should be in the intersection of the potential
target regions obtained from the different measurements. We support this in-detail navigation aspect by providing a 3D visualization,
Figure 8: The target closeup visualization shows the potential target regions (speech tests=blue, movement tests=purple). The spatial context provided by the MRI signal is color coded using the redgreen MER region mapping. The stimulation electrode changes its
color when it is entering the intersection of the potential target area.
This view gives qualitative access to all uncertainty data fused with
structural data in one glance.
which we refer to as the target closeup, that shows the electrode embedded in the potential target regions together with the intended target region. To relate these potential target regions to the electrode
and embed them into the spatial context, we additionally display
the pre-operative MRI scans as a view-dependent projection on the
back faces of the target closeup’s bounding box. The rendering of
the datasets has been made optional as, while it can provide valuable information regarding the intersection between the electrode
and structures of interest, it can also occlude the potential target
regions. The same orientation overlay, which is also used in the
other 3D representations, is used in the electrode closeup to allow a
seamless embedding of this view.
We have to deal with three different types of potential target regions, where the type of a region depends on the way it has been
acquired. First, the planned region, which has been manually defined through segmentation by the surgeon on the basis of the preoperative MRI scans. Therefore, we can apply volume rendering to
show the result of this planning process. The second type is derived
from the MER signal and has considerable uncertainty. This is projected onto the back planes of the detail view with the color coding
that we have also used for visualizing the original MER signal. A
safety margin is displayed to account for the occurring uncertainty
sources. The third type of potential target region is derived from the
patient tests performed when placing the stimulating electrode and
must, therefore, be interactively changeable. The region is shown as
an ellipsoid in 3D assembled by connecting several check points. In
the example, shown in Figure 8, we depict the ellipsoid generated
from multiple speech tests in yellow, while the ellipsoid obtained
from the movement tests is depicted in blue. The safety margins
are depicted by using transparency, as it allows them to be communicated intuitively while still maintaining a moderate level of occlusion. The width of the safety margins is dependent on experience
and the electrode configuration that is used in the intervention.
Similarly to the contextual view, we make it possible for the surgeon to show the approximate location of the electrode as reconstructed from the x-ray scans. This is of particular interest in this
view, as it shows the deviation of the actual position from the predicted electrode position, due to brain shift and bending of the needle. The uncertainty of the reconstruction is shown by blurring the
rendering of the electrode to fill the size of the uncertain region.
Finally, we emphasize the intersection of the displayed target regions as this intersection is the region where an optimal electrode
placement would occur. We compute this intersection interactively
Figure 9: The placement guide gives a quantitative overview of measured data for potential target regions. The top figure is a combination of all data values, where as the lower figures present detailed
information by showing either all values or pairwise combinations.
These combinations enable the surgeon to gain further insight into
the measurements in certain situations, for example, when a measurement proves unreliable during the procedure.
and display it volumetrically with a green hue, the same as is used
for the green-to-red MER mapping. Thus, the surgeon can verify
the overlap and decide in which region to place the electrode. Furthermore, through the extent of the red region, the surgeon gets information about the mismatch between the pre-operative planning
and the MER recordings. To further guide the surgeon during the
electrode placement, we additionally change the color of the electrode when it penetrates the computed interaction of the potential
target regions.
3.4.2 Placement Guide
In addition to the spatial context, as described in the previous section, we also present the potential target regions quantitatively in
a view we call the placement guide. This view is centered on the
intended target region with the access path depth on the abscissa
and the likelihood of the actual target region on the ordinate. By
showing the quantitative data in a line plot, the surgeon immediately sees in which areas most of the measurements agree and is
guided towards that position. As we have multiple decision values
for each depth value, we can combine them in different ways. In
our example we chose a weighted sum, but the combination function can be freely changed according to future research. The values for uncertainty are combined in the same way and are rendered
as transparent areas surrounding the line. Transparency has been
shown to be a good way to convey uncertainty in other contexts [8].
With this representation the size of the transparent area immediately
shows how uncertain a specific value is. This lets the surgeon see
the whole set of information at one time and decide on an optimal
placement position. All combined views use spline-based smoothing on the curves, which can be disabled by the user. This is done so
that the surgeon is not distracted by abrupt changes in the data values which would otherwise draw unwanted attention . The current
position of the electrode is shown as a bar in each of the views.
The top figure in Figure 9 shows the combined data of all values
and is of central interest for the surgeon. It provides an overview of
all measured data and guides the surgeon to the most likely, the
highest, position. The curve is assembled while the stimulating
electrode is being advanced and the patient checks are performed.
Since the combined curve hides possible outliers and gives no feedback on the underlying values, the composition of each sample
point can be examined in the top right as a bar plot that shows the
probability and uncertainty for each decision factor separately. All
bars are justified to the left, with their width directly relating to the
measured value. As with the line plot, a transparent box shows the
area of uncertain values. One bar is rendered for each source of
measurements. So in our case (see Figure 9) there is one bar for
the segmentation, MER, and one for each of the two patient checks.
Including a line alongside the combined data for each of the decision factors would clutter the view and would distract the surgeon
too much, which is why we chose to add the bar plot representation.
There is,however, an optional view showing those decision factors
separately. Furthermore, it is possible to view all pairwise combinations of factors. Showing these auxiliary views, the surgeon
can detect outliers and unexpected behavior immediately without
distracting him from the main view. Toggling of these auxiliary
views can become important in an operation when the surgeon, for
example, notices that the patient checks of this particular patient
are not as reliable as expected or detects a flaw in the segmentation. Furthermore, it can be useful to inspect the data as if one of
the measurements is not considered. This inspection might lead to
finding that one of the measurements is not reliable.
4
I MPLEMENTATION
In this section we describe selected implementation details of our
system.
Multivolume raycasting. For the contextual view we need to render registered multimodal datasets and at the same time integrate
the electrodes and beads into the rendering. For the rendering we
exploit GPU-based volume raycasting as presented by Krüger and
Westermann [16]. We include the geometric information of objects
into this raycasting scheme by modifying the exit points such that
the raycasting process ends at those objects [32]. The objects are
rendered in a separate pass and the results are blended to obtain
the final rendering result. To achieve interactive multivolume raycasting we employ a modified version of the region based scene
description as presented by Lindholm et al. [18].
Skull stripping. We implemented the skull stripping algorithm as
presented by Beyer et al. [4]. Although a complete segmentation
would provide better results, we want to avoid the necessary user
interaction. The basis for this method is opacity peeling [26] which
is, in our case, performed on the conditional ray-casting results.
Depending on a user-selected parameter, the accumulated values
are reset as soon as the early ray-termination criterion is reached.
The skull stripping employed in our system uses registered CT and
MRI scans to determine the boundary between the brain and the
outer layers. Thus, the values accumulated along a ray can be reset when the rays leaves the bone structures and the ray traversal
through the brain starts. This method requires no user interaction
and provides good and stable results.
5
E VALUATION
As computer-aided surgery systems are combining an increasing
number of types of information from different sources, it is important that the interfaces reduce the cognitive load required to use the
system [38, 21]. To evaluate the practicability of the proposed system we have conducted a qualitative user-study with five neurosurgeons, all of whom have extensive experience in the field and perform DBS interventions on a regular basis. The study has been designed with respect to the guidelines for evaluating computer-aided
surgery systems, which have been proposed by Martelli et al. [21].
The neurosurgeons were shown a showcase demonstration of the
system going through all three phases (planning, recording, and
placement). A demonstration was chosen, as a pre-study showed
that the complexity of usage, in the current form, would result in
a biased evaluation. The data for this demonstration was recorded
and reconstructed from a real intervention. After the demonstration the neurosurgeons answered a questionnaire consisting of eight
statements and questions, as well as a text field for arbitrary comments. Each of these statements and questions had a positive and
a negative reply with no opportunity to abstain. The following table shows all statements and questions together with the number of
positive answers:
The user is not disturbed by images, colors, or animation 4
while interacting with the system
The result of any action done by the user is clearly and 4
immediately visualized
The tools provided by the system are easy to use
3
System data is understandable and clearly visualized
4
The actions’ succession proposed by the system is logical 5
from the user’s point of view
The system’s features are compatible with all the user’s ex- 4
pectations
Would you like to use this system during an intervention in 5
addition to the old system?
Is the fusion of MER signals with image data useful for 3
increasing the accuracy?
Discussion. Overall, the feedback from the neurosurgeons was very
positive. The least satisfied expert agreed with only five statements
whereas the most satisfied expert agreed to all. On average the
experts agreed to 6.5 statements if we assume an equal weighting
of the statements. The relatively bad score of only three positive
answers to the last question seems to be the strongest drawback to
the method, but considering that all of the experts would use our
system in the operation room makes it a good result. One expert
emphasized especially the ”correlations between target region and
the neurophysiological data” as an important aspect of the system.
Only two neurosurgeons did not see the benefits of incorporating
the fused views, while they still liked the overlapping view of the
different probable target regions. The third question confirmed the
expected usability result from the pre-study.
6
C ONCLUSIONS
AND
F UTURE W ORK
In this paper we have presented a visualization system that has
been designed to support neurosurgeons during DBS interventions
by fusing measurement data, along with their uncertainties, with
structural modalities to facilitate mental registration. This multimodal information is presented to the surgeon during the intervention both in the context of the rendered imaging data, as well as
a separate view showing the combination of these measurements.
As the MER and patient checks are performed during DBS interventions to improve the localization of the stimulation electrode,
we feed back the results of these checks into our system. When
displaying and intersecting the potential target regions we take into
account the different degrees of uncertainty that result from their
acquisition process. This uncertainty-aware information fusion in
image space, as well as in a profile plot, enables the surgeon to better assess the electrode placement and detect the optimal electrode
placement. To estimate the clinical impact of the presented system
we performed a qualitative evaluation with five neurosurgeons who
regularly perform DBS interventions. The results indicate that the
presented visualization approaches are of great interest and have the
potential to improve DBS interventions.
In the future we would like to further improve the presented system based on the feedback we have received from the surgeons during our evaluation. We would especially like to reduce the learning
curve currently necessary. Furthermore, it would be beneficial to
evaluate its effectiveness in the operation theater. However, a full
evaluation requires a lot of effort as it would require the system to
be certified for usage in the operation theater. While the current system considers most modalities, the integration of DTI could also be
considered in the future. Another source of information we would
like to include is the region of the brain that is affected by the stimulation signal, the volume of tissue activated. This would simulate
the region in the brain that would be influenced by the current electrode position and could thereby function as a prediction as well as
a verification tool. While we have currently focused on the recording and placement phases of the DBS intervention, we would also
like to extend the system for long term horizontal studies of electrode placements. A major part of this horizontal study would be
the ability to compare the measured accuracy with the outcome of
the intervention. As no extensive statistics about the long term effects of varying placement positions and the applied electric fields
exist, this could be valuable information for further improving DBS
interventions in the future.
ACKNOWLEDGEMENTS
We thank all reviewers for their fruitful comments and Matt Cooper
for proof reading. This work was partly supported by grants
from the Excellence Center at Linkping and Lund in Information
Technology (ELLIIT) and the Swedish e-Science Research Centre
(SeRC), as well as VR grant 2011-4113. The presented concepts
have been realized using the Voreen framework (www.voreen.org).
R EFERENCES
[1] M. Åström, E. Tripoliti, M. I. Hariz, et al. Patient-Specific ModelBased Investigation of Speech Intelligibility and Movement during
Deep Brain Stimulation. Stereotactic and functional neurosurgery,
88(4):224–233, 2010.
[2] A. L. Benabid, S. Chabardes, J. Mitrofanis, and P. Pollak. Deep brain
stimulation of the subthalamic nucleus for the treatment of Parkinson’s
disease. Lancet neurology, 8(1):67–81, 2009.
[3] A. Benazzouz, S. Breit, A. Koudsie, and P. Pollak. Intraoperative
Microrecordings of the Subthalamic Nucleus in Parkinson ’ s Disease.
Movement Disorders, 17(3):145–149, 2002.
[4] J. Beyer, M. Hadwiger, S. Wolfsberger, and K. Bühler. High-quality
multimodal volume rendering for preoperative planning of neurosurgical interventions. IEEE TVCG, 13(6):1696–703, 2007.
[5] E. J. L. Brunenberg, A. Vilanova, V. Visser-Vandewalle, et al. Automatic Trajectory Planning for Deep Brain Stimulation : A Feasibility
Study. MICCAI, 10(1):584–592, 2007.
[6] B. Caprile and V. Torre. Using vanishing points for camera calibration.
Journal of Computer Vision, 4(2):127–139, 1990.
[7] P. D’Haese, E. Cetinkaya, P. Konrad, et al. Computer-aided placement
of deep brain stimulators: from planning to intraoperative guidance.
IEEE Transactions on Medical Imaging, 24(11):1469–1478, 2005.
[8] S. Djurcilov, K. Kim, P. Lermusiaux, and A. Pang. Visualizing scalar
volumetric data with uncertainty. Computers and Graphics, 2002.
[9] P. Gemmar, O. Gronz, K. Fisch, et al. Automated Target Location And
Trajectory Selection For Stereotactic Planning In Deep Brain Stimulation. Biosignal, 2008.
[10] G.-M. Hariz, M. Lindberg, and A. T. Bergenheim. Impact of thalamic
deep brain stimulation on disability and health-related quality of life
in patients with essential tremor. Journal of Neurology, Neurosurgery
and Psychiatry, 72:47–52, 2002.
[11] R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer
Vision. Cambridge University Press, second edition, 2004.
[12] S. Hemm and K. Wårdell. Stereotactic implantation of deep brain
stimulation electrodes: a review of technical systems, methods and
emerging tools. Medical & biological engineering & computing,
48(7):611–24, 2010.
[13] K. Herrmann, S. Schmidt, A. Kretz, R. Haenold, I. Krumbein, M. Metzler, C. Gaser, O. Witte, and J. Reichenbach. Possibilities and limitations for high resolution small animal mri on a clinical whole-body
3t scanner. Magnetic Resonance Materials in Physics, Biology and
Medicine, 25(3):233–244, 2012.
[14] W. D. Hutchison, R. J. Allan, H. Opitz, et al. Neurophysiological
Identification of the Subchaiarnic Xucleus in Surgery for Parkinson ’
s Disease. Annals of Neurology, pages 622–628, 1998.
[15] R. Khlebnikov, B. Kainz, J. Muehl, and D. Schmalstieg. Crepuscular
rays for tumor accessibility planning. IEEE TVCG, 2011.
[16] J. Krüger and R. Westermann. Acceleration Techniques for GPUbased Volume Rendering. In IEEE Visualization, 2003.
[17] F. Lenz, J. Dostrvsky, H. Kwan, et al. Methods for microstimulation and recording of single neurons and evoked potentials in the human central nervous system. Journal of neurosurgery, 68(4):630–634,
1988.
[18] S. Lindholm, P. Ljung, M. Hadwiger, and A. Ynnerman. Fused MultiVolume DVR using Binary Space Partitioning. Computer Graphics
Forum, 28(3):847–854, 2009.
[19] T. Luft, C. Colditz, and O. Deussen. Image enhancement by unsharp
masking the depth buffer. ACM Transactions on Graphics, 25:1206–
1213, 2006.
[20] P. Maragos, J. F. Kaiser, and T. F. Quatieri. On Amplitude and Frequency Demodulation Using Energy Operators. IEEE Transactions
on Signal Processing, 41(4):1532–1550, 1993.
[21] S. Martelli, L. Nofrini, P. Vendruscolo, and A. Visani. Criteria of
interface evaluation for computer assisted surgery systems. Journal of
Medical Informatics, 72(1-3):35–45, 2003.
[22] S. Miocinovic, A. M. Noecker, C. B. Maks, C. R. Butson, and C. C.
McIntyre. Cicerone: stereotactic neurophysiological recording and
deep brain stimulation electrode placement software system. Acta
Neurochir Suppl, 2007.
[23] S. Miocinovic, J. Zhang, W. Xu, et al. Stereotactic neurosurgical planning, recording, and visualization for deep brain stimulation in nonhuman primates. Journal of neuroscience methods, 162(1-2):32–41,
2007.
[24] S. Mukhopadhyay and G. C. Ray. A new interpretation of nonlinear energy operator and its efficacy in spike detection. IEEE TBME,
45(2):180–7, 1998.
[25] S. Ohayon and D. Y. Tsao. MR-guided stereotactic navigation. Journal of neuroscience methods, 204(2):389–97, 2012.
[26] C. Rezk-Salama and A. Kolb. Opacity peeling for direct volume rendering. Computer Graphics Forum, 25(3), 2006.
[27] E. Richter, T. Hogue, W. Halliday, et al. Determining the position and
size of the subthalamic nucleus based on magnetic resonance imaging results in patients with advanced Parkinson disease. Journal of
neurosurgery, 100(3):541–546, 2004.
[28] C. Rieder, F. Ritter, M. Raspe, and H.-O. Peitgen. Interactive Visualization of Multimodal Volume Data for Neurosurgical Tumor Treatment. Computer Graphics Forum, 27(3), 2008.
[29] C. Rieder, M. Schwier, H. K. Hahn, and H.-O. Peitgen. High-Quality
Multimodal Volume Visualization of Intracerebral Pathological Tissue. Visual Computing for Biomedicine, 2008.
[30] C. Rieder, A. Weihusen, C. Schumann, et al. Visual Support for Interactive Post-Interventional Assessment of Radiofrequency Ablation
Therapy. Computer Graphics Forum, 29(3), 2010.
[31] M. C. Rodriguez-Oroz, J. A. Obeso, A. E. Lang, et al. Bilateral deep
brain stimulation in Parkinson’s disease: a multicentre study with 4
years follow-up. Brain, 128(Pt 10):2240–9, 2005.
[32] H. Scharsach. Advanced GPU Raycasting. In Central European Seminar on Computer Graphics, 2005.
[33] L. Serra, R. A. Kockro, C. G. Guan, et al. Multimodal Volume-based
Tumor Neurosurgery Planning in the Virtual Workbench. MICCAI,
1998.
[34] R. R. Shamir, I. Tamir, E. Dabool, et al. A method for planning safe
trajectories in image-guided keyhole neurosurgery. MICCAI, 13(Pt
3):457–64, 2010.
[35] D. J. Sperka and J. Ditterich. Splash: a software tool for stereotactic
planning of recording chamber placement and electrode trajectories.
Frontiers in neuroinformatics, 5:1, 2011.
[36] P. Starr, C. Christine, P. Theodosopoulos, et al. Implantation of deep
brain stimulators into the subthalamic nucleus: technical approach and
magnetic resonance imaging-verified lead locations. Journal of neurosurgery, 97(2):370–87, 2002.
[37] M. Tory. Mental registration of 2d and 3d visualizations (an empirical
study). IEEE Visualization, 2003.
[38] H. Visarius, J. Gong, C. Scheer, et al. Man-Machine Interfaces in
Computer Assisted Surgery. Computer Aided Surgery, 2(2):102–107,
1997.
[39] L. D. S. Watanabe. Development of a Visualization and Manipulation
Toolkit for 3D Medical Image.
[40] D. Weiskopf, K. Engel, and T. Ertl. Volume Clipping via Per-Fragment
Operations in Texture-Based Volume Visualization. In IEEE Visualization, volume Cli, 2002.
[41] Y. Zheng and Y. Liu. Camera calibration using one perspective view
of two arbitrary coplanar circles. Optical Engineering, 47(6), 2008.