Difference between revisions of "X4D Overview"

From Visual3D Wiki Documentation
Jump to: navigation, search
(39 intermediate revisions by the same user not shown)
Line 11: Line 11:
 
[[File:X4D_Window.png|800px]]
 
[[File:X4D_Window.png|800px]]
  
=Bone Tracking=
+
==Tutorials==
 +
 
 +
[[X4D_HowTo_Track_Bones|How To: Get Started Tracking Bones]]
 +
 
 +
[[X4D_HowTo_Track_Multiple_Bones|How To: Track Multiple Bones]]
 +
 
 +
[[X4D_HowTo_Optimize_Xray_DRR_Settings|How To: Optimize Xray/DRR Settings]]
 +
 
 +
[[X4D_HowTo_Calculate_Bone_Poses_From_Mocap_Data|How To: Calculate Bone Poses from Mocap Data]]
 +
 
 +
==Bone Tracking==
  
 
{| class="mw-collapsible mw-collapsed wikitable" id="4D_Optimization" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" id="4D_Optimization" width="80%"
Line 53: Line 63:
 
|}
 
|}
  
{| class="mw-collapsible mw-collapsed wikitable" id="HowTo_Get_Started_Tracking_Bones" width="80%"
+
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
! style="text-align:left;" |      How To: Get Started Tracking Bones
+
! style="text-align:left;" |      Objective Function for Optimization
 
|-
 
|-
 
|
 
|
:It is recommended that you track bones one at a time in xray images, unless two or more bones overlap in the images and accounting for this overlap improves tracking. While tracking with optimization, the optimizer randomly positions each bone during each iteration and then evaluates the fit of all of them together. If one bone was moved to a slightly better pose while the others were moved to worse ones (a common occurrence), the overall fit would be worse and the poses of all bones would be discarded. The recommended procedure is to track the largest/easiest bone first, with all others turned off. Then track an adjacent bone, with all others turned off. If the tracking results with the second bone are not good and you suspect it is because of overlap with the first bone, then try tracking both together, as explained in
+
[[File:DSX_ObjectiveFunction.jpg]]
<a href="#C4">Jump to Chapter 4</a>. However, most of the time it is sufficient to track each bone independently of all others. Here is the general procedure for doing so:
 
#Load the subject into X4D and select the session, configuration, and trial.
 
#Scroll through the xray images and identify the range of reporting times over which the bone is trackable.
 
#Check the <i>Show Processed Xray</i> checkbox above each xray image to view the processed xray images. The processing that is performed on the Xray and DRR images is explained in <i>Matching Xray and DRR Images</i>.
 
#Experiment with the xray processing parameters in the <i>Xray/DRR Settings</i> widget. You generally want strong bone edges, to facilitate tracking, but you don't want to "wash out" the finer edges that can help fine tune bone poses. The settings that are particularly important for tracking are the Xray <i>Edge/Intensity Merge</i> value and the lower value of the Xray <i>Edge Capping</i>.
 
#If you have motion capture data, use the guide <i>How To: Calculate Bone Poses from Mocap Data</i> to calculate initial poses for all of the bones. Then skip to step 8.
 
#Manually define poses for the bone every 5 - 10 reporting times through the trackable range. This will keep the pose map splines from extrapolating to unrealistic values, and provide reasonable initial poses for the entire trackable range. You do not need to spend a lot of time carefully adjusting the bone pose at each reporting time; this is just a first-pass manual initialization to replace mocap seeding. Putting the bone within about 10 mm and 10 degrees from its correct pose is sufficient. To do this:
 
##Select the bone in the <i>DSX Configuration</i> widget. Turn on the display of the processed DRR image in each Xray window using the <i>Show DRR</i> and <i>Show Processed DRR</i> checkboxes above each window. If the DRR is not visible in one or both windows, it probably has an initial pose that puts it outside the window. Zooming out in the Xray window with the mouse wheel should enable you to see the DRR.
 
##In the Xray windows, there are two ways to move the bone, and they are both described in the section <i>Xray Windows</i>.
 
##You can also manually pose the bone in the 3D view. Select <i>3D View</i> from the <i>View</i> menu to open the window with the 3D view of the xray lab. With the checkboxes above that window you can turn on the display of bone and the xray images. The commands for moving the bone are described in the section <i>3D View</i> below.
 
##When you are done moving a bone for the current reporting time, be sure to save the pose to the pose map by selecting <i>Add Current Poses to Maps</i> in the <i>Pose Maps</i> widget. If you do not, any changes you made to the pose will be discarded when you change the reporting time.
 
#To see graphs of these initial bone poses, open the <i>Graphs</i> widget from the <i>View</i> menu. Select the bone from the pull-down menu in the upper left of the window to see graphs of all 6 DOFs for this bone.
 
#Save the initial bone poses to a file by selecting <i>Save Pose Maps</i> from the <i>File</i> menu. You will be prompted to browse for a file to create. Then save the subject file with <i>File -> Save Subject</i>.
 
#Go to one of the reporting times where it looks like tracking will be easiest. Manually move the bone so that it's within 2-3 mm and 2-3 degrees of the correct pose.
 
#Open the <i>Object Tracking</i> widget from the <i>View</i> menu.
 
#Set the 6 DOF ranges to 3-4 mm and 3-4 degrees. Set <i>Max iterations</i> to 2000.
 
#Select <i>Solve Current Frame</i>. Turn off the display of the DRRs to make the optimization run faster.
 
#Turn the DRRs back on and check the result. You may need to run the optimization longer or manually move the bone closer to the correct pose. The goal here is to determine a good tradeoff between taking the time to manually adjust the bone to get it close, and taking the time to run a longer optimization. At one extreme you can spend 1-2 minutes manually adjusting the bone to get it as close as possible (at each reporting time), then set the max iterations low (500) and DOF ranges tight (1-2 mm/deg), then run a short optimization. At the other extreme you can rely on mocap data or very quick manual adjustment to get the bone within about 10 mm and 10 degrees, then set the DOF ranges to 15 mm and 15 degrees and run a 5000 - 10000 iteration optimization.
 
#If you find that the optimization does not lock into the correct pose even when you start it close, this could be because the DRR image and xray image are not a good match to each other. This is usually because the intensities and thicknesses of the edges in the DRR image do not match those in the xray image. This can often (though not always) be fixed by adjusting the Xray and DRR image-processing settings in the <i>Xray/DRR Settings</i> widget. If some quick trial and error does not resolve the problem, you can try the guide <i>How To: Optimize Xray/DRR Settings</i>.
 
#<i>File -> Save Pose Maps</i>.
 
#Go to the next or previous reporting time. The initial pose for this time should be pretty close to correct because it's calculated by interpolating the pose map splines. Try <i>Solve Current Frame</i> and check the results. If they are not optimal, try adjusting the tracking parameters and/or manually adjusting the bone pose.
 
#Proceed to solve 2 or 3 more reporting times one at a time. If the results are good and no adjustments are needed, set the <i>Tracking</i> range to a small range of adjacent reporting times and solve them continually with <i>Solve Range</i>.
 
#<i>File -> Save Pose Maps</i>.
 
#Once you have solved a small range of reporting times you can try a larger range (or possibly even the entire trackable range).
 
#<i>File -> Save Pose Maps</i>.
 
#Once the trackable range is solved, you can try fine-tuning and smoothing the results with 4D solving, which is described in the section <i>4D Optimization</i>. Use the command <i>Solve Range 4D</i> to try it.
 
 
 
 
|}
 
|}
  
{| class="mw-collapsible mw-collapsed wikitable" id="HowTo_Track_Multiple_Bones" width="80%"
+
==Motion Capture Data==
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;How To: Track Multiple Bones
+
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 +
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Working with Motion Capture Data
 
|-
 
|-
 
|
 
|
 +
If you have surface-marker-based motion capture data for an x-ray trial, you can use it to calculate initial poses for the bones you want to track in the trial. To use this data in X4D, you must have C3D files with labelled marker trajectories for the motion trial and its associated static trial, and Visual3D files defining a kinematic model and its motion constraints. In the DSX software, this model is referred to as a ''mocap model''. Creating the model template and the constraint scripts requires a good working knowledge of Visual3D. The ''model template'' file (MDH) defines the segments and landmarks in the kinematic model, based on a specific marker set. The ''build script'' (V3S) creates the constraints that define the relative motions of the segments in the model. The ''recalc script'' (V3S) updates the segment poses to satisfy the constraints. Once the model is defined, and applied to a particular DSX trial, it can be used in three modes:
  
<h2 id="C4">Chapter 4</h2>
+
* '''Marker Mode''' - The motions of the segments are driven by the surface markers, with constraints enforced. This mode is used to calculate mocap poses in X4D.
  
:Unless the bones overlap in the X-ray images, it is better to track bones one at a time. This is because for each iteration the optimizer randomly positions each bone and then evaluates the fit of all of them together. If one bone was moved to a slightly better pose while the others were moved to worse ones (a common occurrence), the overall fit would be worse and the poses of all bones would be discarded.  
+
* '''Interactive Mode''' - You can interactively change all of the model's degrees of freedom (the 6 degrees of freedom of the root segment, and the joint angles). Constraints are enforced. This mode is used when you manipulate the bone poses in X4D.
  
:Even if there is overlap of bones, it is often preferable to track them separately. In these cases, a good strategy is to track one bone first, then to lock it in place and track a second bone with the first one still active.
+
* '''DSX Mode''' - The motions of the segments are driven by DSX tracking results (transform matrices). Constraints are not enforced. This mode is used when DSX results are viewed and analyzed in Visual3D.
 
 
:For example, when tracking the bones at the knee there is often substantial overlap between the femur and the patella. The recommended strategy is as follows:
 
#track the femur by itself in all frames;
 
#select the femur and patella;
 
#uncheck the femur box in the Object Tracking widget so that the femur remains fixed in its previously calculated poses; and
 
#optimize to track the patella.
 
 
 
:The femur’s DRR will be included in the optimization, but the femur pose will not change in each frame.
 
|}
 
 
 
{| class="mw-collapsible mw-collapsed wikitable" id="Optimize_Xray_DRR_Settings" width="80%"
 
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;How To: Optimize Xray/DRR Settings
 
|-
 
|
 
To make it easier to find the best set of parameter values for a particular trial, the image optimization widget performs a simulated annealing optimization on the 15 parameters. To use it, follow these steps:
 
 
 
#Choose an x-ray frame in which the bone you're tracking is fairly easy to see. It is recommended that you select a single bone to optimize the image settings.
 
#Manually move the bone into the correct pose as best you can.
 
#In the <i>Object Tracking</i> widget, perform a quick optimization (about 500 iterations with ranges of 1 mm and 1 deg) to fine-tune the pose. If this optimization makes the pose worse, go back to the manually adjusted pose.
 
#Set the tracking optimization ranges to larger values (e.g., 3 mm and 3 deg) and keep the number of iterations at about 500.
 
#In the Image Optimization widget, set the max iterations value. A value between 200 and 500 is recommended. This is the number of tracking optimizations that will be performed on the current bone in the current frame. Each of these optimizations will use a different set of image processing parameters and will run for the number of iterations specified in the <i>Object Tracking</i> widget.
 
#Set the ranges of the 15 image processing parameters to values appropriate for the current frame. You may want to experiment with their values in the <i>Xray/DRR Settings</i> widget to get a sense of how they affect the images. For parameters that have a large effect on the images (e.g., X-ray Min Edge), set their ranges smaller so that the optimizer focuses more on the optimal region.  
 
#Press <i>Optimize</i> to start the optimization. For each iteration, a simulated annealing algorithm will randomly choose values for the 15 parameters within the ranges you specified, then perform a tracking optimization starting from the bone pose you specified at the start.
 
#When the optimization is finished, the image processing parameters will be set to their optimal values. The optimal values are the values that caused the tracking optimization to lock-in on a pose closest to the initial pose. You can then try tracking the bone in other frames to see if the parameter values work on the rest of the trial.
 
 
 
'''Note''': Turning off the display of the DRRs and CT bones will increase the speed of the optimization.
 
 
 
'''Note''': For any given x-ray frame and bone, there is no guarantee that there exists a set of image processing parameters such that the correct bone pose is the global minimum bone pose. This is because of the inherent differences between the x-ray and DRR images and the quality and extent of the CT data. There is also no guarantee that the optimal parameters for one frame will work on any other frame. However, image optimization is still often useful for finding a neighborhood of parameter values that improve the performance of the tracking optimization.
 
|}
 
  
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
+
In addition to generating initial poses for the tracking objects, you can use the mocap model to help with manual positioning of the objects. In the [[X4D_Overview#Mocap_Model_Widget|Mocap Model widget]] in X4D, you can turn on the model's joint constraints, making it easier to move all of the objects together as a group, or to move them relative to each other in a realistic manner (e.g., flexing or twisting the vertebrae in a spine model).
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Objective Function for Optimization
 
|-
 
|
 
[[File:DSX_ObjectiveFunction.jpg]]
 
 
|}
 
|}
  
=Menus=
+
==Menus==
  
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
Line 302: Line 254:
 
|
 
|
 
:The <i>Mocap Model</i> control toggles the appearance of the dockable <i>Mocap Model</i> widget. It can also be controlled using the Ctrl + Shift + M shortcut.  
 
:The <i>Mocap Model</i> control toggles the appearance of the dockable <i>Mocap Model</i> widget. It can also be controlled using the Ctrl + Shift + M shortcut.  
 +
|}
 +
|}
 +
 +
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 +
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Options Menu
 +
|-
 +
|
 +
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 +
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[Image:Select_Gpus_Menu.png]]
 +
|-
 +
|
 +
:When more than one cuda-capable graphic cards are installed in the computer, the ''Select Gpus'' command opens a modal dialog that allows you to select the graphics card to be used for each view. Currently only one gpu per view is supported. The Alt-G shortcut can also be used to open this dialog. The menu item is disabled when there are none or just one cuda-capable graphic card installed.
 +
 +
[[Image:Select_Gpus_Dlg.png]]
 
|}
 
|}
 
|}
 
|}
Line 313: Line 279:
 
|-
 
|-
 
|
 
|
:Opens the application's wiki documentation in the default browser.
+
:The ''Help'' command opens the X4D wiki page in the default browser. The F1 shortcut can also be used.  
 
|}
 
|}
  
Line 320: Line 286:
 
|-
 
|-
 
|
 
|
:Displays the [[About_Dialog|About dialog]].  
+
:The ''About'' option displays a dialog with information about X4D, including the installed version number. It also contains the ''Deactivate'' button, for deactivating your DSX suite license.
 
|}
 
|}
 
|}
 
|}
  
=Widgets=
+
==Widgets==
  
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DSX Configuration Widget
+
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DSX Configuration
 
|-
 
|-
 
|
 
|
Line 369: Line 335:
  
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Parameters Widget
+
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Parameters
 
|-
 
|-
 
|
 
|
[[Image:X4DParametersWidget.png]]
+
[[Image:X4DParametersWidget.png|right]]
  
 +
The Parameters widget displays all the user editable parameters. It is populated when a Subject file is loaded. Their descriptions can be found [[X4D_Parameters|here]].
 +
 +
<!--
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Constant DRR Scaling
 
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Constant DRR Scaling
 
|-
 
|-
 
|
 
|
:This boolean controls whether or not to scale the DRRs for each view by a constant value for all iterations of an optimization. If false, each DRR will be scaled so that its maximum value is equal to the DRR image scale value in the '''X-ray/DRR Settings''' widget. If false, a single scale value will be calculated during the first iteration and be used to scale the DRRs in all subsequent iterations.
+
:This boolean controls whether or not to scale the DRRs for each view by a constant value for all iterations of an optimization. If false, each DRR will be scaled so that its maximum value is equal to the DRR image scale value in the '''X-ray/DRR Settings''' widget. If true, a single scale value will be calculated during the first iteration and be used to scale the DRRs in all subsequent iterations.
 
|}
 
|}
  
Line 392: Line 361:
 
|-
 
|-
 
|
 
|
:This field allows the user to set the interval of the reporting times when generating the pose map for the first pass of the 4D optimization algorithm. For the optimization, a pose map is created by evaluating the object’s pose map at every Nth reporting time. The default value is 5.
+
:This field contains the interval of the reporting times when generating the pose map for the first pass of the 4D optimization algorithm. For the optimization, a pose map is created by evaluating the object’s pose map at every Nth reporting time. The default value is 5.
 
|}
 
|}
  
Line 399: Line 368:
 
|-
 
|-
 
|
 
|
:This field allows the user set the number of frames near the control point to evaluate during the first pass of 4D optimization. A value of 0 means to evaluate only the frame in each view that is closest to the control point. A value of 3 means to evaluate the closest frame plus the 3 closest frames on each side of the control point, for a total of seven. The default value is 1.  
+
:This field contains the number of frames near the control point to evaluate during the first pass of 4D optimization. A value of 0 means to evaluate only the frame in each view that is closest to the control point. A value of 3 means to evaluate the closest frame plus the 3 closest frames on each side of the control point, for a total of seven. The default value is 1.  
 
|}
 
|}
  
Line 406: Line 375:
 
|-
 
|-
 
|
 
|
:This field allows the user to set the interval of reporting times when generating the pose map for the second pass of the 4D optimization algorithm. For the optimization, a pose map is created by evaluating the pose map optimized in the first pass at every Nth reporting time. The default value is 3.
+
:This field contains the interval of reporting times when generating the pose map for the second pass of the 4D optimization algorithm. For the optimization, a pose map is created by evaluating the pose map optimized in the first pass at every Nth reporting time. The default value is 3.
 
|}
 
|}
  
Line 413: Line 382:
 
|-
 
|-
 
|
 
|
:This field allows the user to set the number of frames near the control point to evaluate during the second pass of 4D optimization. A value of 0 means to evaluate only the frame in each view that is closest to the control point. A value of 3 means to evaluate the closest frame plus the 3 closest frames on each side of the control point, for a total of seven. The default value is 3.
+
:This field contains the number of frames near the control point to evaluate during the second pass of 4D optimization. A value of 0 means to evaluate only the frame in each view that is closest to the control point. A value of 3 means to evaluate the closest frame plus the 3 closest frames on each side of the control point, for a total of seven. The default value is 3.
 
|}
 
|}
  
Line 420: Line 389:
 
|-
 
|-
 
|
 
|
:This field allows the user to enter the interval of the reporting time that will be used to generate motion capture seed poses. If the value entered is N then the motion capture data will be evaluated at every Nth reporting time to generate the mocap seed pose map. The default setting is 5.  
+
:This field contains the interval of the reporting time that will be used to generate motion capture seed poses. If the value entered is N then the motion capture data will be evaluated at every Nth reporting time to generate the mocap seed pose map. The default setting is 5.  
 
|}
 
|}
  
Line 427: Line 396:
 
|-
 
|-
 
|
 
|
:This field allows the user to enter the low-pass cutoff frequency for smoothing the cubic splines in the POI pose map. If the value is set to 0.0 or less the smoothing is turned off. The default value is 20.0.  
+
:This field contains the low-pass cutoff frequency for smoothing the cubic splines in the POI pose map. If the value is set to 0.0 or less the smoothing is turned off. The default value is 20.0.  
 
|}
 
|}
 +
 +
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 +
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Simulated Annealing Temperature
 +
|-
 +
|
 +
:This field contains the initial temperature for the simulated annealing algorithm. The higher the initial temperature, the more slowly the search range will decrease during the optimization. The default value is 2000.
 
|}
 
|}
  
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;X-ray and DRR Settings Widget
+
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Xray Opacity
 
|-
 
|-
 
|
 
|
 +
:This field contains the opacity of the xray images in the 2D and 3D windows. Their opacity should be less than 1.0 so that the DRRs can be seen on top of them. The default value is 0.5.
 +
|}
 +
-->
 +
|}
 +
 +
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 +
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;X-ray and DRR Settings
 +
|-
 +
|
 +
[[Image:X4D_XRayDRRSettingsWidget.png|right]]
 +
 
The matching of DRR images to X-ray images is sensitive to the image processing parameters in this widget. Once the DRR images have been generated for a particular set of bone poses, the DRR images and the X-ray images are processed (using an identical method), and then compared to each other.  
 
The matching of DRR images to X-ray images is sensitive to the image processing parameters in this widget. Once the DRR images have been generated for a particular set of bone poses, the DRR images and the X-ray images are processed (using an identical method), and then compared to each other.  
 
:The processing method consists of  
 
:The processing method consists of  
Line 441: Line 427:
 
#multiplying the edge-detection image by a weighting factor and adding it to the original image,
 
#multiplying the edge-detection image by a weighting factor and adding it to the original image,
 
#thresholding the merged image, and  
 
#thresholding the merged image, and  
#scaling the result. The parameters that control this process are in the X-ray and DRR groups of the widget.
+
#scaling the result.
  
[[Image:X4D_XRayDRRSettingsWidget.png]]
+
The parameters that control this process are in the ''X-ray/DRR Settings'' widget. Their descriptions can be found [[X4D_XRay/DRR_Settings|here]].
  
 +
<!--
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;X-ray and DRR
 
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;X-ray and DRR
Line 462: Line 449:
 
|
 
|
 
:These two values values govern the edge thresholding in step 2. Pixels in the edge image whose values are greater than the edge capping maximum are set to zero.  
 
:These two values values govern the edge thresholding in step 2. Pixels in the edge image whose values are greater than the edge capping maximum are set to zero.  
:Very bright pixels (sharp edges) are usually inorganic objects like EMG electrodes or metal plates or wires. They can be removed from the edge image by lowering the edge capping maximum from 100. Every pixel in the edge image whose value is above the edge capping minimum is set to the edge capping minimum. This effectively strengthens weaker edges (those below the edge capping minimum) because the entire image is scaled later. After changing either of them in the GUI, it is a good idea to press the ‘r’ key in each X-ray window to reset the window’s display brightness.
+
:Very bright pixels (sharp edges) are usually inorganic objects like EMG electrodes or metal plates or wires. They can be removed from the edge image by lowering the edge capping maximum from 100. Every pixel in the edge image whose value is above the edge capping minimum is set to the edge capping minimum. This effectively strengthens weaker edges (those below the edge capping minimum) because the entire image is scaled later.
 
|}
 
|}
  
Line 496: Line 483:
 
|
 
|
 
:These two parameter fields work together when evaluating a bone pose by comparing the DRR image to the X-ray image. The fitness of a particular pixel is weighted more heavily if the DRR pixel is bright and if the intensity is much different than the X-ray pixel's intensity. The raw fitness of a pixel is calculated by taking the squared difference between the DRR value and the X-ray value. The raw fitness is squared again if the DRR pixel's intensity is greater than the '''Bright Threshold''' parameter and the difference between it and the X-ray pixel's intensity is greater that the '''Diff Threshold'''. The default value for the '''Bright Threshold''' parameter is 128.0 and the range is from 0.0 to 255.0. The default value for the '''Diff Threshold''' parameter is 75.0 and the range is from 0.0 to 255.0.
 
:These two parameter fields work together when evaluating a bone pose by comparing the DRR image to the X-ray image. The fitness of a particular pixel is weighted more heavily if the DRR pixel is bright and if the intensity is much different than the X-ray pixel's intensity. The raw fitness of a pixel is calculated by taking the squared difference between the DRR value and the X-ray value. The raw fitness is squared again if the DRR pixel's intensity is greater than the '''Bright Threshold''' parameter and the difference between it and the X-ray pixel's intensity is greater that the '''Diff Threshold'''. The default value for the '''Bright Threshold''' parameter is 128.0 and the range is from 0.0 to 255.0. The default value for the '''Diff Threshold''' parameter is 75.0 and the range is from 0.0 to 255.0.
 +
|}
 +
 +
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 +
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;View Weight
 +
|-
 +
|
 +
:This factor is used when evaluating a bone pose by comparing the DRR images to the X-ray images. It is the relative weight of the two views in the final image metric (0.0 = all view 1, 1.0 = all view 2). The default value is 0.5, which means to weight each view equally. If you are tracking objects in single-plane image data (i.e., both views contain the same images), set this parameter to either 0.0 or 1.0 so that X4D will generate and evaluate DRRs for only one view, making the optimization twice as fast.
 
|}
 
|}
 
|}
 
|}
 +
-->
 
|}
 
|}
  
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Pose Maps Widget
+
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Pose Maps
 
|-
 
|-
 
|
 
|
Line 539: Line 534:
  
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Object Tracking Widget
+
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Object Tracking
 
|-
 
|-
 
|
 
|
Line 631: Line 626:
 
|
 
|
 
:Performs a detailed analysis of the DRR/X-ray matching for the current reporting time with the currently selected objects. A summary of the results is written to the Output window, and TIFF images of the X-rays, DRRs, and X-ray/DRR correlations are written to the folder containing the subject file.
 
:Performs a detailed analysis of the DRR/X-ray matching for the current reporting time with the currently selected objects. A summary of the results is written to the Output window, and TIFF images of the X-rays, DRRs, and X-ray/DRR correlations are written to the folder containing the subject file.
 +
|}
 +
 +
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 +
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Poses Saved During Tracking
 +
|-
 +
|
 +
:Sometimes the global minimum of the image fitness function is not the correct pose of the bone being tracked. This is often because the X-ray and DRR image processing parameters are not ideal (see ''Image Optimization'', below). Other times, it is because the bone in the X-ray images is occluded by soft tissue or other bones, or because of inherent differences between X-ray images and DRRs (image resolution, X-ray scatter, CT thresholding, etc.). To account for these other times, the ASA tracking algorithm attempts to save bone poses that result in local minima of the cost function, while still searching for the global minimum. The idea is that if the correct bone pose is not the global minimum, it is likely to be a local minimum somewhere within the search range. The ASA algorithm thus attempts to save a set of local-minimum poses so that you can revisit them once optimization has finished. The ''Max Poses'' field specifies how many local minima to save. When the tracking optimization has finished, if the final bone pose is not the correct pose, you can re-apply the saved poses by dragging the slider for that bone. If one of them is correct (or at least better than the final pose), you can use the ''Add Current Poses to Maps'' command in the ''Pose Map'' widget to add that pose to the map, thus overwriting the final pose.
 +
 +
:Note: When the ASA algorithm evaluates a particular bone pose during optimization, it cannot determine if it is actually a local minimum or not because it does not compute gradients. It estimates if the pose is a local minimum by examining its cost function value and its proximity to other poses with similar values. Here is an example that details the process. If you perform an ASA optimization of 1000 iterations, with ''Max Poses'' set to 20 and ''Distance (squared)'' set to 3.0, when the optimization is complete the 1000 bone poses that were evaluated are sorted from lowest to highest cost function value. The pose with the lowest value is marked as the global minimum and added to the list of saved poses. Then the pose with the next lowest value is checked to see how close it is to the first pose. The sum of the squares of the differences in the 6 DOF values is computed and compared to the ''Distance (squared)'' value (3.0 in this case). If the distance is less than 3.0, it is assumed that this pose is in the same valley of the solution space as the first pose, so it should not be saved as a local minimum. If it is greater than 3.0, the pose is added to the saved pose list. Then the next pose in the sorted list is checked. If it is not within 3.0 units of any pose already in the saved list, it is added to the saved list. This process proceeds until the saved list contains 20 poses. It will take some trial and error to determine the best value of the ''Distance (squared)'' parameter for a particular data set, but a value in the range of 1.0 to 3.0 is reasonable, as the solution spaces are often very bumpy with numerous local minima. The units for the DOF translations are model units (usually mm), and the units for the rotations are degrees.
 
|}
 
|}
 
|}
 
|}
  
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Image Optimization Widget
+
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Image Optimization
 
|-
 
|-
 
|
 
|
Line 644: Line 648:
  
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Graphs Widget
+
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Graphs
 
|-
 
|-
 
|
 
|
Line 662: Line 666:
 
|}
 
|}
  
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
+
{| class="mw-collapsible mw-collapsed wikitable" id="Mocap_Model_Widget" width="80%"
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Mocap Model Widget
+
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Mocap Model
 
|-
 
|-
 
|
 
|
Line 835: Line 839:
 
:‘<’ and ‘>’ rotates the selected objects about an axis perpendicular to the screen.
 
:‘<’ and ‘>’ rotates the selected objects about an axis perpendicular to the screen.
 
:The mouse wheel zooms the xray image in and out.
 
:The mouse wheel zooms the xray image in and out.
 +
:Press and hold the left mouse button to change the image contrast (move cursor left/right) and brightness (move cursor up/down).
 
:Press and hold the middle mouse button to pan the xray image within the window.
 
:Press and hold the middle mouse button to pan the xray image within the window.
 
:'r' resets the position of the xray image in the window, as well as the brightness and contrast.
 
:'r' resets the position of the xray image in the window, as well as the brightness and contrast.
Line 855: Line 860:
 
|
 
|
 
:Shows progress and results of optimizations. Also shows results of image evaluation.  
 
:Shows progress and results of optimizations. Also shows results of image evaluation.  
|}
 
 
=Motion Capture Data=
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Working with Motion Capture Data
 
|-
 
|
 
If you have surface-marker-based motion capture data for an x-ray trial, you can use it to calculate initial poses for the bones you want to track in the trial. To use this data in X4D, you must have C3D files with labelled marker trajectories for the motion trial and its associated static trial, and Visual3D files defining a kinematic model and its motion constraints. In the DSX software, this model is referred to as a ''mocap model''. Creating the model template and the constraint scripts requires a good working knowledge of Visual3D. The ''model template'' file (MDH) defines the segments and landmarks in the kinematic model, based on a specific marker set. The ''build script'' (V3S) creates the constraints that define the relative motions of the segments in the model. The ''recalc script'' (V3S) updates the segment poses to satisfy the constraints. Once the model is defined, and applied to a particular DSX trial, it can be used in three modes:
 
 
* '''Marker Mode''' - The motions of the segments are driven by the surface markers, with constraints enforced. This mode is used to calculate mocap poses in X4D.
 
 
* '''Interactive Mode''' - You can interactively change all of the model's degrees of freedom (the 6 degrees of freedom of the root segment, and the joint angles). Constraints are enforced. This mode is used when you manipulate the bone poses in X4D.
 
 
* '''DSX Mode''' - The motions of the segments are driven by DSX tracking results (transform matrices). Constraints are not enforced. This mode is used when DSX results are viewed and analyzed in Visual3D.
 
 
In addition to generating initial poses for the tracking objects, you can use the mocap model to help with manual positioning of the objects. In the [[Mocap|Mocap Model]] widget in X4D, you can turn on the model's joint constraints, making it easier to move all of the objects together as a group, or to move them relative to each other in a realistic manner (e.g., flexing or twisting the vertebrae in a spine model).
 
|}
 
 
{| class="mw-collapsible mw-collapsed wikitable" width="80%"
 
! style="text-align:left;" | &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;How To: Calculate Bone Poses from Motion Capture Data
 
|-
 
|
 
Once you have created the model template and constraint scripts, the general process for using a mocap model with a DSX trial is:
 
 
'''1)''' Calculate the transform between the x-ray lab frame and the mocap system frame. CalibrateDSX does this using the surface markers and radiopaque beads on the calibration object.
 
 
'''2)''' Create the mocap model using the static trial. This determines the offset between each DSX segment (e.g., right distal femur) and its corresponding mocap segment (e.g., right thigh).
 
 
'''3)''' Apply the mocap model to the marker data for the motion trial. This determines the poses of the mocap segments (and thus DSX segments) for every frame of the x-ray trial.
 
 
To create a mocap model and use it to generate poses for the objects in a DSX trial, perform the following steps:
 
 
'''1)''' xManager: specify the C3D file for the motion trial and its associated static trial.
 
 
'''2)''' xManager: specify the model template (MDH) and build and recalc scripts (V3S) for the static trial.
 
 
'''3)''' xManager: specify the static trial as the reference trial for the motion trial.
 
 
'''4)''' CalibrateDSX: load the calibration object trial and track the beads in it for all frames.
 
 
'''5)''' CalibrateDSX: select Mocap -> Compute DSX/mocap transform.
 
 
'''6)''' X4D: load the static trial and track all of the objects in any one x-ray frame. Save the pose maps.
 
 
'''7)''' CalibrateDSX: load the static trial, then select Mocap -> Create Mocap Model.
 
 
'''8)''' X4D: load the motion trial and select all of the objects. In the Pose Maps widget, select Calculate Mocap Poses.
 
 
|}
 
|}

Revision as of 17:23, 25 March 2019

Language:  English  • français • italiano • português • español 

X4D tracks 3D objects (bones, implants, etc.) in X-ray images by generating digitally reconstructed radiographs (DRRs) of the objects and matching them to the X-ray images. It requires that the 3D X-ray configuration parameters be defined in the subject file and that the X-ray images be corrected.

To match DRR images to X-ray images, both sets of images are processed first. X-ray images are smoothed with a convolution filter, and then an edge detection algorithm is performed on them. The final processed X-ray image that is used for tracking is the edge detection image times a weighting factor, plus the smoothed intensity image. DRR images are not smoothed, but are input to the edge detection algorithm. The final processed DRR image is the edge detection image time a weighting factor, plus the original DRR image.

X4D Window.png

Tutorials

How To: Get Started Tracking Bones

How To: Track Multiple Bones

How To: Optimize Xray/DRR Settings

How To: Calculate Bone Poses from Mocap Data

Bone Tracking

     4D Optimization

4D optimization tracks one or more objects by solving all frames in the tracking range at the same time. It works using a simulated annealing optimization process over two passes. In the first pass, a temporary pose map is created by evaluating the object’s pose map over the tracking range at equal time stamps. The object poses at these times are the control points that the optimizer varies to find the global solution. The spacing between the time stamps is determined by the Frame Spacing 4D Pass1 parameter. For example, if the parameter is set to 4 and there are 20 reporting times in the tracking range, there will be 20 / 4 + 1 = 6 poses in the pose map-- a pose at each end of the range and 4 poses equally spaced between them. The algorithm then proceeds to optimize each control point independently of the others, while evaluating the DRR/X-ray matching for the X-ray frames in the neighborhood of the control point. The size of this neighborhood is determined by the Frame Envelope 4D Pass1 parameter. For example, if the parameter is set to 2, each time the optimizer changes the pose at one of the time stamps, it will evaluate the DRR/X-ray matching at the closest X-ray frame to that time (in each view), and in the 2 frames on either side of that closest frame.

In the second pass, a temporary pose map is created by evaluating the optimized pose map from the first pass at equal time stamps. This process is the same as described above, but uses the Frame Spacing 4D Pass2 parameter to determine the spacing. The algorithm then optimizes each control point independently, evaluating a neighborhood of frames determined by the Frame Envelope 4D Pass2 parameter. For this second pass, the number of simulated annealing iterations per control point is halved, the translation and rotation ranges are divided by four, and the initial temperature is divided by four. Once the second pass is complete, its final pose map is evaluated at the reporting times in the tracking range and those poses are inserted into the object’s pose map.

     Matching Xray and DRR Images

The process of matching the DRR images to the X-ray images is sensitive to the image processing parameters in the X-ray/DRR Settings widget. Once the DRR images have been generated for a particular set of bone poses the DRR images and the X-ray images are processed (using an identical method) and then compared to each other.

The processing method consists of:

  1. performing a Sobel edge detection on the image,
  2. thresholding the edge-detection image (which is controlled by the Edge Capping values in the X-ray and DRR Settings widget),
  3. multiplying the edge-detection image by a weighting factor (which is controlled by the Edge/Intensity Merge value in the X-ray and DRR Settings widget) and adding it to the original image,
  4. thresholding the merged image, and
  5. scaling the result.

Pixels in the edge image whose values are greater than the Edge Capping maximum are set to zero. Very bright pixels (sharp edges) are usually inorganic objects like EMG electrodes or metal plates or wires. They can be removed from the edge image by lowering the Edge Capping maximum from 100.

Every pixel in the edge image whose value is above the Edge Capping minimum is set to the Edge Capping minimum. This effectively strengthens weaker edges (those below the Edge Capping minimum) because the entire image is scaled later. For example, if the Edge Capping values are set to 5 and 30, every pixel in the edge image that is greater than 30 is set to 0. Every pixel that is between 5 and 30 is set to 5. The resulting edge image has all pixels between 0 and 5, with many of the edges of interest set to 5, making them all equally strong.

After the edge image has been weighted and added to the original image, the result is thresholded using the Image Threshold values. All pixels above the Image Threshold maximum are set to the maximum, and all pixels below the Image Threshold minimum are set to 0. Much of the time these thresholds should be left at 100 and 0. However, there are times when it is useful to raise the minimum above zero to mask soft tissue regions, and lower the maximum from 100 to remove artificial edges, such as the end of a CT bone that is within the X-ray image.

Note: the Image Scale Value is used in step 5. The final processed image is scaled linearly so that all pixels are in the range 0 to this scale value. Most of the time the scale values for the X-ray and DRR images should both be set to 255.0, but occasionally there are times when it is helpful to scale one image more than the other (e.g., when most of the bone edges in a DRR image are too dull because of one bright area that scales to 255.0).

Once the X-ray and DRR images have been processed, they are compared pixel-by-pixel to evaluate the quality of the bone poses. The parameters in the Image Fitness group give you access to some of the variables in the fitness equation. To compare a DRR pixel to an X-ray pixel, first the absolute value of the difference between them is calculated. If this difference is greater than the Difference Threshold, and the DRR pixel is greater than Bright DRR Pixel Threshold, the difference is squared. This gives a greater weight to DRR pixels that are bright and which do not match well with their corresponding X-ray pixels. The pixel difference is then squared and multiplied by a brightness factor. This factor is 1.0 plus Bright DRR Pixel Factor times the DRR pixel value. When Bright DRR Pixel Factor is zero, bright DRR pixels are not weighted differently than any others. But when it is greater than zero, the DRR’s brightness is used to weight the error for that pixel. This is a second method of weighting a bright DRR pixel more heavily, without considering its difference with the X-ray pixel (as the first method does). The values for all pixels in each view are then summed to determine the fitness for that view. The sums for the two views are then multiplied to get the overall image correlation value, which the algorithm tries to minimize.

The success of the bone tracking algorithm is particularly sensitive to the Edge Capping minimum for the X-ray images. A change in this value of 2 or 3 (percent) can make a big difference in the solution found by the optimization. If you find that the optimization is locking into a bad pose with the edge capping minimum set to 10, try lowering the minimum to 7 or 8 (or even 5) and trying again.

When you start processing a new data set, it can take some trial and error to figure out the set of image processing parameters that produce the best results. It is recommended that you start by tracking the bones (individually) in a single frame, trying different parameter values until the optimization algorithm can lock onto the correct pose for each bone. Then try solving a few adjacent frames to see if the same parameter values work on those.

     Objective Function for Optimization

DSX ObjectiveFunction.jpg

Motion Capture Data

     Working with Motion Capture Data

If you have surface-marker-based motion capture data for an x-ray trial, you can use it to calculate initial poses for the bones you want to track in the trial. To use this data in X4D, you must have C3D files with labelled marker trajectories for the motion trial and its associated static trial, and Visual3D files defining a kinematic model and its motion constraints. In the DSX software, this model is referred to as a mocap model. Creating the model template and the constraint scripts requires a good working knowledge of Visual3D. The model template file (MDH) defines the segments and landmarks in the kinematic model, based on a specific marker set. The build script (V3S) creates the constraints that define the relative motions of the segments in the model. The recalc script (V3S) updates the segment poses to satisfy the constraints. Once the model is defined, and applied to a particular DSX trial, it can be used in three modes:

  • Marker Mode - The motions of the segments are driven by the surface markers, with constraints enforced. This mode is used to calculate mocap poses in X4D.
  • Interactive Mode - You can interactively change all of the model's degrees of freedom (the 6 degrees of freedom of the root segment, and the joint angles). Constraints are enforced. This mode is used when you manipulate the bone poses in X4D.
  • DSX Mode - The motions of the segments are driven by DSX tracking results (transform matrices). Constraints are not enforced. This mode is used when DSX results are viewed and analyzed in Visual3D.

In addition to generating initial poses for the tracking objects, you can use the mocap model to help with manual positioning of the objects. In the Mocap Model widget in X4D, you can turn on the model's joint constraints, making it easier to move all of the objects together as a group, or to move them relative to each other in a realistic manner (e.g., flexing or twisting the vertebrae in a spine model).

Menus

     File Menu
     X4DLoadSubjectButton.png
The Load Subject command loads a Subject file.
This command can also be accessed with the Ctrl + O shortcut.
     X4DSaveSubjectButton.png
The Save Subject command saves the currently loaded subject to its existing Subject file.
This command can also be accessed with the Ctrl + S shortcut.
     X4DSaveSubjectAsButton.png
The Save Subject As command saves the currently loaded subject to a new Subject file.
     X4DLoadPoseMapsButton.png
The Load Pose Maps command loads an existing pose map file for each selected object. You will be prompted to browse for each file, which must be saved as a .csv file.
     X4DSavePoseMapsButton.png
The Save Pose Maps command saves the pose map of each selected object in the current trial to a .csv file. This file contains 6 values for each pose: the XYZ translation and the XYZ Euler angles representing the transform from the xray lab frame to the object's local coordinate system. If a name for this file is not already specified in the subject file, you will be prompted to browse for the file to create. A link to this file will then be added to the Subject file.
     X4DSavePoseMapsAsButton.png
The Save Pose Maps As command saves the pose map for each selected object in the current trial to a new .csv file. You will be prompted to browse for a new file for each object. A link to this new file will then be added to the Subject file.
     X4DSaveObjectTransformsButton.png
The Save Object Transforms command saves the pose map of each selected object in the current trial to a .txt file. This file contains a 4x4 transform for each pose, expressing the transform from the xray lab frame to the object's local coordinate system. If a name for this file is not already specified in the subject file, you will be prompted to browse for the file to create. A link to this file will then be added to the Subject file.
     X4DSaveLandmarksButton.png
The Save Landmarks command saves the locations in the xray lab frame of each selected object's landmarks in the current trial. This file contains a set of landmark coordinates for each pose in the pose map. If a name for this file is not already specified in the subject file, you will be prompted to browse for the file to create. A link to this file will then be added to the Subject file.
     X4DExportPosesButton.png
The Export Poses to CMZ command exports the pose map of each selected object in the current trial to the session's CMZ file. This requires that the CMZ already contain a C3D file for the current trial, with a kinematic model that includes segments for the selected objects.
     X4DExitButton.png
The Exit command exits the program and all unsaved data will be lost.
The program can also be closed by using the X in the top right corner of the program window.
     View Menu

The view menu controls which screen widgets are displayed

     X4DBodyReferenceFramesButton.png
The Body Reference Frames control toggles the display of the local coordinate systems of the selected objects in the xray windows and the 3D view. It can also be toggled using Alt + R.
     X4DBodyLandmarks.png
The Body Landmarks control toggles the display of the landmarks of the selected objects in the xray windows and the 3D view. It can also be toggled using Alt + L.
     X4DDSXConfigurationButton.png
The DSX Configuration control toggles the appearance of the dockable DSX Configuration widget. It can also be controlled using the Ctrl + Shift + C shortcut.
     X4DParametersButton.png
The Parameters control toggles the appearance of the dockable Parameters widget. It can also be controlled using the Ctrl + Shift + P shortcut.
     X4D3DView.png
The 3D View control toggles the appearance of the dockable 3D View window. It can also be controlled using the Ctrl + Shift + V shortcut.
     X4DOutputButton.png
The Output control toggles the appearance of the dockable Output window. It can also be controlled using the Ctrl + Shift + O shortcut.
     X4DDRRSettingsButton.png
The Xray/DRR Settings control toggles the appearance of the dockable Xray/DRR Settings widget. It can also be controlled using the Ctrl + Shift + X shortcut.
     X4DPoseMapsButton.png
The Pose Maps control toggles the appearance of the dockable Pose Maps widget. It can also be controlled using the Ctrl + Shift + A shortcut.
     X4DObjectTrackingButton.png
The Object Tracking control toggles the appearance of the dockable Object Tracking widget. It can also be controlled using the Ctrl + Shift + T shortcut.
     X4DImageOptimizationButton.png
The Image Optimization control toggles the appearance of the dockable Image Optimization widget. It can also be controlled using the Ctrl + Shift + I shortcut.
     X4DGraphsButton.png
The Graphs control toggles the appearance of the dockable Graphs widget. It can also be controlled using the Ctrl + Shift + G shortcut.
     X4DMocapModelButton.png
The Mocap Model control toggles the appearance of the dockable Mocap Model widget. It can also be controlled using the Ctrl + Shift + M shortcut.
     Options Menu
     Select Gpus Menu.png
When more than one cuda-capable graphic cards are installed in the computer, the Select Gpus command opens a modal dialog that allows you to select the graphics card to be used for each view. Currently only one gpu per view is supported. The Alt-G shortcut can also be used to open this dialog. The menu item is disabled when there are none or just one cuda-capable graphic card installed.

Select Gpus Dlg.png

     Help Menu
     S3DHelpButton.png
The Help command opens the X4D wiki page in the default browser. The F1 shortcut can also be used.
     S3DAboutButton.png
The About option displays a dialog with information about X4D, including the installed version number. It also contains the Deactivate button, for deactivating your DSX suite license.

Widgets

     DSX Configuration

X4DDSXConfigurationWidget.png

     Subject
The Subject field is automatically populated with the ID of the subject when you load a Subject file.
     Session
The Session drop down menu is automatically populated with all of the sessions available available for the current subject.
     Configuration
The Configuration drop down menu is automatically populated with all of the configurations available for the current session.
     Trial
The Trial drop down menu is automatically populated with all of the trials available for the current configuration.
     Object
The Object list is automatically populated with all of the objects in the current trial. Each object is enabled if it is trackable, meaning that it has a 3D image file associated with it, for generating DRRs.
     Parameters
X4DParametersWidget.png

The Parameters widget displays all the user editable parameters. It is populated when a Subject file is loaded. Their descriptions can be found here.

     X-ray and DRR Settings
X4D XRayDRRSettingsWidget.png

The matching of DRR images to X-ray images is sensitive to the image processing parameters in this widget. Once the DRR images have been generated for a particular set of bone poses, the DRR images and the X-ray images are processed (using an identical method), and then compared to each other.

The processing method consists of
  1. performing a Sobel edge detection on the image,
  2. thresholding the edge-detection image
  3. multiplying the edge-detection image by a weighting factor and adding it to the original image,
  4. thresholding the merged image, and
  5. scaling the result.

The parameters that control this process are in the X-ray/DRR Settings widget. Their descriptions can be found here.

     Pose Maps

A pose map is the set of poses for a tracked object. Each pose expresses the transform from the X-ray (lab) frame to the object’s anatomical frame, and is at one of the trial’s reporting times. Each of the 6 DOFs in a pose map is interpolated (and possibly smoothed) with a GCV (generalized cross-validation) spline. Smoothing of pose map splines is controlled by the Pose Map Cutoff Frequency parameter. Pose maps are stored in CSV files.

X4DPoseMapsWidget.png

     Add Current Poses to Maps
For each selected object, add its pose at the current reporting time to its pose map.
     Clear Current Poses From Maps
For each selected object, removes from its pose map the pose at the current reporting time.
     Clear Pose Maps
Clears all poses from the pose maps of the selected objects.
     Smooth Pose Maps
For each selected object, replaces each pose in the pose map with one calculated by evaluating the splines at the pose time.
Note: the resulting splines may still not pass through the smoothed poses because of the nature of GCV cut-off frequency smoothing.
     Object Tracking

X4DObjectTrackingWidget.png

     Optimize
These check boxes let you specify which objects are allowed to move during the tracking optimization. DRRs for all of the objects selected in the Object Configuration panel will be generated, but only the objects selected here will be allowed to move. This allows you to perform hierarchical tracking, such as tracking the femur, then fixing it in its optimized pose while tracking the patella.
     Max Iterations
The maximum number of iterations for one pass of the optimization algorithm. For single-frame optimization, this is the number of iterations for one frame (reporting time). For 4D optimization, this is the number of iterations for the first pass. For the second pass of 4D, half of this number is used as the maximum.
     Translation Ranges
There is a slider for each of the X, Y, and Z translation ranges. Each one controls the amount that the DOF is allowed to change from its initial value during optimization.
     Rotation Ranges
There is a slider for each of the X, Y, and Z rotation ranges. Each one controls the amount that the DOF is allowed to change from its initial value during optimization.
     Tracking
The range of reporting times which will be optimized when using the Solve Range, Solve Poses in Range, or Solve Range 4D commands.
     Solve Range
Starts the frame-by-frame optimization process. The selected objects’ poses are optimized at each reporting time in the tracking range. It starts at the current time and proceeds towards the earliest, then goes back to the current time and proceeds towards the latest. Although each frame (reporting time) is optimized independently of the others, previously optimized frames will modify the pose map, which could affect the starting poses for subsequent frames.
Note: the optimization will execute much faster if you turn off the display of the DRRs and CT objects in the 2D and 3D windows.
     Solve Current Frame
Performs an optimization of the current frame.
Note: the optimization will execute much faster if you turn off the display of the DRRs and CT objects in the 2D and 3D windows.
     Solve Poses in Range
Performs an optimization of each reporting time in the tracking range that already has a pose specified in the pose map.
Note: the optimization will execute much faster if you turn off the display of the DRRs and CT objects in the 2D and 3D windows.
     Solve Range 4D
Starts the 4D optimization process. The entire tracking range is optimized at the same time.
Note: the optimization will execute much faster if you turn off the display of the DRRs and CT objects in the 2D and 3D windows.
     Optimizer (ASA, SPAN, LBFGSB)
The type of optimization algorithm to use. ASA (adaptive simulated annealing) is currently the only recommended algorithm. SPAN is another type of simulated annealing. LBFGSB is a bounded, least-squares algorithm (local optimizer).
     Stop
Stops the currently running optimization.
     Evaluate
Performs a detailed analysis of the DRR/X-ray matching for the current reporting time with the currently selected objects. A summary of the results is written to the Output window, and TIFF images of the X-rays, DRRs, and X-ray/DRR correlations are written to the folder containing the subject file.
     Poses Saved During Tracking
Sometimes the global minimum of the image fitness function is not the correct pose of the bone being tracked. This is often because the X-ray and DRR image processing parameters are not ideal (see Image Optimization, below). Other times, it is because the bone in the X-ray images is occluded by soft tissue or other bones, or because of inherent differences between X-ray images and DRRs (image resolution, X-ray scatter, CT thresholding, etc.). To account for these other times, the ASA tracking algorithm attempts to save bone poses that result in local minima of the cost function, while still searching for the global minimum. The idea is that if the correct bone pose is not the global minimum, it is likely to be a local minimum somewhere within the search range. The ASA algorithm thus attempts to save a set of local-minimum poses so that you can revisit them once optimization has finished. The Max Poses field specifies how many local minima to save. When the tracking optimization has finished, if the final bone pose is not the correct pose, you can re-apply the saved poses by dragging the slider for that bone. If one of them is correct (or at least better than the final pose), you can use the Add Current Poses to Maps command in the Pose Map widget to add that pose to the map, thus overwriting the final pose.
Note: When the ASA algorithm evaluates a particular bone pose during optimization, it cannot determine if it is actually a local minimum or not because it does not compute gradients. It estimates if the pose is a local minimum by examining its cost function value and its proximity to other poses with similar values. Here is an example that details the process. If you perform an ASA optimization of 1000 iterations, with Max Poses set to 20 and Distance (squared) set to 3.0, when the optimization is complete the 1000 bone poses that were evaluated are sorted from lowest to highest cost function value. The pose with the lowest value is marked as the global minimum and added to the list of saved poses. Then the pose with the next lowest value is checked to see how close it is to the first pose. The sum of the squares of the differences in the 6 DOF values is computed and compared to the Distance (squared) value (3.0 in this case). If the distance is less than 3.0, it is assumed that this pose is in the same valley of the solution space as the first pose, so it should not be saved as a local minimum. If it is greater than 3.0, the pose is added to the saved pose list. Then the next pose in the sorted list is checked. If it is not within 3.0 units of any pose already in the saved list, it is added to the saved list. This process proceeds until the saved list contains 20 poses. It will take some trial and error to determine the best value of the Distance (squared) parameter for a particular data set, but a value in the range of 1.0 to 3.0 is reasonable, as the solution spaces are often very bumpy with numerous local minima. The units for the DOF translations are model units (usually mm), and the units for the rotations are degrees.
     Image Optimization

Accurate tracking of bones is sensitive to the X-ray and DRR image processing parameters. In some cases it can take a significant amount of trial and error to determine the parameter values that enable the image matching algorithm to "lock in" on the correct bone pose. That is, the parameter values that cause the global minimum of the image fitness function to be at the correct bone pose. There are 15 different parameters, and sometimes changes of 1 or 2% in one of them can change the "optimal" bone pose by a few degrees or millimeters. The Image Optimization widget is a tool for automatically finding optimal values of the parameters. The sliders allow you to specify ranges for each of the 15 parameters in the Xray/DRR Settings widget, and the Max Iterations slider at the top is for specifying how many tracking optimizations to perform with randomly chosen parameter values within the specified ranges. Each tracking optimization proceeds as if you had selected Solve Current Frame in the Object Tracking widget. It optimizes the selected bones, starting at their current poses, and compares the final poses to the initial ones. The smaller this difference, the better the result. Please see How To: Optimize Xray/DRR Settings for a step-by-step guide on using this widget.

X4DImageOptimizationWidget.png

     Graphs

In the Graphs widget you can display and edit the pose map of one of the selected objects. The graphs show the 6 DOFs of the object's pose at each reporting time. To display the graphs, select an object from the combo box in the upper left corner of the widget. With the checkboxes at the top you can individually control the display of the X, Y, and Z translations and the X, Y, and Z rotations (Euler angles).


X4DGraphsWindow.png

The circles on the graphs represent the poses in the object’s pose map. The dashed lines are GCV splines fit to the circles (control points) and smoothed with the cutoff frequency specified by the Pose Map Cutoff Frequency parameter in the Parameters widget. The gray region behind the curves represents the tracking range specified in the Object Tracking widget. The vertical green line shows the current reporting time.

The graphs can be moved left, right, up, and down in the window using the Shift key and the left mouse button. The Y axis can be zoomed in and out using the mouse wheel. Pressing 'r' resets the graph axes to display the splines for all of the selected curves in that graph. Pressing Ctrl + 'r' resets the graph axes to display all of the circles of the selected curves.

To move a control point, left-click on it and drag it up or down to a new location (you cannot change the time of a control point). You can select multiple points by pressing Ctrl while left-clicking on the points, or by sweeping a region while pressing the left mouse button. You can then move all selected points together by left-clicking on any of them and dragging up or down. To unselect all control points, left-click away from all points or press Esc.

Note: moving control points in the graphs directly modifies the pose map that is loaded into X4D, and there is no undo feature. It does not, however, modify the pose map file; this can only be done by selecting Save Pose Maps from the File menu. Also, whenever a pose map is modified by any method in X4D, it is saved to a backup file in the same folder as the original pose map file.

     Mocap Model

X4DMocapModelWidget.png

     Time Offset
The offset between the start of the x-ray data capture and the surface-marker motion capture. It is specified as the x-ray start time minus the motion capture start time.
     Calculate Mocap Poses, Fixed Body
If Fixed Body is set to none, then Calculate Mocap Poses will clear the pose maps for the objects selected in the Configuration widget and create new poses based on the motion capture data. Poses are created for the entire trial, using the value of the Mocap Seed Spacing parameter. For example, if the spacing is 5, then a motion capture pose will be calculated for every 5th reporting time in the trial.
If Fixed Body is set to one of the tracked objects in the trial, then that object's pose map will not be modified by Calculate Mocap Poses. Instead, the pose maps for the other selected objects will be cleared. Then for each pose in the fixed object's pose map, the poses of the other objects will be calculated such that they satisfy the joint constraints relative to the fixed object and the generalized coordinates of the motion capture data.
Note: One strategy for tracking multiple bones in a trial is to first Calculate Mocap Poses with no fixed body. This will position all of the bones using the surface markers in the motion capture data. Then track one of the bones for all frames in which it is trackable, deleting the extra mocap-based poses when you are done. Then Calculate Mocap Poses again, with Fixed Body set to the bone that you just tracked. The other bones will "snap into place" relative to the tracked bone. In other words, it will use the motion capture data to pose all the bones, then move them as a group so that the fixed bone lines up with its tracked pose.
     Apply Joint Constraints
When this option is selected, the constraints in the kinematic model specified for the trial are applied to the tracking objects. When you select it, the root object remains in its current pose, and the other objects are moved so as to satisfy the kinematic constraints. You can then move the entire model as a rigid unit by selecting and moving any object in it, and you can move the joints of the model by moving the generalized coordinate sliders below the checkbox.
     3D View

The 3D View window displays a 3D scene representing the X-ray configurations and the tracking objects.

The surface models generated from CT data can be manipulated by the user. To do so the model first needs to be selected by left-clicking on it. A trackball consisting of three circles and three lines will be displayed centered in and aligned with the selected object’s local (anatomical) coordinate system. The trackball allows the selected object to be manipulated as follows:

  • When the left mouse button is depressed on one of the circles, the circle will be highlighted to visually indicate it is selected and when the mouse cursor is dragged across the screen (the left mouse button is still depressed), the object is rotated in the plane of the selected circle.
  • When the left mouse button is depressed on one of the lines, the line will be highlighted to visually indicate it is selected and when the mouse cursor is dragged across the screen (the left mouse button is still depressed), the object is translated along the selected line.
  • Left-clicking within the trackball without selecting any of the circles or lines, will highlight all circles. When the mouse cursor is dragged across the screen (the left mouse button is still depressed), freehand rotation of the object will occur.
  • Left-clicking within the trackball without selecting any of the circles or lines but with depressing the shift key, will highlight all lines. When the mouse cursor is dragged across the screen (the left mouse button is still depressed), freehand translation of the object will occur.
  • Left-clicking outside the trackball will make the trackball disappear and deselect the object.

Note: moving an object this way does not move the other objects with it, even if Apply Joint Constraints is turned on.

X4D 3DView.png

     Show Lab
Toggles the display of the X-ray source, X-ray image frame, and projection line.
     Show CT
Toggles the display of the CT bones of the selected objects.
     Show View 1 Image
Toggles the display of the X-ray + DRR image for view 1.
     Show View 2 Image
Toggles the display of the X-ray + DRR image for view 2.
     Show Axes
Toggles the display of the X-ray lab reference frame.
     Show Mocap Markers
Toggles the display of the motion capture markers for the current reporting time.
     Align View with X-Axis
Moves the camera to look down the negative X axis.
     Align View with Y-Asix
Moves the camera to look down the negative Y axis.
     Align View with Z-Axis
Moves the camera to look down the negative Z axis.
     Align View with View 1 Source
Moves the camera to the x-ray source in View 1, and points it along the x-ray projection line.
     Align View with View 2 Source
Moves the camera to the x-ray source in View 2, and points it along the x-ray projection line.
     Xray Windows

The Xray windows display the xrays of each view, as well as the DRRs of all selected objects.

X4D Viewer.png

     Moving the DRRs

To move an object in one of the 2D xray windows, Shift-right-click on the DRR to pick it. A reference frame will be displayed over it, with its origin at the center of the object's CT volume. To translate the object, Shift-left-click on the DRR and drag it to a new location. To rotate the object about the origin of the displayed reference frame, Ctrl-left-click on the DRR and drag the cursor. To unpick the object, press the Escape key or Shift-right-click in the window away from the DRRs.

Note 1: Moving objects with the mouse does not modify their pose maps. To save the new poses you must press the Add Current Poses to Maps button in the Pose Map widget. If you do not save the poses, they will be lost when you change the current reporting time.

Note 2: If Apply Joint Constraints is turned on in the Mocap Model widget, all of the objects in the trial will move with the one you manipulate directly, even if they are not selected.

     Mouse and Keyboard Commands

The following mouse and keyboard commands are available in the 2D windows for interacting with the xray images and DRRs. The commands that move the objects move all objects selected in the DSX Configuration widget, whether or not they have been picked for interaction (with Shift-left-click).

To rotate all selected objects by +0.5 degrees about their local X, Y, and Z axes, press Ctrl and the X, Y, or Z keys.
To rotate all selected objects by -0.5 degrees about their local X, Y, and Z axes, press Alt + Ctrl and the X, Y, or Z keys.
To translate all selected objects by +0.5 mm along their local X, Y, and Z axes, press Shift and the X, Y, or Z keys.
To translate all selected objects by -0.5 mm along their local X, Y, and Z axes, press Alt + Shift and the X, Y, or Z keys.
The left, right, up, and down arrow keys translate all selected objects by 0.5 mm in the plane of the xray window.
Pg Up/Down: moves to the previous/next reporting time.
‘<’ and ‘>’ rotates the selected objects about an axis perpendicular to the screen.
The mouse wheel zooms the xray image in and out.
Press and hold the left mouse button to change the image contrast (move cursor left/right) and brightness (move cursor up/down).
Press and hold the middle mouse button to pan the xray image within the window.
'r' resets the position of the xray image in the window, as well as the brightness and contrast.
's' toggles the display of the xray image (same as Show Xray checkbox)
'd' toggles the displays of the DRRs (same as Show DRR checkbox)
'p' toggles whether the displayed xray is the raw image or the processed image (same as Show Processed Xray checkbox)
'o' toggles whether the displayed DRRs are the raw ones or the processed ones (same as Show Processed DRR checkbox)
Alt + 'r' toggles the display of the selected objects' local coordinate systems
Alt + 'l' toggles the display of the selected objects' landmarks

Note 1: Moving objects with keyboard commands does not modify their pose maps. To save the new poses you must press the Add Current Poses to Maps button in the Pose Map widget. If you do not save the poses, they will be lost when you change the current reporting time.

Note 2: If Apply Joint Constraints is turned on in the Mocap Model widget, all of the objects in the trial will move with the one you manipulate directly, even if they are not selected.

     Output Window
Shows progress and results of optimizations. Also shows results of image evaluation.
Retrieved from ""