Plus applications user manual  2.9.0.d0fd355
Software library for tracked ultrasound image acquisition, calibration, and processing.
Optical Marker Tracker

OpticalMarkerTracker is virtual device that provides position tracking of markers using any webcam or camcorder. You can print your own markers on any regular printer and attach to a flat surface on an object. The device uses algorithms implemented in OpenCV and ArUco software libraries.

Supported hardware devices

  • This is a virtual (software-only) device, which uses images taken by any optical imaging device that Plus supports, such as external webcam, built-in laptop camera - either forward or rear facing, camcorder connected through framegrabber, etc.

Supported platforms

  • This device can be used in combination with any device in Plus that provides a video stream. Most frequently it is used with Microsoft Media Foundation devices (webcams, framegrabbers).

Installation

To get started, print a set of markers and create a configuration file. It is recommended to use "PlusServer: Optical marker tracker using MMF video" configuration file that is included in your Plus installation package (see also at the bottom of this page) and print this sheet that contains a number of markers with different sizes (make sure to print in actual size: scaling should be 100%): marker_sheet_36h12.pdf. If you put the sheet in the field of view of your camera, the device will detect position and orientation of the markers.

To improve tracking accuracy, you need to create a calibration file for your camera as described below. If you need more tracking markers, you can print more by following instructions below.

Calibration

Camera calibration determines optical properties of your camera (distortion, focal length, etc) to improve tracking accuracy.

  • WARNING: Camera calibration uses an external application, which uses meters as unit for length. This is the ONLY place length should be specified in meters. Everywhere else in Plus it is millimeter.
  • Print the aruco calibration chessboard and measure the side length of the markers.
  • To generate a calibration file, run aruco_calibration.exe
    Example:
    aruco_calibration.exe live:0 camera_calibration.yml -size 0.03556
    Reference:
    aruco_calibration.exe live:<camera-index> <output-file-name>.yml -size <marker-size-m>
    where:
    • camera-index is the integer index of the camera you want to use (usually 0 or 1).
    • output-file-name is the desired output file name.
    • marker-size-m is the size of a marker side in meters.
  • With the video window selected (keyboard shortcuts will not work while the command line window is selected):
    • Press a to capture image and add to the calibration image set.
    • Press c to run calibration on the captured images. "repj" is a measure of the calibration error (normal is < 1).
    • Press s to stop calibration.
  • For best results use at least 6 images including:
    • At least one frontal image.
      DeviceOpticalMarkerTrackerFrontal.png
    • Multiple images where the border of the chessboard is near the top or side image border.
      DeviceOpticalMarkerTrackerEdge.png
    • Multiple images where the chessboard is held at an oblique angle to the camera.
      DeviceOpticalMarkerTrackerOblique1.png
      DeviceOpticalMarkerTrackerOblique2.png

Creating printable marker images

  • A premade sheet of ARUCO_36h12 markers in three standard sizes (80mm, 50mm, and 30mm) is located at /ConfigFiles/OpticalMarkerTracker/marker_sheet_36h12.pdf.
  • Some pre-generated marker images from the ARUCO_MIP_36h12 dictionary are included at /ConfigFiles/OpticalMarkerTracker/markers.
  • To generate a single image for printing, run aruco_print_marker.exe
    Example:
    aruco_print_marker.exe 0 marker0.png -bs 75 -d ARUCO_MIP_36h12
    Reference:
    aruco_print_marker.exe <marker-id> <output-file-name>(.jpg|.png|.ppm|.bmp) -bs <bit-size> -d <dictionary>
    where:
    • marker-id is the integer ID of the marker you wish to print.
    • output-file-name is the desired marker image filename.
    • bit-size is the number of pixels per interior image bit
      optional.png
      Default = "50px".
    • dictionary is the marker dictionary you wish to use (we suggest ARUCO_MIP_36h12).
  • To generate printable markers for an entire dictionary at once, run aruco_print_dictionary.exe
    Example:
    aruco_print_dictionary.exe markers_36h12 ARUCO_MIP_36h12 -s 75
    Reference:
    aruco_print_dictionary.exe <output-directory> <dictionary> -s <bit-size>
    where:
    • output-directory is the directory to output the marker images to (must already exist).
    • dictionary is the marker dictionary you wish to use (we suggest ARUCO_MIP_36h12).
    • bit-size is the number of pixels per interior image bit
      optional.png
      Default = "75px".

Device configuration settings

  • Calibration files can be generated as above, under Calibration.
  • An example calibration file generated using a RealSense Gen2 camera unit is found at /ConfigFiles/OpticalMarkerTracker/realsense_gen2_calibration.yml
  • The creation of a custom calibration file for your camera is strongly recommended. Failure to do so can result in erroneous and/or unstable measurements.

  • attribute.png
    Type = "OpticalMarkerTracker"
    required.png
  • attribute.png
    CameraCalibrationFile Camera calibration file containing device-specific parameters measured by the camera calibration utility. Path is relative to DeviceSetConfigurationDirectory.
    required.png
  • attribute.png
    ToolReferenceFrame
    optional.png
    Default = "Tracker"
  • attribute.png
    TrackingMethod Tracking method.
    optional.png
    Default = "OPTICAL"
    • OPTICAL uses just RGB video.
    • OPTICAL_AND_DEPTH uses depth data and RGB video (work in progress).
  • attribute.png
    MarkerDictionary The dictionary whose markers are being used.
    required.png
    • ARUCO_MIP_36h12 Use of this dictionary is recommended. Some pre-generated marker images are included at /ConfigFiles/OpticalMarkerTracker/markers.
    • ARUCO
    • ARUCO_MIP_16h3
    • ARUCO_MIP_25h7
    • ARTAG
    • ARTOOLKITPLUS
    • ARTOOLKITPLUSBCH
    • TAG16h5
    • TAG25h7
    • TAG25h9
    • TAG36h10
    • TAG36h11
  • element.png
    DataSources
    required.png
    • element.png
      DataSource
      required.png
    • attribute.png
      MarkerId The integer identifier of the marker representing this tool.
      required.png
    • attribute.png
      MarkerSizeMm The size of the marker in mm (length of the black side edge).
      required.png

Example configuration file PlusDeviceSet_Server_OpticalMarkerTracker_Mmf.xml

This configuration file can be used with this example 3D Slicer scene: OpticalMarkerTracker_Scene.mrb

<PlusConfiguration version="2.4">
<DataCollection StartupDelaySec="1.0" >
<DeviceSet
Name="PlusServer: Optical marker tracker using MMF video"
Description="Tracking of printed ArUco markers using a simple camera. Marker positions and image data are broadcasted through OpenIGTLink (on port 18944 and 18945, respectively). To use a different camera, change CaptureDeviceId attribute (to 0, 1, 2, ...)." />
<Device
Id="VideoDevice"
Type="MmfVideo"
FrameSize="640 480"
VideoFormat="YUY2"
CaptureDeviceId="1" >
<DataSources>
<DataSource Type="Video" Id="Video" PortUsImageOrientation="MF" ImageType="RGB_COLOR" />
</DataSources>
<OutputChannels>
<OutputChannel Id="VideoStream" VideoDataSourceId="Video" />
</OutputChannels>
</Device>
<Device
Id="TrackerDevice"
Type="OpticalMarkerTracker"
CameraCalibrationFile="OpticalMarkerTracker/realsense_gen2_calibration.yml"
ToolReferenceFrame="Tracker"
TrackingMethod="OPTICAL"
MarkerDictionary="ARUCO_MIP_36h12">
<DataSources>
<DataSource Type="Tool" Id="Marker0" MarkerId="0" MarkerSizeMm="80" />
<DataSource Type="Tool" Id="Marker1" MarkerId="1" MarkerSizeMm="80" />
<DataSource Type="Tool" Id="Marker2" MarkerId="2" MarkerSizeMm="50" />
<DataSource Type="Tool" Id="Marker3" MarkerId="3" MarkerSizeMm="50" />
<DataSource Type="Tool" Id="Marker4" MarkerId="4" MarkerSizeMm="50" />
<DataSource Type="Tool" Id="Marker5" MarkerId="5" MarkerSizeMm="30" />
<DataSource Type="Tool" Id="Marker6" MarkerId="6" MarkerSizeMm="30" />
<DataSource Type="Tool" Id="Marker7" MarkerId="7" MarkerSizeMm="30" />
<DataSource Type="Tool" Id="Marker8" MarkerId="8" MarkerSizeMm="30" />
</DataSources>
<InputChannels>
<InputChannel Id="VideoStream" />
</InputChannels>
<OutputChannels>
<OutputChannel Id="TrackerStream" >
<DataSource Id="Marker0"/>
<DataSource Id="Marker1"/>
<DataSource Id="Marker2"/>
<DataSource Id="Marker3"/>
<DataSource Id="Marker4"/>
<DataSource Id="Marker5"/>
<DataSource Id="Marker6"/>
<DataSource Id="Marker7"/>
<DataSource Id="Marker8"/>
</OutputChannel>
</OutputChannels>
</Device>
<Device
Id="CaptureDevice"
Type="VirtualCapture"
BaseFilename="RecordingTest.igs.mha"
EnableCapturingOnStart="FALSE" >
<InputChannels>
<InputChannel Id="TrackerStream" />
</InputChannels>
</Device>
</DataCollection>
<PlusOpenIGTLinkServer
MaxNumberOfIgtlMessagesToSend="1"
MaxTimeSpentWithProcessingMs="50"
ListeningPort="18944"
SendValidTransformsOnly="true"
OutputChannelId="TrackerStream" >
<DefaultClientInfo>
<MessageTypes>
<Message Type="TRANSFORM" />
</MessageTypes>
<TransformNames>
<Transform Name="Marker0ToTracker" />
<Transform Name="Marker1ToTracker" />
<Transform Name="Marker2ToTracker" />
<Transform Name="Marker3ToTracker" />
<Transform Name="Marker4ToTracker" />
<Transform Name="Marker5ToTracker" />
<Transform Name="Marker6ToTracker" />
<Transform Name="Marker7ToTracker" />
<Transform Name="Marker8ToTracker" />
</TransformNames>
</DefaultClientInfo>
</PlusOpenIGTLinkServer>
<PlusOpenIGTLinkServer
MaxNumberOfIgtlMessagesToSend="1"
MaxTimeSpentWithProcessingMs="50"
ListeningPort="18945"
SendValidTransformsOnly="true"
OutputChannelId="VideoStream" >
<DefaultClientInfo>
<MessageTypes>
<Message Type="IMAGE" />
</MessageTypes>
<ImageNames>
<Image Name="Image" EmbeddedTransformToFrame="Image" />
</ImageNames>
</DefaultClientInfo>
</PlusOpenIGTLinkServer>
</PlusConfiguration>