OpticalMarkerTracker is virtual device that provides position tracking of markers using any webcam or camcorder. You can print your own markers on any regular printer and attach to a flat surface on an object. The device uses algorithms implemented in OpenCV and ArUco software libraries.

Supported hardware devices

This is a virtual (software-only) device, which uses images taken by any optical imaging device that Plus supports, such as external webcam, built-in laptop camera - either forward or rear facing, camcorder connected through framegrabber, etc.

Supported platforms

This device can be used in combination with any device in Plus that provides a video stream. Most frequently it is used with Microsoft Media Foundation devices (webcams, framegrabbers).

Installation

To get started, print a set of markers and create a configuration file. It is recommended to use "PlusServer: Optical marker tracker using MMF video" configuration file that is included in your Plus installation package (see also at the bottom of this page) and print this sheet that contains a number of markers with different sizes (make sure to print in actual size: scaling should be 100%): marker_sheet_36h12.pdf. If you put the sheet in the field of view of your camera, the device will detect position and orientation of the markers.

To improve tracking accuracy, you need to create a calibration file for your camera as described below. If you need more tracking markers, you can print more by following instructions below.

Calibration

Camera calibration determines optical properties of your camera (distortion, focal length, etc) to improve tracking accuracy.

WARNING: Camera calibration uses an external application, which uses meters as unit for length. This is the ONLY place length should be specified in meters. Everywhere else in Plus it is millimeter.
Print the aruco calibration chessboard and measure the side length of the markers.
To generate a calibration file, run aruco_calibration.exe
Example:

aruco_calibration.exe live:0 camera_calibration.yml -size 0.03556
Reference:

aruco_calibration.exe live:<camera-index> <output-file-name>.yml -size <marker-size-m>
where:
- camera-index is the integer index of the camera you want to use (usually 0 or 1).
- output-file-name is the desired output file name.
- marker-size-m is the size of a marker side in meters.
With the video window selected (keyboard shortcuts will not work while the command line window is selected):
- Press a to capture image and add to the calibration image set.
- Press c to run calibration on the captured images. "repj" is a measure of the calibration error (normal is < 1).
- Press s to stop calibration.
For best results use at least 6 images including:
- At least one frontal image.
- Multiple images where the border of the chessboard is near the top or side image border.
- Multiple images where the chessboard is held at an oblique angle to the camera.

Creating printable marker images

A premade sheet of ARUCO_36h12 markers in three standard sizes (80mm, 50mm, and 30mm) is located at /ConfigFiles/OpticalMarkerTracker/marker_sheet_36h12.pdf.
Some pre-generated marker images from the ARUCO_MIP_36h12 dictionary are included at /ConfigFiles/OpticalMarkerTracker/markers.
To generate a single image for printing, run aruco_print_marker.exe
Example:

aruco_print_marker.exe 0 marker0.png -bs 75 -d ARUCO_MIP_36h12
Reference:

aruco_print_marker.exe <marker-id> <output-file-name>(.jpg|.png|.ppm|.bmp) -bs <bit-size> -d <dictionary>
where:
- marker-id is the integer ID of the marker you wish to print.
- output-file-name is the desired marker image filename.
- bit-size is the number of pixels per interior image bit
  
  Default = "50px".
- dictionary is the marker dictionary you wish to use (we suggest ARUCO_MIP_36h12).
To generate printable markers for an entire dictionary at once, run aruco_print_dictionary.exe
Example:

aruco_print_dictionary.exe markers_36h12 ARUCO_MIP_36h12 -s 75
Reference:

aruco_print_dictionary.exe <output-directory> <dictionary> -s <bit-size>
where:
- output-directory is the directory to output the marker images to (must already exist).
- dictionary is the marker dictionary you wish to use (we suggest ARUCO_MIP_36h12).
- bit-size is the number of pixels per interior image bit
  
  Default = "75px".

Device configuration settings

Calibration files can be generated as above, under Calibration.
An example calibration file generated using a RealSense Gen2 camera unit is found at /ConfigFiles/OpticalMarkerTracker/realsense_gen2_calibration.yml
The creation of a custom calibration file for your camera is strongly recommended. Failure to do so can result in erroneous and/or unstable measurements.
Type = "OpticalMarkerTracker"
CameraCalibrationFile Camera calibration file containing device-specific parameters measured by the camera calibration utility. Path is relative to DeviceSetConfigurationDirectory.
ToolReferenceFrame

Default = "Tracker"
TrackingMethod Tracking method.

Default = "OPTICAL"
- OPTICAL uses just RGB video.
- OPTICAL_AND_DEPTH uses depth data and RGB video (work in progress).
MarkerDictionary The dictionary whose markers are being used.
- ARUCO_MIP_36h12 Use of this dictionary is recommended. Some pre-generated marker images are included at /ConfigFiles/OpticalMarkerTracker/markers.
- ARUCO
- ARUCO_MIP_16h3
- ARUCO_MIP_25h7
- ARTAG
- ARTOOLKITPLUS
- ARTOOLKITPLUSBCH
- TAG16h5
- TAG25h7
- TAG25h9
- TAG36h10
- TAG36h11
DataSources
- DataSource
- MarkerId The integer identifier of the marker representing this tool.
- MarkerSizeMm The size of the marker in mm (length of the black side edge).

Example configuration file PlusDeviceSet_Server_OpticalMarkerTracker_Mmf.xml

This configuration file can be used with this example 3D Slicer scene: OpticalMarkerTracker_Scene.mrb

<PlusConfiguration version="2.4">
  <DataCollection StartupDelaySec="1.0" >
    <DeviceSet
      Name="PlusServer: Optical marker tracker using MMF video"
      Description="Tracking of printed ArUco markers using a simple camera. Marker positions and image data are broadcasted through OpenIGTLink (on port 18944 and 18945, respectively). To use a different camera, change CaptureDeviceId attribute (to 0, 1, 2, ...)." />
    <Device
      Id="VideoDevice"
      Type="MmfVideo"
      FrameSize="640 480"
      VideoFormat="YUY2"
      CaptureDeviceId="1" >
      <DataSources>
        <DataSource Type="Video" Id="Video" PortUsImageOrientation="MF" ImageType="RGB_COLOR" />
      </DataSources>
      <OutputChannels>
        <OutputChannel Id="VideoStream" VideoDataSourceId="Video" />
      </OutputChannels>
    </Device>
    <Device
      Id="TrackerDevice"
      Type="OpticalMarkerTracker"
      CameraCalibrationFile="OpticalMarkerTracker/realsense_gen2_calibration.yml"
      ToolReferenceFrame="Tracker"
      TrackingMethod="OPTICAL"
      MarkerDictionary="ARUCO_MIP_36h12">
      <DataSources>
        <DataSource Type="Tool" Id="Marker0" MarkerId="0" MarkerSizeMm="80" />
        <DataSource Type="Tool" Id="Marker1" MarkerId="1" MarkerSizeMm="80" />
        <DataSource Type="Tool" Id="Marker2" MarkerId="2" MarkerSizeMm="50" />
        <DataSource Type="Tool" Id="Marker3" MarkerId="3" MarkerSizeMm="50" />
        <DataSource Type="Tool" Id="Marker4" MarkerId="4" MarkerSizeMm="50" />
        <DataSource Type="Tool" Id="Marker5" MarkerId="5" MarkerSizeMm="30" />
        <DataSource Type="Tool" Id="Marker6" MarkerId="6" MarkerSizeMm="30" />
        <DataSource Type="Tool" Id="Marker7" MarkerId="7" MarkerSizeMm="30" />
        <DataSource Type="Tool" Id="Marker8" MarkerId="8" MarkerSizeMm="30" />
      </DataSources>
      <InputChannels>
        <InputChannel Id="VideoStream"  />
      </InputChannels>
      <OutputChannels>
        <OutputChannel Id="TrackerStream" >
          <DataSource Id="Marker0"/>
          <DataSource Id="Marker1"/>
          <DataSource Id="Marker2"/>
          <DataSource Id="Marker3"/>
          <DataSource Id="Marker4"/>
          <DataSource Id="Marker5"/>
          <DataSource Id="Marker6"/>
          <DataSource Id="Marker7"/>
          <DataSource Id="Marker8"/>
        </OutputChannel>
      </OutputChannels>
    </Device>
    <Device
      Id="CaptureDevice"
      Type="VirtualCapture"
      BaseFilename="RecordingTest.igs.mha"
      EnableCapturingOnStart="FALSE" >
      <InputChannels>
        <InputChannel Id="TrackerStream" />
      </InputChannels>
    </Device>
  </DataCollection>
  
  <PlusOpenIGTLinkServer
    MaxNumberOfIgtlMessagesToSend="1"
    MaxTimeSpentWithProcessingMs="50"
    ListeningPort="18944"
    SendValidTransformsOnly="true"
    OutputChannelId="TrackerStream" >
    <DefaultClientInfo>
      <MessageTypes>
        <Message Type="TRANSFORM" />
      </MessageTypes>
      <TransformNames>
        <Transform Name="Marker0ToTracker" />
        <Transform Name="Marker1ToTracker" />
        <Transform Name="Marker2ToTracker" />
        <Transform Name="Marker3ToTracker" />
        <Transform Name="Marker4ToTracker" />
        <Transform Name="Marker5ToTracker" />
        <Transform Name="Marker6ToTracker" />
        <Transform Name="Marker7ToTracker" />
        <Transform Name="Marker8ToTracker" />
      </TransformNames>
    </DefaultClientInfo>
  </PlusOpenIGTLinkServer>
  <PlusOpenIGTLinkServer 
    MaxNumberOfIgtlMessagesToSend="1" 
    MaxTimeSpentWithProcessingMs="50" 
    ListeningPort="18945" 
    SendValidTransformsOnly="true" 
    OutputChannelId="VideoStream" > 
    <DefaultClientInfo> 
      <MessageTypes> 
        <Message Type="IMAGE" />
      </MessageTypes>
      <ImageNames>
        <Image Name="Image" EmbeddedTransformToFrame="Image" />
      </ImageNames>
    </DefaultClientInfo>
  </PlusOpenIGTLinkServer>
</PlusConfiguration>