DTrace, or Dynamic Tracing, is a performance analysis and troubleshooting tool included by default with FreeBSD. DTrace can help locate performance bottlenecks in production systems and can be used to patch live running instructions. In addition to diagnosing performance problems, DTrace can help investigate and debug unexpected behavior in both the FreeBSD kernel and userland programs.

DTrace provides its own language (D Language) to help users author utilities that are customized to specific needs. While this introductory guide won’t cover writing scripts in D, more detailed information can be found here.

Enabling DTrace Support

To start, all necessary modules will need to be loaded with:

# kldload dtraceall

The DTrace Toolkit has several utilities written in ksh, so support for the Korn shell to fully utilize the system will be needed. 

# pkg install ksh93

The toolkit itself can be installed with:

# pkg install dtrace-toolkit

This includes a variety of pre-made scripts for diagnosing system information. It’s important to note that not all of these scripts have been specifically ported to FreeBSD and may need manual configuration. Many of the toolkit scripts are written in the D Language, which is similar to C++.

Monitoring DTrace 

DTrace utilizes probes, which are instrumentation points for capturing event data. Each probe is associated with its own action. Whenever the condition for a probe is met, the associated action is executed. Possible actions include logging information and modifying context variables. 

To check if DTrace was successfully loaded, and to list all active probes, execute the following: 

# dtrace -l | more

This will list each probe as well as the ID, Provider, module, and function name.

# dtrace -l | wc -l

This command will list the number of available probes, helpful for monitoring what probes can be enabled on each system.

For more information on probe identifiers, refer to dtrace(1).

Enabling Probes

Probes can be enabled with the dtrace command, omitting the -l option from the previous section. For enabled probes, DTrace will perform the default action, indicating that the probe has been fired, but not collecting any further event data. 

You can use the dtrace:::BEGIN probe to fire when DTrace starts:

# dtrace -n dtrace:::BEGIN

Additionally, an action can be assigned to each probe. Actions enable DTrace to interact with the system outside of the DTrace framework. These actions range from recording data, stopping a current process, raising signals on a process, or stop tracing.

# dtrace -n 'dtrace:::BEGIN { printf("Hello FreeBSD!\n"); }'

Use the Ctrl+C key combination to stop the process.

This modification on the previous command includes a specified action to display “Hello FreeBSD!” when DTrace starts.

An action can be assigned to each probe, allowing a massive amount of customization on what is monitored and the specific action DTrace executes when it is fired. Another possible use would be to set an action to the syscall::open*:entry to trace file opens as they happen.

Using DTrace Scripts

Some pre-made DTrace scripts can be found in the DTrace toolkit. It’s a good idea to check if one already exists there before creating your own, though customization is still possible.

The hotkernel script is designed to identify which function is using the most kernel time. It will produce output similar to the following:

# cd /usr/local/share/dtrace-toolkit
# ./hotkernel
Sampling... Hit Ctrl-C to end.
Upon termination, the script will display a list of kernel functions and timing information, sorting the output in increasing order of time:
kernel`_thread_lock_flags                                   2   0.0%
0xc1097063                                                  2   0.0%
kernel`sched_userret                                        2   0.0%
kernel`kern_select                                          2   0.0%
kernel`generic_copyin                                       3   0.0%
kernel`_mtx_assert                                          3   0.0%
kernel`vm_fault                                             3   0.0%
kernel`sopoll_generic                                       3   0.0%
kernel`fixup_filename                                       4   0.0%
kernel`_isitmyx                                             4   0.0%
kernel`find_instance                                        4   0.0%
kernel`_mtx_unlock_flags                                    5   0.0%
kernel`syscall                                              5   0.0%
kernel`DELAY                                                5   0.0%
0xc108a253                                                  6   0.0%
kernel`witness_lock                                         7   0.0%
kernel`read_aux_data_no_wait                                7   0.0%
kernel`Xint0x80_syscall                                     7   0.0%
kernel`witness_checkorder                                   7   0.0%
kernel`sse2_pagezero                                        8   0.0%
kernel`strncmp                                              9   0.0%
kernel`spinlock_exit                                       10   0.0%
kernel`_mtx_lock_flags                                     11   0.0%
kernel`witness_unlock                                      15   0.0%
kernel`sched_idletd                                       137   0.3%
0xc10981a5                                              42139  99.3%

This script will also work with kernel modules. To use this feature, run the script with -m:

# ./hotkernel -m
Other scripts can be run in a similar way, for a full list of pre-made DTrace scripts refer to the dtrace-toolbox github repository.