Skip to main content

Integrating Apache Traffic Server

This document provides instructions on configuring Apache Traffic Server (ATS) to use Backtrace's Invoker as a crash log helper. The workflow upon ATS crash is as follows:

  1. ATS launches the invoker on startup, which monitors the traffic_server process.
  2. Upon the traffic_server crash, the invoker launches ptrace via a script that provides metadata.
  3. This script, in turn, submits the crash file generated by ptrace to the coroner object store.

What You'll Need​

  • A Backtrace account (log in or sign up for a free trial license).
  • Your subdomain name (used to connect to your Backtrace instance). For example, https://example-subdomain.sp.backtrace.io.
  • A Backtrace project and a submission token.
  • Apache Traffic Server (tested with version 6.2.0)
  • Coroner Server running and accessible from the ATS machine. (See Coroner Server installation)
  • Coroner Client package on the ATS machine. (See Coroner Client Installation)
  • Backtrace Ptrace package installed on the ATS machine. (See Ptrace Installation)
  • Backtrace Invoker package installed on the ATS machine: $ sudo apt-get install backtrace-invoker.
  • (Optional but recommended) - Install the morgue command-line queryingtool: (See: Morgue Installation)

Set Up Helper Scripts​

invoker.sh​

#!/bin/sh
exec /opt/backtrace/bin/invoker -t "/home/ats/backtrace.sh %p" $@

backtrace.sh​

This script calls ptrace with any metadata you wish to gather and then submits it to the coroner. Here, we're extracting the version number using traffic_ctl, but feel free to add any additional metadata that you need.

#!/bin/sh

## Change the following as needed
PROJECT=ats
TOKEN=ats
DM=/export/ats
CF=/etc/coronerd/coroner.cf

PATH=/opt/backtrace/bin:/usr/bin:/bin:$PATH

# Add commands to gather any metadata that you wish to pass to the ptrace invocation
VERSION=`/opt/ts/bin/traffic_ctl metric get proxy.node.version.manager.short|cut -d ' ' -f 2`

mkdir -p ${DM}/
ptrace --kv="version:$VERSION" $1 -O ${DM}/ats
if test "$?" == "0"; then
   coroner -c $CF put -u ${PROJECT} ${TOKEN} ${DM}/*.btt

fi

Ensure that your traffic_server processes can run these scripts and have permission to write to the destination folder specified in backtrace.sh (referred to by the variable DM).

$ chmod +x invoker.sh
$ chmod +x backtrace.sh

Configure ATS Crash Log Helper​

To configure the ATS Crash Log Helper, add or edit the following line in /etc/trafficserver/records.config:

CONFIG proxy.config.crash_log_helper STRING /home/ats/invoker.sh

Verify Invoker Launch​

After making the configuration changes, restart ATS and verify that the invoker is running by executing the following commands:

$ sudo /bin/trafficserver restart
$ ps aux | grep invoker
nobody   11616  0.0  0.1   4336  1396 ?        T    16:43   0:00 /opt/backtrace/bin/invoker -t /home/ats/backtrace.sh %p --syslog --wait --host x86_64-unknown-linux-gnu

Test​

To test the setup, send a SIGABRT signal to traffic_server:

$ sudo kill -SIGABRT

Then, verify via the morgue tool that the number of crashes for the project has increased:

morgue list myproj
...

...
     Occurrences: 1 (100.00%)

Troubleshooting​

Launching backtrace.sh Manually​

To check for any errors, you can manually launch backtrace.sh:

$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9

If there are any errors, continue with the troubleshooting steps below to verify the correct installation of Backtrace components and proper permissions.

Verifying ptrace Installation​

Running /opt/backtrace/bin/ptrace on a process you own should generate a .btt file:

$ ps
  PID TTY          TIME CMD
 2291 pts/7    00:00:00 bash
11709 pts/7    00:00:00 ps

$ /opt/backtrace/bin/ptrace 2291
/home/test/bash.2291.1461078102.btt

$ ls *.btt
bash.2291.1461078102.btt

If the file is not generated, verify your ptrace installation. Refer to Ptrace Installation for assistance.

Verifying Coroner Installation​

To confirm that you can send snapshots to the server using coroner, perform the following steps:

$ coroner -c put project1 project1_token bash.2291.1461078102.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9

If you encounter issues, verify your coroner client installation. See Coroner Client Installation for more information.

Launching backtrace.sh Manually​

Try launching backtrace.sh manually to ensure there are no errors:

$ ./backtrace.sh 2291
/export/ats.2291.1461080575.btt
7750a16bfbb8667ca39de9a568c86897944a476143074275631cd13c842ce74d => 607dcbc50c634580951c0fdf832b2fc9

Ensuring ATS Has Permission to Run invoker.sh​

Make sure that ATS has the necessary permissions to execute invoker.sh:

$ chmod +x /path/to/invoker.sh

Verifying Invocation of backtrace.sh and Generation of .btt File​

You can verify if backtrace.sh is being invoked and generating the .btt file in the desired location by commenting out the last 3 lines of backtrace.sh.