Using Charles Proxy To Inspect And Debug Google Analytics

While it may be inglorious, debugging is an important part of web analytics. In order to use data, you need to be able to trust it. The ultimate source of truth for a Google Analytics implementation is being able to see the hits coming out of the web page, and seeing what values are being sent to Google Analytics. If you find anything that looks odd in your data, your first step should be to look at the hit data so you can figure out whether the source of the issue is the on-page tracking or the processing that happens on Google’s servers.

Charles Proxy is a hit-inspection tool that has proven valuable to us here at Bounteous. Charles is a heavy-duty, all-purpose tool that is not specialized to web analytics, which means that it takes extra set-up to use. But because it is general-purpose, it works in a variety of situations that specialized tools sometimes don’t handle.

How It Works

Charles Proxy is a proxy server running on your local machine. Traffic between your browser and the Internet gets directed through Charles, letting you inspect, record, or modify any and all traffic.

This set-up (called a Man in the Middle attack) is what The Bad Guys™ try to do when they hack your computer. There are no security problems when you do this to yourself, since the only person gaining access to your secure data is you, but many software security systems will complain until you configure them to recognize that Charles is not one of The Bad Guys™.

Setting Up Charles

After downloading and installing Charles, you will need to configure some settings for Charles to be able to inspect Google Analytics traffic. While Charles is available for all three operating systems, these instructions are specific to Windows.

Windows Proxy: This changes a setting in Windows that tells all applications (including browsers) to direct their internet requests through Charles.

SSL Proxy Settings: By default, Charles does not intercept secure (HTTPS) traffic unless specifically instructed to. Because Google Analytics uses a secure connection by default, Charles must be instructed to intercept Google Analytics traffic. You can also allowlist other analytics tools that you may need to inspect, such as Google Tag Manager, Adobe Analytics, or Tealium.

Installing the Charles Certificate: To read secure traffic in both directions, Charles needs to un-encrypt secure traffic from the server, and replace the server’s encryption with its own. For this to work, we need to install a certificate that tells your browser to trust Charles. Charles can provide such a certificate, but it takes a lot of clicking to make it work.

Under the Help menu in Charles, select “SSL Proxying” and then “Install Charles Root Certificate.” At the next pop-up window, select “Install Certificate.” At the next pop-up window, select the appropriate user, click next, and then select the option to choose a certificate store. Click browse to choose the certificate store, and select “Trusted Root Certification Authorities.”

Now you should be good to go!

Update

A recent updated to iOS has introduced a new hoop to jump through. If your device is running iOS 10 or higher, then after installing the Charles root certificate, you will additionally need to configure your device to trust that certificate. If you are importing certificates from a previous version of iOS, old certificates will be trusted by default, but new certificates must be trusted manually.

In order to trust the certificate, go into the Settings, and then select General > About. Scroll all the way down to the bottom of the About menu, and enter Certificate Trust Settings. This will have a list of your certificates, including the Charles Proxy certificate, where you can enable it as a trusted certificate for your device. See the screenshot below for a guide through the menus.

Things You Can Do With Charles

There is a large variety of things you can do with Charles. We will cover some common uses, in increasing order of how complex they are. The most complicated uses are also the most valuable, because they accomplish things that can’t be done with the common browser extensions and dev tools.

0. Hit Inspection

This use is numbered 0 because it is the most basic use of Charles for web analytics, and also what everything else is based on. It also requires no configuration, and minimal set-up.

After you’ve done the set-up described above, have Charles open, and select the “Sequence” view instead of “Structure.” Now go to your browser, and open a site with Google Analytics installed (such as bounteous.com). Now you see every single request the web page is making! (You may see additional requests from other web pages or applications you have open, especially email, IM, and file sync services.)

In order to filter for Google Analytics traffic, type “google-analytics.com” into the search box. Now you will only see the hits coming to and from Google Analytics, including the analytics.js file as well as the tracking beacons. In order to see the values being sent with the hit, select the “Request” tab at the top, and then the “Query” tab at the bottom (these are highlighted in the screenshot below).

The biggest downside to using Charles compared to integrated tools is that Charles does not label these parameters with what they represent to Google Analytics. In order to interpret these hit values, you have to refer to the documentation for the Measurement Protocol. With that resource in hand, we can see that the above hit is a pageview hit (t parameter) to the Bounteous homepage (dl parameter), and we can see the value of my Google cookie (cid parameter) and what property the data is being sent to (tid parameter). This immediately allows us to check simple data integrity issues, like “What property am I sending data to?” and “are any custom dimensions being set, and to what value?”

As you navigate around a website, Charles will continue to parse and record all the hits that you generate. You can easily go back and forth and compare values between hits for consistency.

Pro-tip 1: If there are multiple Google Analytics beacons on the page, you can use your UA number as the search string instead of “google-analytics” to find what’s being sent to each one.

Pro-tip 2: For very large hits (such as ecommerce hits with large shopping carts), the hit parameters will be sent in the body of the request instead of as query parameters. If this is the case, the hit will have a “Body” tab next to the “Query String” tab.

Pro-tip 3: The hit response can be copied and pasted into Excel.

1. Session Recording

After navigating around a site and generating a bucketload of hits, you can save it and open it back up in Charles later. This is most useful when you need to wait for Google Analytics to process data, such as when doing QA on Enhanced Ecommerce implementations. Having a full record of the hits that were generated allows you to drill down into the cause of errors, even if you didn’t notice them at the time.

2. Google Analytics Hit Validator

The “Map Remote” function in Charles will re-write a request to one URL to point to a different URL instead. You can use this to automatically send all Google Analytics hits on a webpage to the Hit Validator tool, providing automatic feedback from Google on whether your hits are correctly structured. Inside the Map Remote window, click “Add” to add a new mapping rule, and “www.google-analytics.com/collect” as the “Map From” and “www.google-analytics.com/debug/collect” as the “Map To” (Charles should parse the URL into the relevant fields).

This has the added benefit that your QA hits will not be collected and end up in reports. To view the validation, select the “Response” tab instead of the “Request” tab.

3. Swapping Out GTM Containers

Using Map Remote, as described in the previous tip, you can change a request for one GTM container into a request for another GTM container. You can achieve this by adding a new Map Remote rule. To replace the container GTM-XXXXXX with the new container GTM-YYYYY, enter “www.googletagmanager.com?id=GTM-XXXXXX” as the “Map From” and “id=GTM-YYYYY” as the “Map To” (if the hostname is left blank, it is used as a filter but not modified).

This is useful when you want to make radical changes to a container (such as clear-cutting one beyond hope of salvation), especially if that container has Custom HTML tags that push more data to the Data Layer. You can create a fresh new GTM container, and route requests to the old container to the new one instead, freeing you from dealing with the interference of the old container while it’s in development.

By modifying the target URL, you can also cause the request to serve specific versions or environments in GTM. This is somewhat less useful now that Workspaces exist, but can be useful if you are pushing against the Workspace limit in your container. This same principle also works with other Tag Management solutions such as Tealium and Ensighten.

4. Map Local

Where the “Map Remote” feature of Charles points a request from one URL to a different URL, the “Map Local” makes a request to a remote URL to instead return a file on your own hard drive. Many websites have a “helper file” or “library file” with analytics helper functions installed. This was more common before tag management solutions become popular, but some websites still use it to load data into the Data Layer on page load.

You can use Map Local to test changes to this file by saving the file, modifying it, and using Charles to direct requests to the original file to instead point to the modified version on your hard drive. You will need to figure out on your own the remote and local paths that need remapping, but the interface for the remote URL is the same as in the Map Remote tips, above.

5. Debugging External Devices

Perhaps the most useful feature of Charles is that it can proxy traffic any source, not just from the browser where you already have your dev tools and your extensions installed. The team at Bounteous has used Charles to inspect Google Analytics hits from some of the following sources:

• Web traffic from Android and iPhone browsers
• Mobile tracking from Android and iPhone apps
• Mobile tracking from an Android development kit running in an emulator
• App tracking from a PlayStation 4 app
• Measurement Protocol hits generated from a local Apache install running server-side tracking code

While some of these environments have their own specialized tools, it’s convenient to have a single tool that we know well and that works almost anywhere.

This is the most advanced usage of Charles, and requires the most set-up. First, you need to figure out the IP address of the computer where you’re using Charles. On a PC, you can run “ipconfig” from the command line. A quick Google search for “how do I find my IP” turns up results for other operating systems. The most important thing is to make sure you are finding your network IP, not your external IP (in particular, this means you should NOT use whatismyip.com).

Next, you need to configure Charles to accept external connections. From the Proxy menu, select Access Control Settings, and then add an IP address. You can use the “0.0.0.0/0” wildcard to allow all devices, or you can find the network IP address of the device that you are trying to connect (the latter is an extra step, but theoretically more secure).

Finally, you need to configure the external device to use Charles as a proxy for its internet access. Unlike your web browser, which uses a proxy setting from your own computer that Charles can modify, there is nothing in your phone or Playstation that Charles can access to tell it to use Charles. You have to configure this manually. The exact details will vary from one device to another, but the general idea remains the same.

First, connect your device to the same internet connection that your laptop is using, preferably over Wifi. Next, go into the configuration of your device, and find where you can change the settings of that internet connection. There should be area called “Proxy Settings,” with an option called “Manual,” where you can explicitly tell your device to use Charles as a proxy server. Enter the IP address of your laptop, and 8888 for the port (this is Charles’ default, but can be changed under Proxy Settings). The screenshot below is for an iPhone, but is representative of other devices.

Check your connection by visiting a website on your phone, and seeing if a) the internet connection works b) you can see the hits on Charles. Charles will prompt you to accept the connection the first time you do this.

Finally, we need to install Charles’ security certificate on the remote device, the same way we did for the main computer. To do this, you need to already be proxying your connection through Charles because the certificate is specific to each Charles installation. On your device, navigate to www.charlesproxy.com/getssl, and follow the instructions to install the certificate. Again, the steps vary for each device, but generally clicking “OK” a bunch of times in a row seems to work most of the time.

Once you’ve done this, try any site with Google Analytics and see if Charles can see the hit data being generated by the mobile traffic. Most apps use the same security certificates that the browser uses, so you should be able to see hits being generated by apps the same way. App hits (whether using the SDK or otherwise) use the same Measurement Protocol that website hits do, so you should be able to read them the same way as described in tip #0 at the beginning of this post.

Conclusion

Charles Proxy is a powerful tool. We rarely need to call on, but in situations where other tools may not work, Charles has proven reliable. The one drawback is the amount of configuration needed to get it into a working state, but hopefully the guidance in this blog post is enough to allow other people access to the power that Charles provides.